My university (the University of Toronto, Toronto, Canada) recently developed a new policy and new procedures on undergraduate student evaluations [Policy on the Student Evaluation of Teaching in Courses]. The policy was the work of a committee that began with the assumption that student evaluations were a good thing. As far as I can tell, the committee did not spend any time examining the pedagogical literature to see if the evidence supported their assumptions. As you can see from the title of the policy, the assumed purpose of student evaluations is to judge the quality of teaching.
The university has a website: Course Evaluations. Here's what the administrators say ...
Course evaluations are read by many people at UofT, including instructors, chairs, deans, the provost and the president. They are used for a variety of purposes, including:
- To help instructors design and deliver their courses in ways that impact their students’ learning
- To make changes and improvements to individual courses and programs
- To assess instructors for annual reviews
- To assess instructors for tenure and promotion review
- To provide students with summary information about other students’ learning experiences in UofT courses
It is essential that students have a voice in these key decision-making processes and that’s why your course evaluation is so important at the University of Toronto.
Your feedback is used to improve your courses, to better your learning experience and to recognize great teaching.
Student evaluations are an important part of the undergraduate experience at most universities. But how effective are they?
You would think that universities might have studied this question and applied critical thinking and evidence-based reasoning to the question. You might think that the popularity of student evaluations at universities is largely because they have proven to be reliable indicators of teaching effectiveness.
Think again. The pedagogical literature contains dozens of studies indicating that student evaluations are not reliable indicators of teaching effectiveness. Here are some recent papers that show you what the experts are thinking and what kind of evidence is being published.
Gormally, C., Evans, M. and Brickman, P. (2014) Feedback about Teaching in Higher Ed: Neglected Opportunities to Promote Change. CBE-Life Sciences Education 13, 187-199. [doi: 10.1187/cbe.13-12-023]
Despite ongoing dissemination of evidence-based teaching strategies, science teaching at the university level is less than reformed. Most college biology instructors could benefit from more sustained support in implementing these strategies. One-time workshops raise awareness of evidence-based practices, but faculty members are more likely to make significant changes in their teaching practices when supported by coaching and feedback. Currently, most instructional feedback occurs via student evaluations, which typically lack specific feedback for improvement and focus on teacher-centered practices, or via drop-in classroom observations and peer evaluation by other instructors, which raise issues for promotion, tenure, and evaluation. The goals of this essay are to summarize the best practices for providing instructional feedback, recommend specific strategies for providing feedback, and suggest areas for further research. Missed opportunities for feedback in teaching are highlighted, and the sharing of instructional expertise is encouraged.
Osborne, D. and Janssen, H. (2014) Flipping the medical school classroom: student acceptance and student evaluation issues (719.4). The FASEB Journal 28, 719.4.
Flipping the classroom has generated improvement on summative exam scores in our first-year medical students; however, faculty members adopting this teaching methodology often receive lower satisfaction rating on student evaluations. In previous years, these same professors received outstanding evaluation ratings when the same topics were presented using standard didactic lectures. We feel that this decreased student satisfaction may be result of two distinct causes. First, students who have been accustomed to didactic lectures often come to class unprepared and therefore are incapable of the critical thinking and problem solving skills need in the flipped classroom. Second, the evaluation tool which was appropriate for didactic lectures is inappropriate for the analyzing the flipped classroom methodology. The student evaluations have improved in that last several years; however, the transition was not accomplished without difficulty. Anyone planning to pursue this teaching approach should be prepared to weather the storm of sub-standard student evaluations and, if possible, prepare their administrators for this potential outcome. Our experience suggests that faculty persistence targeted at changing student culture and expectations can help in this process. Accurately determining student’s acceptance of this relative new teaching methodology is important. Improvements in the teaching methodology can be made only when the evaluation tool is valid. It is felt a new evaluation tool can be developed based on results obtained in student focus-groups coupled with cognitive assessment outcomes.
Wilson, J.H., Beyer, D. and Monteiro, H. (2014) Professor age affects student ratings: halo effect for younger teachers. College Teaching 62, 20-24. [doi: 10.1080/87567555.2013.825574]
Student evaluations of teaching provide valued information about teaching effectiveness, and studies support the reliability and validity of such measures. However, research also illustrates potential moderation of student perceptions based on teacher gender, attractiveness, and even age, although the latter receives little research attention. In the present study, we examined the potential effects of professor age and gender on student perceptions of the teacher as well as their anticipated rapport in the classroom. We also asked students to rate each instructor's attractiveness based on societal beliefs about age and beauty. We expected students to rate a picture of a middle-aged female professor more negatively (and less attractive) than the younger version of the same woman. For the young versus old man offered in a photograph, we expected no age effects. Although age served as a detriment for both genders, evaluations suffered more based on aging for female than male professors.
Blair, E. and Valdez Noel, K. (2014) Improving higher education practice through student evaluation systems: is the student voice being heard? Assessment & Evaluation in Higher Education, 1-16. [doi: 10.1080/02602938.2013.875984]
Many higher education institutions use student evaluation systems as a way of highlighting course and lecturer strengths and areas for improvement. Globally, the student voice has been increasing in volume, and capitalising on student feedback has been proposed as a means to benefit teacher professional development. This paper examines the student evaluations at a university in Trinidad and Tobago in an effort to determine whether the student voice is being heard. The research focused on students’ responses to the question, ‘How do you think this course could be improved?’ Student evaluations were gathered from five purposefully selected courses taught at the university during 2011–2012 and then again one year later, in 2012–2013. This allowed for an analysis of the selected courses. Whilst the literature suggested that student evaluation systems are a valuable aid to lecturer improvement, this research found little evidence that these evaluations actually led to any real significant changes in lecturers’ practice.
Braga, M., Paccagnella, M., and Pellizzari, M. (2014) Evaluating students’ evaluations of professors. Economics of Education 41:71–88. [doi: 10.1016/j.econedurev.2014.04.002]
This paper contrasts measures of teacher effectiveness with the students’ evaluations for the same teachers using administrative data from Bocconi University. The effectiveness measures are estimated by comparing the performance in follow-on coursework of students who are randomly assigned to teachers. We find that teacher quality matters substantially and that our measure of effectiveness is negatively correlated with the students’ evaluations of professors. A simple theory rationalizes this result under the assumption that students evaluate professors based on their realized utility, an assumption that is supported by additional evidence that the evaluations respond to meteorological conditions.
Stark, P.B. and Freishtat, R. (2014) An Evaluation of Course Evaluations. [PDF]
Student ratings of teaching have been used, studied, and debated for almost a century. This article examines student ratings of teaching from a statistical perspective. The common practice of relying on averages of student teaching evaluation scores as the primary measure of teaching effectiveness for promotion and tenure decisions should be abandoned for substantive and statistical reasons: There is strong evidence that student responses to questions of “effectiveness” do not measure teaching effectiveness. Response rates and response variability matter. And comparing averages of categorical responses, even if the categories are represented by numbers, makes little sense. Student ratings of teaching are valuable when they ask the right questions, report response rates and score distributions, and are balanced by a variety of other sources and methods to evaluate teaching....
Recommendations
- Drop omnibus items about “overall teaching effectiveness” and “value of the course” from teaching evaluations (SET): They are misleading.
- Do not average or compare averages of SET scores: Such averages do not make sense statistically. Instead, report the distribution of scores, the number of responders, and the response rate.
- When response rates are low, extrapolating from responders to the whole class is unreliable.
- Pay attention to student comments—but understand their limitations. Students typically are not well situated to evaluate pedagogy.
- Avoid comparing teaching in courses of different types, levels, sizes, functions, or disciplines.
- Use teaching portfolios as part of the review process.
- Use classroom observation as part of milestone reviews.
- To improve teaching and evaluate teaching fairly and honestly, spend more time observing the teaching and looking at teaching materials.
We've been discussing Ann Gauger's claim that evolution is impossible because she was unable to transform a modern enzyme into another related one by changing a small number of amino acids.
I pointed out that this is not how evolution works. In some cases, you can easily show that two enzymes with different specificities can evolve from a common ancestor that could carry out both reactions. Such enzymes are said to be "promiscuous."
Here's Ann Gauger's latest post: In Explaining Proteins (and Life), Here's What Matters Most. She says ...
So now, let's address enzyme evolution and the divergence of enzymes to produce related families and superfamilies. Larry Moran says that modern enzymes evolved by specializing from a promiscuous ancestor. As evidence, he says modern enzymes can sometimes catalyze reactions with several substrates (the chemicals they bind to and change), and that it is possible to shift these enzymes to favor one substrate over another. He gives several examples or provides links to them.
Here's another place where he and I agree. Promiscuous enzymes can be shifted with just a few mutations to a new reaction specificity, provided the capacity for the reaction already exists in the starting enzyme, and each step is small and selectable. They can evolve easily, because they can already carry out the reaction in question. Larry Moran's description of the process is actually quite good, despite the digs he takes at us.
It strikes me that Larry Moran would know we agree with him on these points if he had read our papers.
So, what's the problem?
Turns out that changing one related enzyme into another with a different specificity wasn't the goal of her experiment. Here's what she was really trying to do ...
The Big Problem
Here's the big problem -- the arrival of novelty.
Novelty or innovation means the appearance of something not already present. It's the opposite of promiscuity. So a way to create novelty is absolutely essential to explain modern cells, as I will demonstrate.
...
Here's the heart of the matter. Promiscuity cannot solve the problem of novelty. Mutation, natural selection, and drift cannot drive the creation of novelty of all those new protein folds. That's what Doug Axe and I have been testing all along, from Doug Axe's 2004 paper to this most recent one. Based on our experiments, the problem of how innovation originates remains unsolved.
Now I get it (not). What she and Doug Axe were really trying to do was to intelligently design an entirely new enzyme.
They failed.
Therefore Intelligent Design Creationism is falsified.
That seems logical to me.
Here's a video by Dr. Lodge McCammon. He has a website: lodgemccammon.com. Here are his credentials.
Dr. Lodge McCammon is an educational innovator. His career began in 2003 at Wakefield High School in Raleigh, North Carolina, where he taught Civics and AP Economics. McCammon received a Ph.D. from NC State University in 2008 and continued his work by developing innovative practices and sharing them with students, teachers and schools across the world. McCammon is a musician who spends much of his time in the recording studio composing curriculum-based music. His songs and related materials can be found in Discovery Education Streaming. He is also an education consultant who provides professional services, including keynote speeches, presentations, curriculum development, and a variety of training programs.
Watch the video and discuss. I think you can guess what I think. I reject one of the basic premise; namely that online courses are taught by the very best teachers. How do we know who is the best teacher just by watching videos?
Here's a question for your consideration. It concerns "reflective teaching." Imagine that you record yourself teaching an incorrect version of the citric acid cycle or a flawed version of the Central Dogma of Molecular Biology. How many times do you have to watch that video to recognize that what you are teaching is wrong? Is it more than three? Less than ten?
The students in my lab course measured the activity of trypsin at different pH's. They discovered that the enzyme was most active at a pH of about 8.0-8.5 and that activity fell off rapidly at pH values above and below this optimum. This is consistent with results in the published literature (see figure from Sipsos and Merkel, 1970). Here's the exam question ...
What was the pH optimum of trypsin activity? Can you explain this in terms of the normal biological function of the enzyme and the physiological conditions under which it is active? Do you expect there to be a strong correlation between the optimal pH of an enzyme’s activity and the pH of the cell/environment where it is active?
Sipos, T., and Merkel, J. R. (1970) An effect of calcium ions on the activity, heat stability, and structure of trypsin. Biochemistry, 9:2766-2775 [doi: 10.1021/bi00816a003]
Most biochemistry students are taught that enzymes are highly specific. It's certainly true that the stereospecificity of some enzymes is extraordinary but is it true in general? Here's one of the exam questions that the students in my course had to answer ....
All three of the enzymes (trypsin, alcohol oxidase, β-galactosidase) that you assayed in the past three months are active with several different substrates substrates. Is this behaviour typical or are most enzymes highly specific? Aminoacyl tRNA synthetases are the classic examples of enzymes that are highly specific. Why? Do aminoacyl-tRNA synthetases ever make mistakes?
One of the other exam questions is based on an experiment students did with an enzyme they purified. They digested the enzyme with trypsin and then analyzed the peptides by mass spectrometry. They were able to match the peptides to the sequence databases to identify the protein and the species. The exam question is ...
Nobody knows for sure how many functional protein-encoding genes there are in the human genome. About 20,000 potential protein-encoding genes have been identified based on open reading frames and sequence conservation but it is not known if all of them are actually expressed. How can you use Mass Spec to find out how many functional protein encoding genes we have? [see the cover of Nature from May 29, 2014: click on about the cover]
The students in my lab course are writing their final exam. Prior to the exam they were given 22 questions and they knew that five of them would be on the exam. I thought that Sandwalk readers might enjoy coming up with answers to some of the questions.
The possible remains of King Richard III of England have recently been discovered. His identity has been confirmed by DNA PCR analysis. Descendants of his mother in the female line have the same mitochondrial DNA as King Richard. However, the results with the Y chromosome were surprising. None of the descendants in the all-male lineage had the same Y chromosome markers as King Richard. This is almost certainly due to something called a "false-paternity" event. (There are other ways of describing this event.) Given what you know about PCR, what are some possible sources of error in this analysis? Would you be prepared to go back in time and accuse one of the Kings of England of being a bastard? [Identification of the remains of King Richard III]
(The lab experiment was to analyze various foods to see if they were made from genetically modified plants.)
Note: It's extremely unlikely that the "false-paternity" event occurred in the lineage leading directly to any of the Kings and Queens of England.
MicroRNAs are a special class of small functional RNA molecules. The functional RNA is only about 22 nucleotides long and most of the well-characterized examples bind to mRNA to inhibit translation and/or destabilize the message.
The big questions for many of us are how many different microRNAs are there in a typical cell and how many of them have a real biological function. These questions are, of course, part of the debate over junk DNA. Are there thousands and thousands of microRNA genes in a typical genome and does this mean that there's a lot less junk DNA than some of us claim?
The journal Cell Death and Differentiation has devoted a special issue to microRNAs [Special Issue on microRNAs – the smallest RNA regulators of gene expression]. There are four reviews on the subject but none of them address the big questions.
That didn't stop the journal from leading off with this introduction ...
It is now well recognised that the majority of non-protein-coding genomic DNA is not “junk” but specifies a range of regulatory RNA molecules which finely tune protein expression. This issue of CDD contains an editorial and 5 reviews on a particular class of these regulatory RNAs, the microRNAs (miRs) of around 22 nucleotides, and which exert their effects by binding to consensus sites in the 3'UTRs of mRNAs. The reviews cover the role of miRs from their early association with CLL to other forms of cancer, their importance in the development of the epidermis and their potential as disease biomarkers as secreted in exosomes.
I'm not certain what the editors mean when they say that "it is now well recognised ..." I interpret this to mean that there are a large number of scientists who are completely uniformed about the structure of genomes and the debate over junk DNA. In other words, it is now well recognized that some scientists don't know what they are talking about.
I don't know any expert who would claim that 50% of large genomes consist of genes that specify regulatory RNAs involved in fine-tuning protein expression. Do you?
On a related issue, Wilczynska and Bushell begin their review with ...
Since their discovery 20 years ago, miRNAs have attracted much attention from all areas of biology. These short (~22 nt) non-coding RNA molecules are highly conserved in evolution and are present in nearly all eukaryotes.
Sequence conservation is an important criterion in deciding whether something is functional. In order to use conservation as a measure of function you have to establish some standards that let you distinguish between sequences that are "conserved" by negative selection and those that have drifted apart by random genetic drift.
What do Wilczynska and Bushell mean when they say that microRNAs are "highly conserved"? The most highly conserved genes exhibit about 50% sequence identify between prokaryotes and eukaryotes. They are almost identical within mammals. Other highly conserved genes are about 80% identical within animals (e.g. between insects and mammals). As far as I know, the sequences of most putative microRNAs aren't even similar within mammals and certainly not between mammals and fish.
The phrase "highly conserved" has become meaningless. It's now a synonym for "conserved" because nobody ever wants to just say "conserved" and they certainly don't want to say "moderately conserved" or "weakly conserved" even if it's the truth.
Ann Gauger and her creationist collaborator, Doug Axe, have been swapping amino acid residues in one kind of protein hoping to show that they cannot change it into another. They have deliberately ignored any clues that might be derived from assuming that evolution happened.
They have succeeded in their goal. None of their constructs have a different activity. They conclude that evolution is disproved.
This video was the winner in the 2013 FASEB competition for "Stand Up for Science." The title was "Funding Basic Science to Revolutionize Medicine."
I'm sure their hearts are in the right place but I fear that videos like this are really just contributing to the problem. It makes the case that basic research should be funded because ultimately it will pay off in technologies to improve human health. If you buy into that logic then it's hard to see why you should fund research on black holes or studies of plate tectonics.
Don't we have a duty to stand up for ALL basic research and not just research that may become relevant to medicine? Besides, if the only important basic research that deserves funding is that which has the potential to contribute to medicine, then shouldn't funding be directed toward the kind of "basic research" that's most likely to pay off in the future? Is that what we want? I don't think evolutionary biologists would be happy but everyone working with cancer cells will be happy.
The best argument for basic research, in my opinion, is that it contributes to our knowledge of the natural world and knowledge is always better than ignorance. This argument works for black holes, music theory, and for research on the history of ancient India. We should not be promoting arguments that only apply to our kind of biological research to the exclusion of other kinds of basic research. And we should not be using arguments that reinforce the widespread belief that basic research is only valuable if it leads to something useful.
Intelligent Design Creationists have found it impossible to make a positive case for intelligent design and the existence of a supernatural designer. Instead, they concentrate on trying to prove that evolution is wrong. Turns out, they're not very good at that either.
The latest attempt is by Ann Gauger posting on the best Intelligent Design Creationist website.1 She outlines her case at: Is Evolution True? Laying Out the Logic.
Michael Behe is one of the few Intelligent Design Creationists who have come up with reasonable, scientific, defenses of creationism. I give him credit for that and for the fact that it often takes some effort to show why he is wrong.
However, when he has been proven wrong he should admit it and move on. He should instruct his fellow creationists to move on as well. That's not what happened with respect to his book on The Edge of Evolution. Last July he doubled down when a new study appeared that refuted his claims. Amazingly, Behe said that his ideas were vindicated. I questioned his logic in: CCC's and the edge of evolution and Michael Behe's final thoughts on the edge of evolution.
After a thorough discussion, we conclude that Behe is wrong about his edge of evolution. There's nothing in evolutionary theory, or in experimental results, that prevents the evolution of new functions with multiple mutations.
Now Ken Miller has posted an article making many of the same points [Edging towards Irrelevance]. Miller rightly demands an apology from Behe but his main point is that Michael Behe's recent behavior has made him largely irrelevant in the debate over Intelligent Design Creationism. I agree.
What this means is that there is nobody left in the Intelligent Design Creationist community who deserves serious attention from scientists. It will still be fun dealing with them but the game has become more like whac-a-mole than science. PZ Myers1 makes the same point: Aren’t we all more than a little tired of Michael Behe?.
It's kinda sad.
1. Unfortunately, PZ weakens his case by misrepresenting Behe's argument. PZ says, The hobby horse he’s been riding for the past few years is the evolution of chloroquine resistance in the malaria parasite: he claims it is mathematically impossible." That's just not true. Behe's entire case rests on the fact that chloroquine resistance is well within the edge of evolution. Behe has no problem with the evolution of chloroquine resistance.
I believe that we need to change the way we teach. But not the way you probably think. Watch this video to see what's really important about teaching.
Hat Tip: Alex Palazzo, who I hope will help us make the transition to 21st century teaching.
My son Gordon and his friend Colby are working on a new game called "For the King." It looks really cool. (I'm playing with an early version.)
Check out the website: IronOaks Games. They are also on Facebook: IronOaks Games. Twitter is @IronOakGames where you can make suggestions and follow development.
This is a game that reminds you of Hack and Nethack except with much better graphics. I think all the old people (like me) are going to like it. Ms Sandwalk asked for a carnival and got it (see below) and I'm pumping for a university.
