Monday, December 15, 2014

On the importance of course evaluations at the University of Toronto

My university (the University of Toronto, Toronto, Canada) recently developed a new policy and new procedures on undergraduate student evaluations [Policy on the Student Evaluation of Teaching in Courses]. The policy was the work of a committee that began with the assumption that student evaluations were a good thing. As far as I can tell, the committee did not spend any time examining the pedagogical literature to see if the evidence supported their assumptions. As you can see from the title of the policy, the assumed purpose of student evaluations is to judge the quality of teaching.

The university has a website: Course Evaluations. Here's what the administrators say ...
Course evaluations are read by many people at UofT, including instructors, chairs, deans, the provost and the president. They are used for a variety of purposes, including:
  • To help instructors design and deliver their courses in ways that impact their students’ learning
  • To make changes and improvements to individual courses and programs
  • To assess instructors for annual reviews
  • To assess instructors for tenure and promotion review
  • To provide students with summary information about other students’ learning experiences in UofT courses
It is essential that students have a voice in these key decision-making processes and that’s why your course evaluation is so important at the University of Toronto.

Your feedback is used to improve your courses, to better your learning experience and to recognize great teaching.
There's so much wrong with these statements that I hardly know where to begin. First, course evaluations may be read by some instructors but certainly not by chairs, the provost, and the President of the University of Toronto. At best, the departmental chair might routinely read a summary of the scores on course evaluations for a few lecturers. It's absurd to suggest that the President and/or Provost reads course evaluations in a university of 75,000 students taking on average five courses per semester.

What are the legitimate purposes of student evaluations since we know that course evaluations are not good indicators of teaching effectiveness and since we know that students are not in a position to judge whether the material is accurate or up-to-date?

To help instructors design and deliver their courses in ways that impact their students’ learning
This may be true in some cases. Sometimes students submit valuable comments that can make a difference. This is especially true in upper level courses. On the other hand, most students are not knowledgeable about the different ways that courses can be taught. In a memorize/regurgitate course, for example, they are not demanding a switch to student-centered learning. Most of the fundamental changes in teaching come from instructors who really care about learning and not from student evaluations.
To make changes and improvements to individual courses and programs
Again, there are a few times when comments from the students lead to minor improvements but, for the most part, changes and improvements are motivated by other concerns. It's rare that we encounter a student who really understands how to design a program and how it could be improved. They usually don't see the big picture.
To assess instructors for annual reviews
In my experience at this university (36 years), this doesn't happen very often with professors. It does happen with sessional lecturers (see video below) but given what we know about the effectiveness of student evaluations from the pedagogical literature, it shouldn't happen at all.
To assess instructors for tenure and promotion review
There might be rare exceptions when student evaluations are abused in this way but I've never seen an example of a tenure decision or a promotion that's been significantly affected by student evaluations. Maybe it's more important in other departments but I doubt it. This is a myth that needs busting. It's not how things work at a major research-intensive university. Nobody wants to admit this.
To provide students with summary information about other students’ learning experiences in UofT courses.
It's probably true that some students pick some courses based on students evaluations. (They are made public.) The important questions are: should students be basing their decisions on student evaluations, and should the university be encouraging this?
Here's a scary video from the University of Toronto website. It's scary because we are supposed to be teaching, and practicing, critical thinking and there's no evidence of that in the video. We are supposed to base our decisions on evidence and rationality but the pedagogical literature is almost unanimous in condemning students evaluations as realistic measures of teaching effectiveness. I wonder if the two professors in this video have actually studied the issue and read the literature?

Maybe they should read this article: Students don’t know what’s best for their own learning.
... universities that rely on student evaluations are likely to punish good teachers and encourage those who simply make it easy for students. Most universities have codes of conduct that require decisions to be made on valid evidence. Any manager discussing student evaluations when reviewing lecturers’ performance is probably breaching that part of their own job requirements. Given the evidence, student evaluations are a distraction from the responsibility to provide the best possible education for the nation.

For university teachers, the challenge remains the same it has always been: keeping students motivated, while ensuring that they learn. Part of achieving that requires teaching students how to learn, not just what to learn, and to ask them what they are working on and how hard they are trying. But it is probably best to avoid asking if they are happy with the course.

One of the additional motivations for change was to cut down on class time spent on student evaluations. The university decided to switch to online evaluations as promoted in this embarrassing video.

Guess what happened? The participation rate plummeted so that in many courses less than 25% of the students filled out the evaluations. Who could possibly have seen that coming?

Here's a paper that was just published but similar results have been published over the years in the pedagogical literature and those papers were certainly available to the decision makers at the University of Toronto before they decided to go online.

Capa-Aydin, Y. (2014) Student evaluation of instruction: comparison between in-class and online methods. Assessment & Evaluation in Higher Education, 1-15. published online December, 2014 [doi: 10.1080/02602938.2014.987106]
This study compares student evaluations of instruction that were collected in-class with those gathered through an online survey. The two modes of administration were compared with respect to response rate, psychometric characteristics and mean ratings through different statistical analyses. Findings indicated that in-class evaluations produced a significantly higher response rate than online evaluation. In addition, Rasch analysis showed that mean ratings obtained in in-class evaluation were significantly higher than those obtained in online evaluation. Finally, the distributions of student attendance and expected grade in both modes were compared via chi-square tests, and were found to differ in the two modes of administration.


  1. According to Rate my Professor, Prof. Moran is given a rather low grade. I hesitate to say this but I wonder if this has something to do with his low opinion of the process.

  2. Male professors tend to get higher scores than female:

  3. Random thoughts about student evaluations:

    The better students in the class are pretty good at evaluating the prof, other things being equal, because they can distinguish between the prof, the difficulty of the subject, and their own skills. The poorer ones are less consistent, often thinking problems with the class are because of the prof when it isn't true.

    Evaluations are never that useful to me. By the end of class, I generally know if it has gone well or badly, from results of assignments, tests, and discussion in class. Evaluations tended to be consistent with my own observations. When one part of my performance is graded badly, I'm usually not really surprised about which one it is.

    The end of class is an emotional time for me in several ways (and I'm tired), so I don't add to that by reading the evaluations then. I set them aside, loose them, and read them the next term or the next year, whenever they resurface.

    I have gotten some good suggestions about how to improve the class from evaluations. Not often. And I do value some of the positive written comments. (And fret obsessively over some of the bad ones, especially the absurdly wrong ones.)

    When student evaluations are on line and thus optional, many students don't bother. Students who care deeply do bother. Students doing badly tend to care more about evaluating the prof. This is not good.

    Heads of departments have rarely bothered with evaluations, but sometimes they do, especially when they can just look at the summary stats.

    I have found that looser grading standards (if the students figure them out before evaluation time) get me better evaluations.

    Overall, I think that it's good to have student evaluations, especially the written comments, for my own use but they should be used carefully (if at all) by the university for evaluating profs.

    (Though my evaluations have varied, they tend to be in the B+/A- range where most teachers end up unless the class actively dislikes them.)

    1. The better students in the class are pretty good at evaluating the prof, other things being equal, because they can distinguish between the prof, the difficulty of the subject, and their own skills.

      Most of the pedagogical research shows that this is generally not true, even in upper level courses where students get better at it.

      (Though my evaluations have varied, they tend to be in the B+/A- range where most teachers end up unless the class actively dislikes them.)

      This is fairly typical, as you point out. We all know how to get better evaluations if we want them. It's pretty easy to game the system.

      For the past few years I've been in an experimental mode with respect to teaching. I'm trying lots of new things in my courses. Sometimes they work and sometimes they don't. I don't recommend this approach if you want good student evaluations every year.

    2. "Most of the pedagogical research shows that this is generally not true, even in upper level courses where students get better at it." -- Interesting. Thanks!

  4. kids in their late teens and early twenties are not smart about anything. Call me a ageist if you must!
    their kids! What the hell do they know about good teaching!!The high scorers think the prof is great and the low scorers don't. duh.
    To be a professor in the old days meant to PROFESS knowledge kids couldn't get otherwise. Oddly enough textbook writers, like the host here, are a threat to profs .
    Does Prof Moran give his students what his textbook doesn't give? i don't know.
    As long as the kids understand/memorize what they need for a foundation before further study and for exams then the prof has done his job.
    Do the kids have insight in how a teacher matches up?
    Hmmm. They could only compare based on how much they absorb.
    I don't see them as good judges.
    indeed however Profs are dealing with kids and not adults with decades behind them in thinking ability.
    I know from my watching youtube and other sources that kids are a bad source for accurate evaluation of profs/serious researchers in most subjects.
    I still say the kids memorize things far and ahead of understanding them .
    i did two mail courses in french successfully without learning ANY french.
    Memorization is a tool and not a door to knowledge.
    Uof t students have lots of problems by the way. A write off.

    1. And here we have the great Paul Lynde on video doing his amazing Robert Byers impersonation.

      Byers: "i did two mail courses in french successfully without learning ANY french."

      I'm shocked. Well, I'm glad to see he's better at absorbing science... from all the science books... he said he read...

  5. When I was teaching full time my department (of biochemistry, University of Birmingham) didn't have student evaluations, so I have no personal experience). However, it seemed to be generally known who the students considered to be the best teacher, and it wasn't anyone that I would have guessed. The main criterion seemed to be giving out detailed course notes that made it very clear what was expected in the exam. A capacity to hold students' interest (this guy wasn't interesting) or to explain difficult ideas clearly didn't seem to enter it. If we'd had evaluations I'm pretty sure I'd have been rated very low, because I was teaching stuff the students didn't like and found difficult (enzyme kinetics).

    Never having heard Larry lecture I can only guess why some students rate him as poor: it's clear from everything he writes here that he attaches a huge importance to correctness, and I suspect that the poorer students don't share that opinion. They'd prefer someone who'd give high grades to answers that are more or less right. I'm more impressed by the student who said that Larry's lectures were what decided him to go into science.

    Incidentally, to call the second video embarrassing is to overrate it: it's awful. The first one shines by comparison.

    1. Never having heard Larry lecture I can only guess why some students rate him as poor: it's clear from everything he writes here that he attaches a huge importance to correctness ...

      I don't give "lectures" any more because my courses are all based on student-centered learning. In the past, when I did give lectures, I tried to emphasize basic concepts and ideas and I concentrated on those areas where the students where most likely to have misconception (junk DNA, Central Dogma, evolution, enzyme pathways, photosynthesis).

      The idea, supported by experimental data, is that you can most easily stimulate critical thinking if you make the students re-evaluate something they thought was correct. Some students find this exciting. Some don't like it.

      I think it's important to push students out of their comfort zone. If you do it correctly, you can get high praise from your students. For me, it works most of the time but I'm still learning how to do it correctly every time.

  6. God forbid that I should be cynical or anything but the rationale offered for the use of student evaluations may not be relevant to their actual function. May I suggest that the administration views the student evaluations as a measure of the success the teacher has in leading the student to satisfaction with the product? If the university is regarded as a business selling courses to student customers, customer satisfaction surveys are indispensable. For those who believe a successful learning experience means the student was lead to invest, then emotionally satisfied, this is also true.

    If this is the case, lack of evidence that the student evaluations can address the issues raised by the rationale is entirely irrelevant I think.

    1. Yes, I think you've put your finger on the problem.

      My administrators want me to announce in class that students evaluations are important but they want me to give the reasons they put on the web site. In other words, they want me to lie.

      Maybe we should just tell our students the real reason why university administrators want good student evaluations?

    2. Regarding evaluations – false positives are far more likely than false negatives IMO.

      The problem is then that higher expectations lead to greater frustrations and greater frustrations can also often lead to lower evaluations.

      Of course, higher expectations are a good thing which brings us full circle.

      ITMT - I can only guess that many of Larry’s students are of the Pre-Med kind, a fact which would color this discussion considerably.

    3. @Tom

      I don't want this discussion to be about me and my student evaluations. For the record, I'm very happy with my student evaluations. That doesn't make them right.

    4. Hi Larry - gotchya

      I was merely attempting to identify yet another confounding variable...


      ... if I may be allowed to repeat my self yet again: I would love to be one of your students.

      best regards

  7. In my experience on my University's Promotion and Tenure committee, the best way to send your student ratings down the toilet is to properly deal with academic dishonesty.

    Unfortunately, real evaluation of teaching by a faculty colleague requires as much effort as reviewing a grant proposal, and who has time for more of that?

  8. We have manadatory course evaluations, usual given in the last lecture of the quarter, with the students filling out mark-sense forms and writing comments on forms for 15 minutes. These include about 8-12 questions that we can submit, in addition to the questions already on the form. It's a nuisance to administer. I don't think it has much influence on my promotion, since the last time I was up for promotion was 36 years ago.

    But actually I have found the student responses very useful. Each year, of the 18 or so areas in which they rate me, the ones in which I do well or do worse are the same ones. That is a message worth paying attention to (and I keep trying harder in those areas). The questions I submit, which are about how well different aspects of the course did (the guest lectures, the homeworks, the computer exercise, the posting of audio recordings and slides from the lectures) are also very helpful.

    The students are also asked to wax eloquent in four areas (how much the course challenged them, what aspects were good, were bad, and how it could be improved). I pay a lot of attention to those written comments which raise issues that are particularly of concern to them. I should say that I am not teaching in any large classes, so I can easily read all of the responses.

    We also have mandatory peer-evaluations with a colleague sitting in on one of our lectures each year. When I do see those evaluations, I am usually disappointed, because they take a lecture in which I know I screwed up some, and describe it far too positively.

  9. I know a man who goes to my church who won teacher of the year/medical teacher of the year or something at UoT last year i think. Faulty or students being the judge i'm unsure but the point is I'm sure he would say its very determined by how the kids take stuff in .
    Not so much about bigger ideas of the big concepts of science.
    he is by the way a YEC creationist and sees adam/eve as real and not from apes.
    Possibly being a creationist helps. Just kidding. We want to win on the merits and not make profs everywhere switch for career reasons.

  10. On WEIT, there's news of a biology professor who was called out for teaching creationism, with substantial evidence of his proselytizing available from student reviews. So I guess they can be useful sometimes.

    1. If the students call out a prof for teaching important truths then they should be called out of their university and work in the mines and the miners kids get those university student positions.
      If creationism is censored or punished its not only immoral but illegal in a free nation and a Christian one.
      Banning conclusions is banning the truth in a free nation.
      Its the same thing as saying religious conclusions are false.
      to say that breaks social contract as no power is to establish religious truth/untruth.