There's an extensive pedagogical literature on the issue [Student Evaluations: A Critical Review] [New Research Casts Doubt on Value of Student Evaluations of Professors] [Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors] [Evaluating Methods for Evaluating Instruction: The Case of Higher Education] [Of What Value are Student Evaluations?] [Why Student Evaluations of Teachers are Bullshit, Some Sources ].
Much of this literature points out what is obvious to any university instructor; students evaluations are not very good at measuring teaching effectiveness.
Most of you won't have time to read several thousand papers in the pedagogical literature but that doesn't mean you should ignore the conclusions of the experts.1 After all, we are scientists so shouldn't we base our policies on evidence? If there's no evidence to support the claim that student evaluations measure teaching effectiveness then why in the world would university departments use them in tenure decisions, promotions, and salary increases? And why would universities use student evaluations almost exclusively to give out teaching awards?
In order to get some feel for one of the problems with student evaluations, I suggest you read an article by Stanley Fish in the New York Times (2010): Deep in the Heart of Texas. And here's an opinion piece by "Dr. Myron L. Fox," famous for the Dr. Fox Effect—the pedagogical equivalent of the Sokal Hoax.
No longer is it possible for an intelligent, well-informed person to believe that summative student evaluations accurately measure student learning or teaching effectiveness. But the widespread, almost universal use of summative student course evaluations of teachers, together with the nonchalant manner in which professors ritualistically distribute them at the end of every semester, can give the impression to both students and professors that the majority of professors support the idea that course evaluations are good indicators of how well the teachers are teaching and of how well their students are learning. The fact is that the majority of professors, across all disciplines, believe that summative student evaluations aren't good or fair measures of such things (see Birnbaum; Crumbley). This is one of academia's dirty little secrets. Dirty, because this belief, though well-founded, manifests a cynicism about contemporary higher education that tragically lies right at the core of the educators' relationship and interaction with students. It is secret, because - outside of a specialized academic literature which virtually preaches to the choir - the matter is almost never discussed candidly, and because many professors - believers and unbelievers alike - so often actually pay lip service to the view that student evaluations are valuable in these ways. The unbelievers who talk the orthodox talk do so mainly to keep their jobs or simply to avoid rocking the boat. But many have spoken out. They come from every department and every type of institution large and small, research-oriented and teaching-oriented. Below are some examples. Most of them warn that if summative course evaluations continue to be used in an attempt to measure teaching-effectiveness, then academic standards and ideals will be, perhaps irreparably, subverted: grades will continue to inflate and college courses will continue to be dumbed down and excellent professors will continue to be denied tenure and promotions, or jobs altogether, simply because they failed to pander to the increasing desire of students to be entertained or at least to be relieved of the hard work that genuine higher education requires; for there are many who are willing - if not happy or eager - to cave to these pressures and the job market in academia remains tight and the competition high.The CAUT (Canadian Association of University Teachers) is one group of academics who recognize the problem. They have fomulated a Policy on the Use of Anonymous Student Questionnaires in the Evaluation of Teaching
Wouldn't it be nice if universities in Canada (and elsewhere) adopted this policy? Wouldn't it be nice if they went even further? Wouldn't it be nice if my own department (Biochemistry) acted like scientists?
- Any procedure initiated by the administration or the senior academic body to evaluate teaching performance, including any proposal to employ anonymous student questionnaires, should have the agreement of, or have been negotiated with the academic staff association, and should be incorporated in the collective agreement or faculty handbook. Academic staff associations should be aware, when negotiating the use of student questionnaires, that anonymous student evaluations of teachers may serve as vehicles for transmitting popular misconceptions, expectations and prejudices, to the disadvantage of, for example, women and visible minorities. Such procedures should be fair and include an appropriate procedure for an academic staff member to comment on any set of ratings and to contest any assessment or decision made on the basis of those ratings. Academic staff associations should provide expert advice and counsel to academic staff members in reviewing their own results, and should also support academic staff members in whose cases student ratings are being used inappropriately.
- Procedures for the evaluation of teaching should take into account all relevant sources of information about teaching. Anonymous student ratings should never be the primary measure of teaching performance. Rather, the systematic use of a teaching dossier should be encouraged. Unless negotiated as discussed under Article 1, results of anonymous student ratings should be placed in that dossier only with the consent of the academic staff member.
- Surveys of student opinion about teaching should not be characterized or described as if they measure teaching effectiveness. While students are uniquely placed to comment on their own reactions to what happens in the classroom, they are not in a position to assess all of the components of teaching effectiveness.
- In post-secondary institutions where the results of student surveys are considered to be part of the individual's confidential personnel file, the results of such surveys should be accorded the same degree of protection as students' academic records. When student comments and/or survey results are published, they should not be included in the personnel file.
- Where/when student organizations conduct anonymous student surveys and publish the results in order to assist students in the selection of their courses, academic staff participation should be optional, and no penalties direct or indirect should follow a refusal to participate. Such student-organized evaluations should not be used by post-secondary institution administrations as a means of assessing teaching performance.
1. I hang out with some of these experts when I go to conferences on university education. It's quite rare to find someone who defends student evaluations. Many of my colleagues at these meetings are teaching professionals who have suffered from college administrators who use student evaluations to make hiring and firing decisions.