Questions Frequently Asked About Student Rating Forms: Summary of Research Finding

Matthew Kaplan, Lisa A. Mets, Constance E. Cook

What do we know about the relationship between grades and student ratings?

Teachers in classes with higher expected grades have slightly higher ratings on teacher evaluations. Such results could mean that student ratings are biased and that instructors receive higher ratings because they give higher grades. However, in studies of courses with multiple sections, there are differences among instructors' average ratings when there is no difference in average expected grade. Therefore, we cannot conclude that instructors receive higher ratings because they give higher grades. Research done in experimental settings confirms that the effect of grades on student ratings may be overstated. In these experiments, sections graded on a "C" curve do not give significantly lower evaluations than those graded on a "B" curve. Grading fairness, on the other hand, does appear to influence student ratings. Studies indicate that instructors need to grade fairly and consistently and give students realistic expectations about their grades. When instructors give grades that contradict students' performance and expectations, they do, in fact, receive lower student ratings.

Should we believe student ratings reflect an instructor's ability to "entertain"?

Early experiments indicated that instructors who were 'entertaining' or 'expressive' (witty, enthusiastic, theatrical, or engaging) received high student ratings even though they delivered very little information in their lectures. Later analysis revealed serious flaws in these studies and showed the relationship between expressiveness and ratings to be exaggerated. More recent studies indicate that expressive instructors receive higher ratings because their expressiveness helps students learn. When students are not highly motivated (e.g., in introductory, required courses), instructor expressiveness has a larger effect on student achievement than does the amount of content covered. Expressive instructors stimulate and maintain student attention, and students learn more when they are engaged in the subject. Students and faculty agree that instructor enthusiasm is an important element of effective teaching. Furthermore, expressiveness includes a range of specific behaviors related to good lecturing, such as speaking emphatically, using humor, and moving about during lecture. Trained observers found that highly-rated faculty exhibit these behaviors more frequently than other faculty. While expressiveness was important, it was by no means the only factor that explained high ratings. Highly rated instructors also used examples, stressed important points, and asked questions.

What do student ratings tell us about teaching effectiveness?

The primary goal of teaching is learning. Studies have shown that student ratings are higher in courses where student achievement is higher (as measured by students' performance on final exams). However, the ratings are not necessarily consistent among all items that may appear on an instructor's evaluation form. In general, ratings of the course or instructor on overall items ('Overall this is an excellent instructor' 'Overall this is an excellent course') show a consistent positive relationship to achievement. For example, in classes where the achievement level is high, instructors tend to receive high ratings; when achievement is low, instructors tend to receive low ratings. Ratings of the instructor that ask about specific skills (such as organization, enthusiasm, clarity) are more variable in their relationship to student achievement. As a result, most researchers recommend that administrators include data from overall items in personnel decisions and use results of more specific questions as feedback to help instructors improve. They also suggest further research so that we can better explain exactly which aspects of teaching have the greatest impact on learning in various types of courses.

How do we know that students make consistent judgments about teaching effectiveness?

Comparisons of ratings among students in a course show a fairly high level of agreement among raters. It is important to note that this level of agreement increases as the number of students in the course increases. If fewer than 15 students (or less than two thirds of the class) respond, the results of student ratings should be viewed with caution. Moreover, studies show that ratings of the same instructor by current students and alumni are similar. Likewise, alumni's overall ratings of an instructor are similar to the ratings they gave when they were students.

Instructors will derive the greatest benefit from their student evaluations if they discuss the results with a colleague or a teaching professional. CRLT consultants will help instructors design evaluations, interpret results, and develop strategies for incorporating student feedback. CRLT also strongly encourages the use of multiple methods of evaluation and provides services to help instructors and units gather information from a variety of sources. For assistance call 734-764-0505.

Bibliography

For good overviews of the literature:

Aleamoni, L.M. (1987). Typical faculty concerns about student evaluation of teaching. In Techniques for evaluating and improving instruction, New Directions for Teaching and Learning, no. 31 (pp. 25-31). San Francisco: Jossey-Bass.

Braskamp, L.A., Brandenburg, D.C. & Ory, J.C. (1984). Evaluating teaching effectiveness. San Francisco: Jossey-Bass.

Marsh, H.W. (1987). Students' evaluations of university teaching: Research findings, methodological issues, and directions for future research. International Journal of Educational Research, 11:253-388.

Marsh, H.W. & Dunkin, M.J. (1992). Students' evaluations of university teaching: A multidimensional approach. In J.C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 8, pp. 143-233). New York: Agathon Press.

Centra, J.A. (1993). Reflective faculty evaluation. San Francisco: Jossey-Bass.

For information on grading leniency:

Abrami, P.C., Dickens, W.J., Perry, R.P. & Leventhal, L. (1980). Do teacher standards for assigning grades affect student evaluations of instruction? Journal of Educational Psychology, 72:107-118. Braskamp, L.A., Brandenburg, D.C. & Ory, J.C. (1984). Evaluating teaching effectiveness. San Francisco: Jossey-Bass.

Howard, G.S. & Maxwell, S.E. (1980). The correlation between student satisfaction and grades: A case of mistaken causation? Journal of Educational Psychology, 72:810-820.

Howard, G.S. & Maxwell, S.E. (1982). Do grades contaminate student evaluations of instruction? Research in Higher Education, 16:175-188.

Marsh, H.W. & Overall, J.U. (1980). Validity of students' evaluations of teaching effectiveness: Cognitive and affective criteria. Journal of Educational Psychology, 72:468-475.

Peterson, C. & Cooper, S. (1980). Teacher evaluation by graded and ungraded students. Journal of Educational Psychology, 72:682-685.

Snyder, C.R. & Clair, M. (1976). Effects of expected and obtained grades on teacher evaluation and attribution of performance. Journal of Educational Psychology, 68:75-82.

For information on expressivity:

Abrami, P.C., Leventhal, L. & Perry, R.P. (1982). Educational seduction. Review of Educational Research, 52:444-464.

Aleamoni, L.M. (1987). Typical faculty concerns about student evaluation of teaching. In Techniques for evaluating and improving instruction. New Directions for Teaching and Learning, no. 31 (pp. 25-31). San Francisco: Jossey-Bass.

Feldman, K.A. (1988). Effective college teaching from the students' and faculty's view: Matched or mismatched priorities? Research in Higher Education, 28:291-344.

Marsh, H.W. (1984). Experimental manipulations of university motivation and their effect on examination performance. British Journal of Educational Psychology, 54:206-213.

Marsh, H.W. & Dunkin, M.J. (1992). Students' evaluations of university teaching: A multidimensional perspective. In J.C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. 8, pp. 143-233). New York: Agathon Press.

Marsh, H.W. and Overall, J.U. (1980). Validity of students' evaluations of teaching effectiveness: Cognitive and affective criteria. Journal of Educational Psychology, 72:468-475.

Marsh, H.W. and Ware, J. (1982). Effects of expressiveness, content coverage and incentive on multidimensional student rating scales. Journal of Educational Psychology, 74:107-116.

Murray, H.G. (1985). Classroom teaching behaviors related to college teaching effectiveness. In J.G. Donald & A. M. Sullivan (Eds.), Using research to improve teaching. New Directions for Teaching and Learning, no. 23 (pp. 21-34). San Francisco: Joss ey-Bass.

Murray, H.G., Rushton, J.P. & Paunonen, S.B. (1990). Teacher personality traits and student instructional ratings in six types of university courses. Journal of Educational Psychology, 82:250-261.

Perry, R.P. (1985). Instructor expressiveness: Implications for improving teaching. In J.G. Donald & ; A. M. Sullivan (Eds.), Using research to improve teaching. New Directions for Teaching and Learning, no. 23 (pp. 35-49). San Francisco: Jossey-Bass.

For information on student ratings as a measure of effective teaching:

Abrami, P.C., d'Apollonia, S. & ; Cohen, P.A. (1990). Validity of student ratings of instruction: What we know and what we do not know. Journal of Educational Psychology, 82:285-296.

Cohen, P.A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51:281-309.

Cohen, P.A. (1986, April). An updated and expanded meta-analysis of multisection student rating validity studies. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco. (ERIC: ED 270 471)

Feldman, K.A. (1989) The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Research in Higher Education, 30:583-645.

For information on the accuracy and consistency of student ratings:

Centra, J.A. (1974). The relationship between student and alumni ratings of teachers. Educational and Psychological Measurement, 34:321-326.