Skip to content

On Data Types and SETs (Student Evaluations of Teaching)

November 12, 2013

‘Tis the season of faculty reviews.

I myself recently submitted my 5-year, post-tenure review. In my packet (207 pages long; 20 pages of text and the remainder was supporting documents; crazy, I know), I had to speak about my teaching evaluations. We use a mechanism called the IDEA form put out by the IDEA Center (based in Manhattan, KS). It asks students to rate their learning progress on learning objectives that you, the professor, have determined are important for the course you are teaching. In my field it is things like “gaining factual knowledge”, “learning fundamental principles”, etc. Students are asked to respond to these prompts with one of the following statements: “No Apparent Progress”, “Slight Progress”, “Moderate Progress”, “Substantial Progress” and “Exceptional Progress”. These are assigned numbers, 1-5, increasing from “No Apparent Progress” (1) to “Exceptional Progress” (5). The report we get as individual faculty is 4 pages long and actually reports the number of students responding in each of the categories for each prompt. It then calculates the average and the standard deviation for the responses for that prompt. It also averages across learning goals to produce a “Progress on Relevant Objectives” where the objectives are weighted by how important you considered a particular learning objective. And here is where the problems begin to arise.

THESE DATA ARE ORDINAL DATA. Ordinal data are categorical data than can be ranked. To take an example from an interesting recent paper, Olympic medals are ordinal. They are categories (Bronze, Silver and Gold) that can be ranked (Gold > Silver > Bronze). But, as ordinal data, they cannot be added, subtracted, multiplied and divided as interval data can be. Let’s say we designate them with short-hand symbols: Gold=3, Silver=2 and Bronze=1. If these were interval data you could add them together (e.g, if you earn 3 Bronze medals that would be the equivalent of earning a single Gold medal) or average them (but what is the average for a person who gets a Silver and a Bronze? Bronze and a half?) But, as the two examples indicate, conducting mathematical operations on ordinal data is ridiculous. But, of course, people do it all the time.

In Likert’s original paper, he argues that you should not examine average responses to individual prompts. However, Likert did advocate averaging (or summing) across a group of related “Likert type items” (the individual prompts) to create a “Likert Scale” (these are terms invoked by social scientists). This ability rests on two assumptions (see page 42): (1) the distribution of responses is normally distributed across the categories (even though his own data in this original paper showed skew) and (2) the distances between the categories are equidistant in whatever scale is assumed to underlie the categories. Of course, the measurement scale of these categories is problematic as I will show below.

I will illustrate why this is not a valid practice going back to the original example of Olympic medals.

Let us assign Gold the value 3, Silver the value 2, and Bronze the value 1. A value of 0 is assigned to placing outside the top three regardless of place.

Let us say that we have three swimmers and we want to distinguish between the three of them. Their races and the results are below.

Person 1

Person 2

Person 3

Medal

“Score”

Medal

“Score”

Medal

“Score”

50 m freestyle

Bronze

1

Gold

3

Silver

2

100 m breaststroke

Bronze

1

Gold

3

Silver

2

100 m backstroke

Bronze

1

4th

0

5th

0

200 m freestyle

Bronze

1

4th

0

Silver

2

200 m crawl

Bronze

1

4th

0

5th

0

400 m ind. Medley

Bronze

1

4th

0

5th

0

Likert Scale (summed)

6

6

6

Likert Scale (mean)

1

1

1

Who is the better swimmer. Neither Likert Scale sheds light on this question. Is it better to have 2 Gold medals and four 4th place finishes or 6 Bronzes. Is Person 3 equivalent to Person 2 even though Person 2 has 2 Golds and has beaten Person 3 in all races except the 200 m freestyle. I think most swimmers would rather have two Gold medals than 6 Bronze medals (but I could be wrong).

I made up the data above which you might say is unfair. Let’s look at real data. If Olympic medals really are interval scale data (as Likert fans would like us to believe), the difference between Gold and Silver should be the same as the difference between Silver and Bronze. The fact that I speak of differences in this way implies that these are interval data because I am implying that I can subtract a Silver from a Gold. If we did this, the difference between Gold and Silver would be 1 (Gold-Silver = 3-2 = 1) and the difference between Silver and Bronze would also be 1 (Sliver-Bronze = 2-1 = 1). Indeed, one of Likert’s untested assumptions is that when people see a Likert item, they assume that the “distance” between them is equidistant. This assumption has been tested, and found lacking, more than once but the most recent test I could find is here.

I really like the men’s 100 m. A very exciting race. Let’s look at some data.

Table 2: Actual times in the Olympic men’s 100 m run for the last 6 Olympic games.

2012

2008

2004

2000

1996

1992

100 m dash

Time

Time

Time

Time

Time

Time

Gold

9.63

9.69

9.85

9.87

9.84

9.96

Silver

9.75

9.89

9.86

9.99

9.89

10.02

Bronze

9.79

9.91

9.87

10.04

9.9

10.04

When you calculate the differences between these times, you get the following:

Table 3: Time differences between medal winners in 6 Olympic runs of the men’s 100 m run.

2012

2008

2004

2000

1996

1992

Gold-Silver

-0.12

-0.2

-0.01

-0.12

-0.05

-0.06

Silver-Bronze

-0.04

-0.02

-0.01

-0.05

-0.01

-0.02

What is important here is that the differences between the times of the various medal winners are not equidistant. This is especially apparent when we graph the data. The difference between Gold and Silver is, in 5 out of 6 cases, larger than the difference between Silver and bronze.

Image

The reason for this is that we have taken ratio scale data (you can be 2x faster than someone else) and converted it to ordinal data by awarding medals. And, when we did, we lost the ability to determine the magnitude of difference between these categorical data. If you had told me that in 2008 Usain Bolt won Gold and Yohan Blake took Silver, all I would have known is that Bolt is faster than Blake. I would not have known that Usain Bolt crushed Blake by 0.2 seconds which is forever in a 100 m race.

Indeed, when a student responds “Exceptional Progress” on a particular learning objective and another responds “Substantial Progress” on the same objective, is the difference one unit? Or is it more? Or is it less? Is a person with an average score of 4.0 on a learning objective twice the teacher as someone with a 2.0? It is impossible to know because these are ordinal data that do not have scale in the way that a ruler has scale.

So, ordinal data, (which is what Likert Type and Likert Scale data are):

  1. Should not be added or subtracted or any other mathematical combining (What is Bronze + Bronze?) including averaged ((Bronze +Silver)/2 = Bronze and a half?).
  2. Do not make any claims about the distance between one category and another. If one assumes that they do, one is likely to be wrong.

Now, all is not lost. You can examine responses to Likert-type data and even Likert scales. You simply have to treat them as the ordinal data they are and look at their medians and modes or their actual distributions. These distributions of responses can be surprisingly informative when you use them to examine things like student responses to teaching evaluations using Likert-type responses. I will talk about this in forthcoming post.

Advertisements
No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Insect Ecology Lab

at the University of Dayton (Dr. Chelse Prather & students)

ecoqui

Quantitative Genetics and more

Arthropod Ecology

Writings about arthropod ecology, arachnids & academia at McGill University

Random Walks

Mr. Chase blogs about math

Dynamic Ecology

Multa novit vulpes

Small Pond Science

Research, teaching, and mentorship in the sciences

%d bloggers like this: