Contextualizing and interpreting student responses
Student responses to SETC can be difficult to interpret and put to use. The first step in getting useful data is to create “good” questions (see the guide here). The next step is to use the following parts of the SETC report to frame your interpretation and set limits on its generalizability:
- Instructor Narrative
- Response Rate
- Response Distribution
- Questions about Course Difficulty, Engagement, and Overall Experience
- Comment Question
Instructors have the opportunity to write a narrative that will be displayed on the course report at the end of the term. Narratives can go a long way to contextualize student responses. Some examples of comments to include are:
- If the course is a requirement for a degree program that students feel they haveto take rather than are taking out of personal interest.
- If the course is new / being trialed, or a new pedagogical approach is being tested.
- Whether extra support was given for EAL students, and how.
- Whether time was spent upskilling first year students on how to do well at university;
In Spring of 2017, the Advisory Committee for the student Evaluation of Teaching and Courses (ACE) asked the SETC team to investigate the question, “which students are responding to the SETC survey at SFU?” A subsequent analysis found:
- Females are more likely to respond than males
- Domestic students are more likely to respond than international students
- Students with higher GPAs are more likely to respond
This is general information that warrants further inquiry. However, it may be helpful to frame your interpretation by generalizing the results to likely respondents (female domestic students with relatively higher GPAs). Mainly, this analysis showcased the need for further demographic data to understand whose learning experiences are currently reflected in SETC reports and, of equal importance, whose learning experiences are not reflected.
In addition to averages, standard deviations, and response counts, SETC reports display the response distribution for each question. Examine the response distribution to determine if the mean is an accurate measure of central tendency. For example, a response distribution that has scores clustered at the lower and upper ends of a scale (i.e. a bimodal distribution) will have a mean in the middle of the scale, even though the middle of the scale was infrequently selected by students. In this case, it is better to use a description of the response distribution than to rely on the mean as an accurate reflection of student experience.
Questions about course difficulty, engagement, and overall experience
Three questions were added to every SETC questionnaire to help bolster the validity of the response data and inform ongoing measurement improvements. Each of the following questions reflects a student experience variable that features prominently in the SET literature as having a large influence on student course ratings:
- Course difficulty: How easy was the course? (Very easy, Easy, Medium, Hard, Very hard)
- Engagement: I attended class… (All the time, Most of the time, About half of the time, Rarely, Never)
- Overall experience: Overall, the quality of my learning experience in this course was… (Very good, Good, Fair, Poor, Very poor)
- Course difficulty
The course difficulty question provides context for interpreting results. If a student perceives a course to be difficult, it can lower scores on other questions. When interpreting a SETC report, compare the difficulty rating to outcomes on the other questions; you may notice that other scores are lower than expected. A lower score may be due to factors outside of the instructor’s immediate control (i.e. mismatch between course and pre-requisites, a high proportion of registrants may be non-majors, the course is conceptually difficult). (See Greenwald and Gillmore, 1997; Marsh 2007 in the Reading Room).
- Course engagement
A student who always attends class may have a different experience than a student who attends infrequently. Keep the responses to the attendance question in mind when looking at other questions that involve in-class experiences or activities. Students with high attendance are better equipped to answer these questions. On the other hand, if a course has a lot of resources available outside of the classroom, with an emphasis on flexible online learning, for example, than responses to the attendance question will be less relevant.
- Students’ overall experience
This “global” question of students’ overall experience was added as a criterion measure (AERA, APA, NCME, 2014). A criterion measure can be compared against a student’s score selections to other questions to see if those other questions are varying in the same way as the criterion measure. More concretely, if a student rated their overall learning experience as very good (a score of ‘5’) it would be expected that they would rate aspects of their learning experience, as represented by other questions (e.g. “The instructor created a respectful learning environment”) highly. The idea is that all questions on the SETC form are related to a student’s overall learning experience, and, are therefore, related to criterion measure.
In cases where there is a mismatch, or negative relationship between the criterion measure and another question, it is important to review the other question for validity:
- The overall learning experience is rated highly, and another question was rated low.
- The overall learning experience is rated low, and another question was rated high.
In both of these cases, we can consider that the question may have a weak impact on student learning experience and therefore may have low utility. Alternatively, the question may be unclear to students. Either way, this is an important signal that the question should be investigated across reports and reviewed using the principles in the question-making guide.
Finally, responses to the comment question can provide a rich understanding of student experiences. Students often reflect on aspects of their learning experience in the comment section that were not directly addressed by the rest of the evaluation, or they go more in depth on a topic covered by a response scale question. Look for comments that have constructive or specific reflections to help interpret learning experience versus vague or unspecific language. For example, “the text book wasn’t very helpful compared to the lecture notes, I barely used it,” versus, “great course!”
Additionally, look for themes across comments and, if possible, over course iterations to get an understanding of student experiences and relate those themes back to the response scale questions. This is not to say that a single comment should be dismissed. The benefit of this qualitative space is that individual voices can be heard.