Multiple Choice Questions that Assess Higher Order Cognition

In the last T&L conversation, Cindy Xin (from TLC) and I talked about our experience at the Pacific Northwest Summer Institute (PNSI) on Scientific Teaching.  During the discussion on Sept 13, we briefly touched on the challenges of using multiple-choice questions (MCQs) to assess higher level thinking skills. In one of the workshops at PNSI, we discussed whether our current assessment practices assessed higher orders of cognition, and we reviewed some ways to improve MCQs as assessment tools. The context for these methods is a large lecture classroom.

What are Higher Order Cognition (HOC) questions?

The workshop used Bloom’s taxonomy to define HOC. Essentially, these are questions that ask the student to analyze, synthesize or evaluate. Whereas questions that access lower order cognition (LOC) ask the student to demonstrate knowledge or comprehension.

A Bit about Multiple Choice Formats

We reviewed the formats of MCQs. In the example below, the question is called the Stem, the options (A. – C.) are termed Distractors, and the correct answer is referred to as the Key (in this case, D).

Q. Is your kitten cute?

A. yes

B. no

C. What is a kitten?

D. She is a poop factory.

The classic format can be useful for assessing HOC, but only if strong distractors are used. Weak ones can lead the student to the correct answer without having to think about the question. Other MCQ formats that are useful for assessing HOC include Two-tiered questions, multiple true/false and alternate choice questions.

Two-tiered questions are linked questions that first ask a student to select a response and then justify the response in a second questions. These are very useful for exposing misconceptions, however, they are time consuming and a poorly written first question might lead the student response in the second question.

Multiple true/false questions that use at least 5 questions are useful since they reduce error associated with guessing. There is an excellent example of this type of question in the workshop materials that I have attached as a PDF.  The question asks the students to analyze a diagram and then answer a series of true or false questions.

MCQ formats that are not very suitable for assessing HOC include matching questions and complex (K-type) MCQs. K-type questions have several options in which students have to select the best combination of answers. Although these questions are useful for testing analytical skills, they are typically misleading and confusing to students.

You may be using k-type questions if the following distractors look familiar:

A. #1
B. #1 and #2
C. #3, #7, but not #4
D. #1-#10
E. Do not have enough information to answer the question

??

Blooming your MCQs

A useful activity for getting started is to ‘bloom’ your MCQs with a colleague. Essentially this involves reviewing your assessments and categorizing your questions as HOC, LOC or somewhere in between. I propose that we have a ‘Blooming Party’ perhaps as part of the T&L conversations at which we can review our MCQs and brainstorm ways to improve them. We could also try taking a few short answer questions and play around with converting them to HOC MCQs. Some guidelines for blooming your MCQs are in the PDF of the workshop materials.

WARNING: This activity might stimulate a conversation about the learning outcomes in a given course, ways to achieve those learning outcomes and using formative assessments in the classroom that provide feedback to students.

Resources

I have attached the workshop materials as a PDF. It includes some excellent examples of the question types that are reviewed in this post. It’s inspiring to know that this is possible, yet totally daunting to consider the amount of work a full conversion would take. Advice from the workshop: start small, just do a bit at a time.

  1. Download sample HOC MCQs here!

  2. A more scholarly article: Biology in Bloom

  3. Guidelines developed by the University of Texas: http://tinyurl.com/boe6lky

  4. A manual developed by the University of Cape Town: http://tinyurl.com/9se2k7l