Many students think they know how to read a journal article. However, this often means scanning the title, and abstract, reading the introduction and discussion and perhaps perusing a few figures from the results. Few undergraduate students will critically question the methodology and conclusions of a journal article. This is not surprising given that students are overloaded with course work, employment, life-in-general and methods are thick and technical. It requires a certain degree of scholarship to fully appreciate scientific methodology.
However, critique of methodology is central to understanding a paper and ‘big picture’ issues. Moreover, it’s a skill that requires practice, thus early introduction in the undergraduate program will create opportunities for students to make repeated attempts. One barrier to early implementation is that at many papers use discipline-specific terminology and students in first and second year are not yet fluent in the terms.
Simply telling a student that methods are important is not sufficient. I think that students have to experience the importance in the context of their own understanding to develop the motivation to practice this skill. My primary goal for this case study was to create an ‘Ah Ha’ moment for students so that they would leave class with a thought experience that impressed them with the need for reviewing methodology. I kept the subject matter light, humorous, simple, accessible and jargon-free. In addition, I wanted the students to experience the ‘mistake’ of not reading the methods in a low stakes, safe setting. I decided to use French Fries as a topic and HSCI 212 (a large lecture class – 150 students) as guinea pigs. Here is what happened.
A brief pre-assignment was given to students to identify their starting point. This tool was also used to inform the post-assessment to determine whether or not the in-class case study was effective. Students were asked to select an article (shown below) from The Annals of Improbable Research. They worked through a list of questions aimed at determining their reading strategy and brought their notes to class. In class, students were asked about their section rankings (question 5) and on which section they had spent the majority of their time. As predicted, the introduction and discussion sections were at the top of the list, followed by results and methods in last place and mostly ignored.
1) “The Effect of Acute Increase in Urge to Void on Cognitive Function in Healthy Adults,” Matthew S. Lewis, Peter J. Snyder, Robert H. Pietrzak, David Darby, Robert A. Feldman, Paul T. Maruff, Neurology and Urodynamics, vol. 30, no. 1, January 2011, pp. 183-7.
2) “Dizziness in Discus Throwers is Related to Motion Sickness Generated While Spinning,” Philippe Perrin, Cyril Perrot, Dominique Deviterne, Bruno Ragaru and Herman Kingma, Acta Oto-laryngologica, vol. 120, no. 3, March 2000, pp. 390–5.
3) “Is a Sigh ‘Just a Sigh’? Sighs as Emotional Signals and Responses to a Difficult Task,” Karl Halvor Teigen, Scandinavian Journal of Psychology, vol. 49, no. 1, 2008, pp. 49–57.
4) Microbiological Laboratory Hazard of Bearded Men,” Manuel S. Barbeito, Charles T. Mathews, and Larry A. Taylor, Applied Microbiology, vol. 15, no. 4, July 1967, pp. 899–906.
Note: One student volunteered that she thought the article titles ‘were hilarious.’
1) What are the main sections of the paper?
2) Which sections did you read first?
3) Which sections were most informative?
4) In which sections did you spend the most time?
5) Rank the sections in order of importance.
6) If you were to review the article in a lot of depth, what strategies would you use to approach the material?
introduction => discussion => results
The case study was then used to demonstrate why the reading strategy used by the majority of students leads to poor understanding of study conclusions. I introduced a broad, slightly vague French Fry inspired ‘hypothesis’ that was to be tested by two separate groups (Group A and Group B):
Cooking potatoes in fat creates a more palatable product than cooking potatoes in water
Since most students are familiar with French Fries, they had a lot of pre-existing assumptions about that statement. Including that potatoes cooked in oil would taste better.
I then showed them ‘conclusions’ for the two studies:
Thus, in the context of the case, two research groups came to different conclusions even though they were testing the same problem. I then led the class through both studies using the class reading strategy (Introduction => Discussion => Results) to determine why these two groups had come to different conclusions. I had prepared a brief model sections for each study, along with a description of the role of each section in a paper.
States problem: Raw potatoes are unpalatable and starchy with an unpleasant crunch.
Discusses what has been done before: Previous studies have shown that throwing potatoes into the hot coals produces a softer inside but a charred outside (Caveman et al.).
Says why they are doing it: In this study we expanded on the work of Caveman et al. and explored different cooking mediums to determine whether cooked potatoes could be better than raw potatoes.
The class had a brief small group discussion and there was general recognition that the introduction section was not likely to reveal WHY two studies might come to different conclusions.
Summarize observations, support hypothesis?: Our observations did not support the hypothesis. We determined that cooking potato in water produced a product that peeled easily yielding a softer inside, whereas the oil-poached product was greasy to the palate and covered in grit.
Discuss reasons for the observations with supporting data: It is possible that boiling in water removed grit from surface due to the solubility of the medium.
Mention ways you will refine the work: Future studies will determine whether or not peeling the potato will produce a more superior product.
Summarize observations, support hypothesis?: Our observations supported the hypothesis. We determined that cooking potato in oil produced a satisfying and crunchy product. Those cooked in water had no remaining texture and were almost powdery.
Discuss reasons for the observations with supporting data: Our observations suggest that hot oil cooks out the water in the surface of the potato yielding a crisp outside and tender inside.
Mention ways you will refine the work: Future studies will determine whether double-frying techniques will produce a crisper product.
Students observed that there were some general explanations in the discussion that might lead to reasoning of ‘why’ two different groups came to different conclusions. They started to: infer that Group A and Group B might have taken different approaches, and observe reasons why they might have developed different conclusions. At this point, some students started to ask ‘how’.
Although students were starting to ask ‘how’, we went onto the results (Figure 1 & 2 for group A and B, respectively) section since this modeled the class reading strategy from the pre-assessment. Results from group A and group B (see below) were shown to the class. The graphs showed a ‘palatability index’ of potatoes cooked in fat or water. In comparing results from Group A and Group B, Group A showed higher palatability of potatoes cooked in Water whereas Group B showed higher palatability of potatoes cooked in oil.
At this point students started to ask about the definition of ‘palatability’, who were the taste testers, how were the potatoes cooked by group A vs group B. The class kept asking ‘how was X done?” There was a resounding echo of “How How How”.
At this point it became very clear to them that the methods are the ‘how’ and that the rest of the paper lacked context without understanding the methodology. Students really wanted to understand what was being done to the potatoes, who was doing the taste testing, and what was meant by the metric of ‘palatability’.
We then reviewed the methodology used by group A and B. I had developed vastly different (and slightly unrealistic) methods for cooking potatoes. The methodology clearly demonstrated the reasoning behind the different conclusions. At this point, the class felt it was fairly obvious that a clear understanding of methodology was required to understand the results and conclusion of the potato study. They had now experienced why it mattered so much. This is exactly what I was aiming for.
At the end of the class, there was general consensus that it was very obvious to look at the methods for this study. I then left them with this question: If this information is important for cooking potatoes, how much MORE important is this going to be for a studies that test an anti-HIV-1 drug or a malaria vaccine?
In a post-assessment exercise, students provided feedback on what they learned in the exercise, by reflecting on their previous strategy and how they would change their approach as a result of the exercise. There was qualitative feedback questions in which students described whether or not they learned from the exercise, and quantitative data in which they scored their perceived value of the approach.
As shown in Figure –, 55% scored the exercise value as 8 or higher out of ten, whereas 11 percent of students scored the exercise value as 5 or lower. However, perceived value, did not correlate to whether or not the exercise actually worked. In spite of a low perceived value (or they thought it was silly) many of the students who gave the exercise a poor rating, still learned from it and stated that they would change their reading approaches. In fact, Figure — indicates that 80% of students said that they would change their reading strategies as a result of the exercise; 17% of students suggested that they were already following a methods-centered approach.
- Several commented on the fact that they learned something they could actually apply to other classes (I valued this comment the most!)
|- Would really value this exercise in first year|
|- Some people really enjoyed the subject|
|- Some people thought the subject was silly|
|- Some people just wanted to be ‘told’ that the methods were important (not effective)|
|- Several people thought the exercise could be shorter (true, next time…)|
|- Some suggest more practical advice like a demo of taking a paper in part (This would be a logical follow-up)|
Although, this exercise is a simplistic model of how we break things down scientific methodology, I think there is value in the simplicity since students experienced the ‘knowing of why’ in personal context. In addition, it provided an opportunity to ‘experience the mistake’ in a safe and comfortable setting. In future iterations, I will follow it up with a couple of real studies that come to different conclusions on an important health issue. The exercise requires some refinement, but overall I achieved my goal for teaching and I will continue to explore this approach in the future.
By N. van Houten