Evaluating Stories: The Seven Criteria

The students used seven criteria adapted from Project Censored-USA to evaluate and select the stories that proceeded to the next stage. To be considered an 'under-reported story' for PCC, each story must concern a subject that is of interest to a majority of Canadians and has not received sufficient exposure and coverage in the mainstream national news media. The story must have potential effects of major significance and have received little coverage (searches through indices such as CBCA verify the volume of coverage). The story must present a clear, easily understandable concept; it must be national or international in scope and timely. Finally, the exposure of the story should encourage both journalists and members of the general public to seek our more information about the story.

By applying these criteria to all the nominations, seventy percent were dropped. We archived the information about all rejected stories in the project's office (along with a summary, prepared by the student researcher, of the coverage it had received, and a concise statement of why the seminar felt the story was a 'no go'). The basic information from each story remains in the PCC computer database. By keeping all information gathered about all nominated stories, we retain the opportunity to update and revisit any story if we receive new information about it and to conduct future research on Canada's overall news agenda.

About 30 per cent of the nominated stories went to the next stage: the student responsible for a story prepared a one-page written synopsis which was distributed to the whole seminar. In producing a synopsis, the researcher often attempted to flesh out or up-date the story. They may have telephoned relevant individuals or organizations for more information about the story, searched for additional published information beyond that contained in the original file and they may have sought out opposing or competing viewpoints to the one presented in the story. Seminarians individually scored each story (based on the synopsis presented in class) on a 20-point scale based on the seven selection criteria. Specifically, up to 5 points were awarded for each of four key dimensions: significance; validity/verifiability/clarity; amount of coverage; and overall potential as a Top Ten story. In practice, the story scores tended to cluster within a fairly narrow range, but scores were useful as a flexible guide for the next and most challenging step - one which completed the students' involvement in the research process - a marathon session during which the seminar participants collectively selected the best under-reported stories to forward to the national panel of judges. At this point, the Windsor and Simon Fraser synopses were pooled. The SFU seminar, assisted by project founder Bill Doskoch who visited for the occasion, took responsibility for selecting the 18 stories which were sent to this year's panel.

The judging panel had been jointly selected by the co-directors, the student seminarians, and Bill Doskoch of the CAJ, with an eye to regional, gender and political balance. Of the 26 members on the panel, 19 responded, 16 of them before the deadline for calculating the final story ranking. Each judge was forwarded the 18 synopses, and was asked to decide whether or not each story met the project's selection criteria. Seventeen of the 18 stories received a 'thumbs up' from a majority of the judges. Then each judge ranked his/her top 10 selections in order. By assigning 10 points to each first place selection, 9 to second, etc., PCC co-directors calculated a score for each story, yielding a rank-order for the 17 stories. In the event of ties, the story selected by the larger number of judges was ranked higher.

For purposes of researching the amount of national news coverage each story received, we adopted, in principle, David Taras's definition (in his book The Newsmakers, p. 87) of the dominant English-language "big players that have the power to set the agenda among the Canadian media", namely, The Globe & Mail, Toronto Star, Financial Post, Maclean's, CBC Prime Time News, CTV National News, CBC radio news, Canadian Press/Broadcast News, and Southam wire service. In practice, only the first four media, with respect to headlines of major articles only, were covered by CBCA. Therefore, to achieve a more accurate assessment of coverage, those stories that made it to the synopsis stage were searched on the several versions of Infomart and Infoglobe to which PCC subscribed. These services provided the full text of articles in the first four abovementioned media, as well as several other major dailies. We would like to have undertaken more extensive full-text searches, but even with the educational discount supplied by these services, budgetary constraints forced us to limit our use of these on-line databases. Indeed, the costs of the Infomart service containing broadcast news transcripts proved to be prohibitive, so strictly speaking, our research of nominated stories was limited to the national print media.

[Return to main Methodology page]

NewsWatch homepage