Review. . .
* www.useit.com/papers/heuristic/
Extra: Preece et al. Interaction Design, page 343 for an introduction to the method and section 13.4 for a basic description of the method. Read the box: Dilemma on page 411 as well.
Definition: Heuristic = an experience-based technique that helps in problem solving, learning or discovery. A heuristic method is on that is used to rapidly come to a solution. Heuristics are "rules of thumb". In design evaluation, heuristics are usability principles, derived from experience, which aim to ensure that a particular style of interface is usable for a particular population of users. There are more than one set of heuristics used in usability studies. We will use some of Jakob Nielsen's heuristics for this assignment.
Introduction: There are several ways to evaluate a computer system and determine if it is any good for a person to use. Most developers simply create the system, try it out themselves until they are satisfied with it, and then release it to the user audience. The result is usually a product that people have problems with.
One of the most common ways to evaluate a human computer interface is a usability study which evaluates how usable an interface is, and what areas need to be changed. Usability studies come can be done with different methods. One type of evaluation method is Analytical Methods. Analytical methods are typically involve one or more experts (e.g., cognitive psychologist, interaction designers, usability engineers) who use a systematic set of steps to review the interface, looking for strengths, limitations and problems. This approach does not involve endusers.
These methods are called analytical because an expert analyzes the interface. A heuristic evaluation is one kind of analytical method. There are other kinds of analytical methods. For example, a cognitive walkthrough or an interface critique (more on this in later lectures…). A heuristic evaluation is also often called an inspection method, because experts inspect the interface. It is also a kind of discount method because it is simple, fast and inexpensive to implement.
In this assignment you will use an analytical inspection method to conduct a usability study in which you will act as the experts who will evaluate the system using heuristics. Heuristics are sets of general guidelines related to how an interface should be designed. There are different heuristics depending what the focus of the usability study is. You will be using a subset of Jakob Nielsen's heuristics. A heuristic evaluation then involves asking experts to compare the interface to a particular set of heuristics.
Project Overview
Imagine that you work for Usability Inc., a consulting firm that specializes in evaluating web interfaces. You and your team have been contracted to do a quick usability study of the the new SIAT web site using the heuristic evaluation method. Your deliverable will be a report written for Dr. Marek Hatala, Director of the School of Interactive Arts and Technology, SFU.
See: SIAT Web Site Brief
The new SIAT web site can be found here during its testing phase. During the course of the assignment, it may migrate to it's permanent location here so don't be surprised if the previous URL stops working.
Your job (description of your role on project): You have been hired by Dr. Hatala. Your job is to do a discount usability study of the proposed SIAT web site based on a set of Nielsen’s heuristics deemed important to Dr. Hatala. Your report will include descriptions of how you went about looking for design problems, what problems you saw, and what changes you recommend. Depending upon how convincing and articulate you are, Dr. Hatala may use your report to influence a redesign of the website for SIAT.
The assignment will be done in teams and will follow the major steps below:
1. Planning:
Expert selection: As students in IAT 432 I am assuming you have some level of expertise with interaction design (at least some team members will). Each member of your team will be an expert and will conduct an evaluation individually. Once each member has conducted the evaluation, you will compare results, discuss disagreements and resolve conflicts to present a unified consensus on the major problems in the site.
If you were in a situation where you were the only expert on a team, then you would try and recruit other interaction design professional to conduct the evaluation. For our purposes, I want you each to have the opportunity to do the evaluation.
(a) Tasks: A heuristic evaluation requires one or more experts to go through the interface. The quality of the evaluation is improved it the experts use a predefined set of mock tasks to direct their inspection. These are often critical or typical tasks. Usually, it would be your job to work with the client to finalize a set of tasks that you as the experts will try to perform. These tasks should be realistic ones that typical users would try to do with the system! But how do you discover what those typical tasks are? In this case, you have been given a list of 15 tasks in the SIAT Web briefing. Part of your job is to narrow down this list to a manageable number. From the list of tasks given, choose five that you think are most typical or critical and use these for your evaluation. Be sure to make your choice based on the redevelopment goals set out for the SIAT web site, rather than your own personal experience as an SFU student. You will be asked to justify your task decisions in the final report.
(b) Heuristics: For a heuristic evaluation, you need to choose which heuristics are relevant for the client and the particular tasks chosen. You will use Nielsen's heuristics.
(c Data Sheets
You will need to create a data sheet for each expert evaluator. You should have three or four data sheets depending on the number of people in your team. The data sheet should be formatted along the lines of the example given below:
Project:
Evaluator:
Task & specific location in task
Heuristic broken
Usability defect description
Evaluator’s comments regarding defect
e.g., Task 1: finding the semester for New Media Images course. Visibility of System Status - System seems suspended while system searches database & no wait message is displayed; User should be alerted that system is active and searching …
You should also prepare a single data summary sheet as shown below. This is used to collate the findings, assign "severity" ratings based on the severity rating rubric described by Nielsen here. Resolve and document conflicts between evaluators and suggest recommendations.
Project:
Evaluators:
Date:
Usability defect description Found by Evaluators: Evaluator’s comments regarding defect
Severity Rating Recommendations
Task 1: System seems suspended while system searches database & no wait message is displayed Smith, Jones, Leung. User should be alerted that system is active and searching … 3 Add a system status message for waits longer than 5 seconds.
2. Doing the Heuristic Evaluation
Individual Inspections: Once you have done all the preparation steps you are ready to evaluate, have all team members individually perform a heuristic evaluation on the assigned web site using the tasks you determined above. It is a good idea for each expert to run through the tasks twice. In the first case, run through each task so you can familiarize yourself with the interface and tasks. In the second run through of each task, you are using the heuristics to look for problems. Use the task list and heuristics checklist you developed, and the data sheets you prepared (see above).
Team Summary of Problems: Once you have all individually done the heuristic evaluation meet as a team and review your findings. You may want to use stickies to group together common findings under heuristics. Summarize your findings on your single data summary sheet. Include severity ratings. Be sure to compare findings from each of you and adjust if necessary. Brainstorm recommendations for each problem. Add this to your summary sheet.
3. Reporting
Your write-up (report) should be oriented towards a management person who will make the major decisions on the interface and system changes. Here is a template for you to follow.
Executive Summary: A summary of the key problems of the system and some recommendations for improvement.
Section 1. Scenario: Give a very brief reminder (overview) on what the interface is and then explain the role (and objective) of your product evaluation team.
Section 2. Methodology: Explain what you did. Assume that your client knows what usability methods are and their purpose. Include the number of experts, the five tasks you chose, and the heuristics you used. You must provide a list of the five tasks that you chose, and explain why you’ve included them.
Section 3: Results & Interpretation: Summarize the findings of heuristic evaluation. Present findings in order of priority based on expert consensus and severity ratings. Identify common and important problems of the system. This should be more than a list of all the heuristics broken. Try to generalize problems that are related, and use examples to highlight them. Include annotated screen shots where appropriate.
Section 4: Suggested improvements: Describe five important changes that you would make to the design of the system, with explanation. Refer back to your heuristic evaluation summary results, the discussion on web design as covered in the text and your professional experience.
Note: You must stay within the style of interface presented. For example, your modification cannot suggest to convert a form fill-in system into a graphical map or change design constraints related to the SFU brand.
Appendix 1: Analysis of heuristic evaluation method for usability studies. For future usability studies, you will want to tell your product team what worked well and what did not in this usability study. Briefly summarize your experiences with the heuristic approach. Did your evaluators find the same problems? If not, why not? Would you recommend this approach? If so,why?
Appendix 2: Raw data. All original data sheets and data summary sheet from heuristic evaluation should be included here.