Purpose
This assignment will give you first hand experience at understanding how to evaluate an existing user interface. You will learn how to interact with study participants, develop usability study tasks, understand what problems users face with the interface, and understand what successes they have. You will also learn how to document your findings.
Overview
You have been asked by a company to evaluate the user interface for one of the systems listed below. You will perform a usability study with four participants and utilize the think-aloud and constructive interaction techniques. You will create a usability report detailing the findings of your study and recommendations for system redesign.
Focus your usability study on the main features of the site. Include the frequent tasks people need to do plus the infrequent yet important tasks. You should avoid going beyond this; do not test the entire user interface as this can easily make the assignment too broad in scope.
You will be evaluating Google+ Hangouts for this assignment. This is a multi-person system for chatting and meeting over video/audio connections. One of the team members should login to Google+ such that they can receive Hangout calls from the participant. The researcher should not provide help though from the remote location.
Topics that you should consider exploring as part of your tasks:
- How do you actually begin a Hangout with another person?
- How do you invite a third person to a hangout? how do you invite them before the call starts? how do you invite them after the call has already started? (You can test for this, but because it is only a single participant using the system, they should not expect an answer from the third party. Just test that they know how it could be done.)
- How can you start up an app with the remote person? or play a game? (e.g., How can you "Share a Story Before Bed"?) (Note: don't dive in to too much of the details of how third party apps work as this starts to test beyond Hangouts and more of the third-party apps)
- How can you text chat in a hangout?
- How can you share a file in a hangout?
- How can you ensure your camera and microphone are working properly and adjust their functionality, if needed?
- How can you start a shared drawing session?
- How can you share your screen and know what the other person is seeing?
- How can you rejoin a hangout if accidentally close your window?
- Etc.
You should use the interface and become familiar with how users can accomplish all of the tasks that you create. Each task MUST be more than the above simple questions. They should explicitly ask the user to do something to accomplish a goal.
Steps
1. Plan the Study: Plan out the usability study by creating:
- Pre-Test Questionnaire: include 8-10 questions that ask the user about their general demographics, computer skills, how often they have used the system in question, and what they think of the system in question
- Post-Test Questionnaire: include questions that ask the user how they felt about the usability of the system, what was easiest to do, what was hardest, what suggestions they would have for improving the interface
- Representative Tasks: create a set of realistic tasks that the users will do during the study. Ideally you would generate these from randomly sampling users about their tasks, or performing some design ethnography or interviews prior to the study. It is fine to generate some tasks based on your own experiences with the software though (at least for the assignment). So try it out, play with it, and then create some realistic user tasks. Make sure these vary from common tasks to infrequent yet very important ones. Create 5-7 different tasks.
2. Pilot Study: Try out your usability study with someone before you actually do it with real participants. This is important so you are sure your tasks and questionnaires make sense, you can run the study in less than one hour, and that users won't become extremely frustrated.
3. Find Participants: Find family members or friends who are willing to be participants in your usability study. Try to choose a range of users (e.g., novices, experts, young, old).
4. Perform the Study: Perform the usability study by first having the participant complete the pre-test questionnaire. Have two participants perform the tasks using the think aloud protocol. Have the remaining two participants perform the tasks using the constructive interaction protocol. Ensure that one participant will act as the novice and the other as the coach. It works best if the novice is actually a novice and the coach is an experienced user. Otherwise the coach will just do everything while the novice watches.
Throughout the study, try to get an understanding of what the user's conceptual model is of the system. It is a good idea to ask users about their conceptual model when they first see the interface.
While the experimenter should not help the subject with the task, there are a few exceptions to this rule. If a subject has problems getting started, record the problems and give them a hint to get going. This is OK, because if they can't get started, they will not be able to do the tasks! If a subject cannot complete a particular task after a reasonable amount of time, tell them to stop and start them on the next task. Or, give them a hint if they cannot overcome some conceptual problem necessary to trying out other parts of the system. Again, record all problems.
Collect data using one or more of the observational methods discussed in class (e.g., paper notes, audio, video).
After completing the tasks, have the participant complete the post-test questionnaire. Conduct a short retrospective interview to understand where the participant feels they had the most problems with the interface.
5. Analysis: Analyze your findings to see what usability problems are common amongst users. You may want to use affinity diagramming. Talk with your TA about this process in labs.
6. Report: Create a report that details your findings. This should include detailing the core problems people had with the interface and suggestions for redesign (see below for more detail).
Deliverables
You will provide a detailed report (roughly of 5-10 pages in length, single spaced + appendices). It should include the following sections:
- Introduction: describe the situation you are studying and why
Methodology: describe your study methods
- explain that you performed a usability study and describe your study steps
- describe your participants' demographics briefly
Observations: describe your observations and where appropriate use data to articulate what you saw (e.g., participant quotes, interview data, questionnaire data, screen captures)
Interpretations: describe the strengths and weaknesses of the system. This should be more than a simple checklist. Try to generalize them when all possible (e.g., if each screen is difficult to navigate through, then rather than describing each screen as having a problem, describe this as a general navigation issue with the user interface).
Suggested Improvements: describe five key improvements that could be made to improve the interface. Don't provide nit-picky improvements (e.g., I would change the button color to blue). Try to provide high level improvements (e.g., The buttons are not easy to see so I would improve the color contrast in the interface in several ways...)
Conclusion: summarize briefly and conclude elegantly
Appendix 1: include a comparison of the think-aloud method to the constructive interaction method. Which worked better? What were the faults and successes of each? Which would you use in the future and why?
Appendix 2: include all raw data (e.g., handwritten notes regardless of how messy)