Readings
1. Information about the Intrinsic Motivation Inventory or Questionnaire for measuring Enjoyment
2. Bentley et al. Note that this paper discusses the Cued Recall Debrief Evaluation method. The paper also discusses measuring participant’s physiology (e.g., heart rate). You will not be using physiological measures in this assignment.
Resources
- Cued Recall Debrief form
- IMI Questionnaire
- Scoring for IMI Questionnaire
- Inter-rater reliability example
- Marking Guide
The Game
You and your team are going to use a field study evaluate four people’s subjective emotional and affective experiences using a whole body interface style video game. Your goal is to find out how the participants felt during and after game play. You will use this information to suggest improvements to the design of two aspects of the interface (see redesign goals below). Choose a “full-body” game for one of the following platforms: Nintendo Wii, Sony PlayStation EyeToy or Eye, Microsoft Kinect.
The Location
Ideally you will run this as a field study. This means that you should evaluate the players in any environment where you might naturally find them playing these kinds of games. For example, you could host a video game party with several friends and then conduct the evaluation in someone’s living or party room. If this is not possible, you can run the study as a lab style study in any space where you get set up the equipment (e.g., bookable room at SFU). In the latter case you should be explicit in your report that you ran a laboratory style study not a field study.
The Redesign Goals
The goals for this assignment are to explore player’s emotional and affective responses to playing “full body” video games in order to offer suggestions about what works and what needs redesign. The focus is to understand and improve two aspects of the interface design: controller mappings and display layout. For each, you should report on key features to keep and key features to redesign with redesign suggestions.
1. Controller Actions to Game Response Mappings – how can you improve the design of the mappings between how you move the game controller and the corresponding game response? Use information from the data about where people feel that the controls are easy to use and enjoyable and/or frustrated and annoyed to argue for your redesign suggestions.
2. Screen Layout - how can you improve the design of the screen layout to improve game play including navigation and task completion? Screen layout includes things like status bars, navigational features, scores, world view etc. Use information from the data about where people feel that the layout design is easy to understand and enjoyable and/or confusing or difficult to argue for your redesign suggestions.
The Evaluation Methods
You will be evaluating each of four participant’s affective experiences using two methods:
1. A simple post-play questionnaire called the Intrinsic Motivation Inventory (IMI) which measures the overall emotional experience of enjoyment of the game or activity.
2. The cued-recall debrief method is a first person, video-based method of reviewing and talking about the participant’s play session with the participant after the play session in order to elicit information about the short term affective states the player experienced during the game or activity. See paper above and notes on affect and emotion below. You must read the Bentley paper in order to understand how to use this method.
Background on Affect and Emotion
Much of our technology and technical evaluations revolve around enhancing either intelligence or productivity or both. However, much of our everyday lives revolve around our emotions. Increasingly it is becoming important to develop methods to evaluate people’s affective experiences.
Emotion can be defined as “the person’s overall feeling and can be influenced by (among others) context, past experience, recent experiences, personality, affect, and the net cognitive interpretation of these influences.” (from Bentley et al. article). The IMI questionnaire reflects the emotion of enjoyment and several predictors of enjoyment. That is the overall feelings of enjoyment, measured AFTER the play session.
Affect can be defined as “a short term, discrete, conscious subjective feeling that may have an influence upon a person’s overall emotion.” (from Bentley et al. article). The cued debrief recall method attempts to measure the short terms affective aspects (e.g., momentary frustration, joy, sense accomplishment) experienced DURING play, although the data is also collected after the session.
1. Things to prepare ahead of time
Participant selection: You are required to have four participants. You can use members of your team or recruit friends. For each session, you will also need evaluators. Evaluators should be members of your team who are NOT the participant for that session.
Equipment:
1. Full-body controller and game system. Wii or PS2 with EyeToy can be signed out for 24 hours from library reserve. You will need to hook this game station up to a display either on campus or in your home.
2. Game to go with game platform above.
3. You will need one or more video cameras (though you could use Camtasia to capture a first person view). Video cameras can be signed out from library. You can also use a smartphone camera but the quality will be less (in an industry setting you'd get a high quality camera).
4. Lighting kit optional.
Video Games: Your whole team should be familiar with the game and controller before you start this study. This is not a study about learning a new system. All team members who are participants should have played the game for at 5 minutes before the start of their session.
User Session Protocol: Be familiar with this protocol before you start. You will have four sessions each with one participant. Each session has the same protocol which has three parts. A session will look like this:
1. PLAY: One participant will play a game (e.g., one level). Their experience will be video taped in first person – the video should capture what the participant is seeing. That is you will need a camera directly behind them or mounted on their head to capture what they see while they play.
2. IMI QUESTIONNAIRE: After they are done playing, the participant is to fill out the IMI questionnaire using the enjoyment, perceived competence, perceived choice and pressure/tension subscales.
3. CUED RECALL DEBRIEF: Immediately after the questionnaire part, is the review or debriefing part. One team evaluator and the participant will sit together and review the video of the play session and “debrief” or talk about what the participant was feeling during the play session. The participant will talk about this as they watch the video of the session. You may need to use prompts (like in think aloud) to encourage them to talk about what they were feeling. The debrief part should also be video or audio taped. This means you will need to find a quiet place where you can replay your video and discuss the session. Note that you need one camera to play the video (or alternative play back display) and a second camera to video tape the interview (focusing on audio).
When you are done the sessions you will have four videos of participants playing (shot in first person) and four videos of participants talking about their play (eight videos total). You are not done yet. You will still need to analyze all your data.
Game Details: Determine in advance what game and exactly what part of the game you will have participants play. The play session should begin at the beginning of the game or level and end at a logical end point (e.g., completion of a task, level or the game). Play sessions should be between 5 and 8 minutes. You should choose the same game segment for all participants.
Video Game Equipment Set Up: For the play session you will need to set up the video camera to capture WHAT THE PLAYER SEES as per the instructions in the Bentley paper.
For the debriefing session you will need a video camera (and display if you want) to play back the play session video.
You will also need to set up the second video camera to capture BOTH the participant and the evaluator. Be sure you can clearly hear the participant speaking. Test this. If you don’t capture clear audio you will not be able to analyze your data.
Post-test IMI Enjoyment Questionnaire: Print out four copies of the questionnaire which you will find at the end of this assignment.
Cued Recall Debriefing Script: You will need a short script for the evaluator to use in the debriefing. You should list what kinds of things the evaluator will say to the participant to prompt affective comments in the debrief part of the session. Do not refer explicitly to feelings or affect words in your cues. This would bias the data. Ask them about their experience in general terms. For example, “What are you experiencing at this point?” NOT -> “Are you feeling frustrated here?” Do not put words into their mouths.
Data Analysis: Be familiar with how you will analyze your data before you start. See section 4 below.
Print out instructions for analyzing the IMI Questionnaire sheet as below.
Print out six Cued Recall Debrief Affect Comment Data sheets before you start to analyze the cued recall debrief video data. A sample sheet can be found at the end of the assignment.
2. The Game Play Session
Get the video camera set up. Remember to video tape the participant’s first person point of view or what they see while playing the video game or activity.
Make sure the participant has 5 minutes to familiarize with the game before the session starts.
Start the video camera. Start the game. Ask participant to play the game.
Allow the participant to play until the task or level is complete. Tell them in advance when you will stop them. Stop them when they are done.
3. Post-Play IMI Enjoyment Questionnaire
After they are done playing, ask the participant is to fill out the IMI questionnaire. Remind them of how the rating scale works.
4. The Cued Recall Debriefing Session
As soon as the questionnaire is complete do the debriefing session. You need to have two video cameras. One video camera (or playback with display) will be used to replay the first person view game session. You can use a larger display if you like.
Use the second camera to video tape the debriefing session. Make sure you can hear the participant in the video. Test this first.
Have the evaluator sit down with the participant and review the video session of the play session. Encourage the participant to speak freely regarding all aspects of their interaction during play session. You will also want to draw their attention to aspects of the controller mappings and screen layout that impacted their experience with the game (either positively or negatively). Do not censor them in any way. You are capturing this audio data on video for further analysis.
See the Bentley et al. paper page 4-5 for a more detailed description of the method.
5. Data Analysis
5.1. IMI Enjoyment questionnaire:
Score the questionnaire as given in the instructions below. You will need to use some simple descriptive statistics to do this (mean and standard deviation). Refer to web site for clarification
Notes: The Intrinsic Motivation Inventory (IMI) is a multidimensional measurement device intended to assess participants’ subjective experience related to a target activity in laboratory (or contrived field) experiments. The questionnaire assesses participants’ interest/enjoyment, perceived competence, perceived choice and felt pressure and tension while performing a given activity, thus yielding four subscale scores for our purposes.
The interest/enjoyment subscale is considered the self-report measure of intrinsic motivation. Although the overall questionnaire is called the Intrinsic Motivation Inventory, it is only the one subscale that assesses intrinsic motivation, per se. As a result, the interest/enjoyment subscale often has more items on it that do the other subscales.
The perceived choice and perceived competence concepts are theorized to be positive predictors of both self-report and behavioral measures of intrinsic motivation.
Pressure/tension is theorized to be a negative predictor of intrinsic motivation.
5.2. Affective Content of Cued Recall Debriefing Session:
After all four of the play-debrief sessions are finished, you will analyze the four debrief video/audios (i.e., not the video of playing). To do this you need two evaluators for each of the four sessions. You should not evaluate a debrief session in which you were the player.
Two evaluators will each review each debriefing session to listen for affect comments. They will record these comments on a data sheet. Thus, you will need the eight copies of this sheet you printed out (2 per participant).
Have two evaluators individually (i.e., separately) review each participant’s debriefing session. Focus on what the participant said (i.e., audio only). Your goal is to write down (on the data sheet) any comments the participants makes which contain affective words or contain affective expressions. If these words or phrases are related to controller mapping or screen layout design you should add a note about this.
Separate out positive, negative and neutral comments as per the examples in the sheet. Sort comments using categories for overall experience, controller mappings, screen layout and other interface or game features. Refer to the Bentley et al. paper on page 4-5 for further details.
Group together similar types of responses and count how many times each was said. Examples:
- FUN: I’m laughing here. This was fun. That’s funny.
- ENJOYMENT: I really like this. This is great.
- FRUSTRATION: I’m frustrated. I can’t make my character jump [note: this as
about controller mapping]. This was annoying. This isn’t working like it should. I can’t find the target. [note: this is about screen layout]
Count (add up) the total number of positive, neutral and negative comments.
Classify into categories for comments about overall experience, controller mapping design, screen layout design and other game or interface features.
For each of the four participant’s debriefing sessions, you should have two affect comment data sheets, one by each evaluator – for a total of eight sheets.
5.3. Inter-rater Reliability on Cued Recall Debrief Affect Comment Coding
After you fill in all eight sheets, compare the two sheets for each session. For each session/participant, compare the results of the two evaluators/raters. Did you find the same kinds of affective statements? Did you find the same number of for each kind? Did you find the same overall number of positive, neutral and negative affect comments?
You also need to calculate the inter-rater reliability value for each session. Use the excel example file posted with this assignment to do so.
The inter-rater reliability or R value indicates how similar the results from the two evaluators are. For each session your R should be better than .90. If it is not, have the two evaluators work together to reach a better consensus.
You will have four inter-rater reliability values, one for each session.
6. The Deliverables
Hand in 6.1, 6.2 and all data sheets and Excel sheets. See the Marking Guide for how you can structure the report and what we are looking for. It basically shows the information below.
6.1 Report: Include a title page with the course, assignment # and title, instructor’s name, TA’s name, your team member’s names and student IDs and due date. Also use headers and footers. Use 12 or 10 point font either times roman, arial or verdana.
Answer the following questions. Your total report length not including the title page should be 5 to 7 pages single spaced.
1. What does your analysis of the IMI/Enjoyment questionnaire data tell you about each of the player’s experiences with game play in terms of how the controller mappings and the screen layout affected their experiences? Consider the enjoyment subscale and the perceived confidence subscales.
2. What does the affective content analysis of participant’s cued recall debriefing session tell you about each of the player’s experiences with game play in terms of how the controller mappings and the screen layout affected their experiences?
3. For each participant, were there any parallels or inconsistencies between the questionnaire and the cued recall debrief results?
4. What do you think the issues, challenges and strengths of using the cued recall debrief method versus the questionnaire are?
5. Controller-Mapping -- Describe 2-3 key aspects of the controller mapping design that you think works (backing up your choices with data) and 2-3 aspects of the controller mapping design that you think could be improved (backing up your choices with data) with suggestions for improvement. Be sure to relate your design choices and suggestions for improvement on your data not to your own personal opinions.
6. Screen Layout -- Describe 2-3 key aspects of the screen layout design that you think works (backing up your choices with data) and 2-3 aspects of the screen layout design that you think could be improved (backing up your choices with data) with suggestions for improvement. Be sure to relate your design choices and suggestions for improvement on your data not to your own personal opinions.
6.2 Video: Compile a video summary of the highlights, findings and redesign suggestions of your evaluation of user’s affective experiences focusing on controller mapping and screen layout design. You should mention this focus in your titling or voice over to set the scope and goals for the evaluation. It is to be no more than 2-4 minutes long.
Include the following information:
A title page with the course, assignment #, assignment title “Evaluating User’s Emotional and Affective Experiences Playing with Nintendo Wii (or Sony Eye Toy)”, course name, instructor’s name, TA’s name, your team member’s names.
A narrated and/or text introduction which explains the goals and methods of the assignment including a brief description of the Intrinsic Motivation questionnaire and cued recall debriefing methods.
A narrated summary of your key findings using segments of your play sessions AND debriefing sessions to illustrate key points and support your findings. This should focus on controller mapping and screen layout design and be based on your responses to questions 5 and 6 above (in 6.1).
A narrated overview of your suggestions for redesign of the controller mappings and screen layout using video footage to support your suggestions. Include: proper titling, narration, captions and a brief credits section at the end.