3 C ASE S TUDIES
3.7 LightGame (VII)
3.7.4 Evaluation I
84
shared instantly with speech output, and after each exercise set, a summation grade of one, two, or three stars is displayed with a projector and projection screen or shown on a laptop screen. The optional projector was sometimes also used to display signature images of the Light and the Shadow characters or images of animals present in the story.
Physically, the setup consists of a laptop PC, a pair of active speakers, a moving head lighting fixture, optionally fixed lights, a projector and a projection screen, and either a PlayStation Move controller® or a regular mouse for input. This economical setup can be assembled for a couple thousand euros (at the time of printing).
3.7.3 Challenges
There were several characteristics that made this user experience evaluation case study challenging. First, having schoolchildren as the main user group posed issues, such as how much and what can be asked from the children while still gaining useful information. In addition, how should the questions or statements be phrased so the children will understand them but they do not become too simple as regards the evaluation? Another issue that needed consideration was having yet another user group involved, the teachers. We were interested in teachers’ educational, usefulness, motivational, and practical points of view, but how would they perceive the overall picture, interpret children’s reactions, and report them based on our inquiries? Second, the school context raised challenges considering the timing and extent of the questionnaires: what to include in the questionnaires and when to fill them in without cutting down the time for actual physical activity but still gathering important information. Moreover, there were some very practical issues considering the physical environments, such as physical space limitations and how to make the space dark enough for the lights to be effective.
3.7.4 Evaluation I
This user experience evaluation was conducted in a school environment with altogether 110 schoolchildren. User experiences were gathered mainly with statement-based questionnaires.
Context
The case concerned a school environment, and the physical environment of the evaluation sessions was a school’s small gym. The equipment described in Section 3.7.2 was set up, and the windows were covered to make the space dark enough. At least the participating group of schoolchildren, the group’s teacher, and one or more researchers were present at the evaluation sessions. Questionnaires were filled in at the scene or afterwards in a classroom.
……………
85 Participants
We had a total of 110 participants (56 girls, 54 boys), aged 6–11 years (mean=9.11, SD=1.11). About 76 percent of the respondents (n=106) reported they play videogames. About 97 percent (n=107) even stated they like physical exercise, and about 82 percent of the respondents (n=104) exercised in their free time. Furthermore, about 43 percent (n=108) reported they practice some team sport (e.g., floorball, football, ice hockey). The participants did not get any compensation for their participation.
Procedure
The user evaluation was conducted as one-time evaluation sessions lasting about an hour. Each session, a group of schoolchildren played the game following the story and instructions. The session procedure is presented in Table 13.
Evaluation phase Content
Usage • "Playing the game,” i.e., moving and performing tasks according to the instructions given in the game
After the usage • Experiences questionnaire (incl. background information) Table 13. The procedure of the LightGame case’s (VII) Evaluation I.
In the beginning of the sessions, the children were asked to form a circle around the trolley, and the game was briefly introduced by a researcher.
Then the game itself was started, and the researcher controlled the game and rated the performance of the group after sections. The experiences questionnaires were filled in after the evaluation sessions in the gym or in another classroom. Some of the sessions were also videotaped and observed by our project partners. They also conducted some interviews for the teachers.
Subjective data collection
Background information. Background information was gathered together with the experiences in the beginning of the experiences questionnaire. We inquired about the participants’ age, gender, and whether they play videogames, like physical exercise, exercise in their free time, and practice some team sport (e.g., floorball, football, ice hockey). The liking of physical exercise and exercising in their free time were asked to see whether the participants' general attitude towards exercising would have an effect on their experiences about the game. Practicing team sports was requested because previous experience with group activities might have an effect on how the participants experience playing the game as part of the group.
User experiences. In this evaluation, we gathered user experiences with a questionnaire consisting of 13 experience statements, an overall rating of the game, and three open-ended questions. The questionnaire utilized ideas used in the testing of the initial version of the game. For the experience
……………
86
statements, the original questionnaire used a scale consisting of happy, neutral, and sad smiley faces. We found some of the answers given on such a scale hard to interpret. Thus, here, we wanted to minimize the need for interpretation—both for us and the children—and used the options “Yes,”
“No,” and “I don’t know” for answering. The questionnaire included the following user experience statements:
1. Playing was hard.
2. I would like to move this way again.
3. Exercising was now more pleasant than usually on PE classes.
4. I understood the instructions of the exercise tasks well.
5. I understood the speech well.
6. The speech voice sounded pleasant.
7. The music and the voices of the game were compelling.
8. The lights of the game were compelling.
9. I found the game irritating.
10.The story of the game was interesting.
11. The exercise tasks were too easy.
12. I could move with my own style.
13.I felt like an outsider in the game.
In addition to these, the overall liking of the game was inquired about by asking, “How much did you like the game as a whole?,” which was answered on a five-step smiley face scale (see Figure 27). Furthermore, the open-ended questions were the beginnings of sentences to be completed: “The best in the game was...,” “The worst in the game was...,” and “The game would be more interesting if...”
Figure 27. The smiley face scale used for rating the overall liking of the game.
Supportive, objective data collection
Our project partners made some videorecordings and observations during the evaluation sessions. However, those data were not included in the analysis done for this dissertation.
Results
The statement results are presented in Table 14. Of the respondents, 78 percent (n=106) stated they would like to move this way again (statement 2), and about 72 percent (n=109) thought exercising was now more pleasant than usually on PE classes (statement 3). Although the music and the voices of the game were found compelling by the majority (65%) of the participants (statement 7), the lights seem to have been a success, as 78 percent of the participants reported them to have been compelling (statement 8). Only 6 percent of the respondents (n=107) found the game irritating (statement 9),
……………
87
and astonishingly, equally only 6 percent of the respondents (n=108) felt like an outsider in the game (statement 13).
Statement Yes No I don’t
know
Number of respondents
(n) freq. % freq. % freq. %
1. Playing was hard. 5 5 84 78 19 18 108
2. I would like to move this
way again. 83 78 11 10 12 11 106
3. Exercising was now more pleasant than usually on PE
classes. 79 72 15 14 15 14 109
4. I understood the instructions
of the exercise tasks well. 80 76 8 8 17 16 105
5. I understood the speech well. 83 76 17 16 9 8 109 6. The speech voice sounded
pleasant. 69 63 16 15 24 22 109
7. The music and the voices of
the game were compelling. 72 65 17 15 21 19 110
8. The lights of the game were
compelling. 86 78 10 9 14 13 110
9. I found the game irritating. 6 6 96 90 5 5 107 10. The story of the game was
interesting. 79 75 15 14 12 11 106
11. The exercise tasks were too
easy. 30 28 47 44 31 29 108
12. I could move with my own
style. 53 49 38 35 18 17 109
13. I felt like an outsider in the
game. 7 6 88 81 13 12 108
Table 14. User experiences per statement from the LightGame case’s (VII) Evaluation I. The most promising results are highlighted with grey.
Overall, the participants liked the game very much. Figure 28 shows the results for the question, “How much did you like the game as a whole?” Of the respondents, 61 percent (n=107) answered the question with the extremely happy face, while 23 percent selected the little happy face. No one gave the game the worst rating, i.e., the extremely sad face.
……………
88
Figure 28. A boxplot presentation of the results (n=107) about liking the game overall (“statement” 14) in the LightGame case’s (VII) Evaluation I.
The children’s answers to the open questions maintain the same positive attitude toward the game. About 13 percent of the respondents (n=109) stated that everything was the best in the game, while 55 percent (n=110) reported there was nothing worst in the game, either by explicitly writing that or answering with a ”–.“ The Shadow was clearly the best thing in the game, as it was mentioned in 37 percent of the answers. Physical exercise, e.g., jumping, running, or moving in general, was included in 14 percent of the answers. Other positive things reported by the children were the animals and the sounds, for example. According to the open question data, there was no clear negative that would have been mentioned by many children. Animals were mentioned by 9 percent of the respondents, but other answers were scattered. Negatives reported by a few participants at the most concerned, e.g., the game being too easy or too difficult, or the lights being too bright. To make the game more interesting, 16 percent (n=110) of the respondents would make it last longer. Fifteen percent suggested improvements that dealt with the Shadow somehow, for instance, that the Shadow would see the players or that it would have been present more. Eleven percent wished for improvements related somehow to the atmosphere, such as adding suspense, and 7 percent wanted to increase the difficulty of the game.