125
5 Conclusions
The research done for this dissertation focused on the issue of how to evaluate user experience in practice. The research questions were:
• How to evaluate the user experience of interactive systems in challenging circumstances, i.e., context or user groups?
• How to apply known methods to create an appropriate evaluation approach for a specific user experience evaluation case?
These wide and general-level questions are answered through numerous details and summaries in this dissertation. As the main contribution, and as a broad, yet comprehensive answer to the first research question, I have proposed a process model for evaluating the user experience of interactive systems. The model is based on the findings of eight user evaluations conducted with real users outside of laboratories and the expertise gained through the research. The case studies have comprised seven interactive systems and a range of contexts and user groups, as well as new interaction techniques still not consistently used, and especially studied, in the field of human-technology interaction. Because of the varying circumstances, mainly case-by-case-designed user experience evaluation approaches have been required. To gain an appropriate evaluation approach for each evaluation case, it has been necessary to apply already existing methods and to bring in newly created elements and approaches. The thorough descriptions of the used evaluation approaches in the case studies answer the second research question.
After describing the starting point for my research in Chapter 1, I introduced the basics of user experience and its evaluation in Chapter 2.
There, I also defined “user experience” as I see it and presented two evaluation methods used, and partly created, within the work done for this
……………
126
dissertation. In Chapter 3, the seven user experience case studies were carefully reported. In two of the studies, the MediaCenter (I) and the SymbolChat (III) cases, the main challenges arose from the user group, the first involving users with visual impairments, and the latter, having users with intellectual disabilities as the target user group. Two of the studies were related to the context of work environment, although from very different fields: The DrillSimulator case (II) concerned the drilling industry, while the Dictator case (VI) studied utilizing speech recognition in the healthcare domain. In the EventExplorer (IV) and the EnergySolutions (V) cases, the evaluations were conducted in public environments, and extra challenges were posed from assessing experientiality, i.e., something beyond the English term “experience.” The LightGame case (VII) and its two evaluations induced challenges from both the context of the school environment and the user group of schoolchildren. Furthermore, the SymbolChat (III) and the LightGame (VII) cases demonstrated situations in which the system under evaluation may have, in fact, more than one user group. Based on the case studies and the practical work experience, in Chapter 4, I finally proposed the process model for evaluating the user experience of interactive systems.
To conclude, the main contributions of my work are:
• The process model for evaluating the user experience of interactive systems. The model is based on eight practical user experience evaluations with differing circumstances and their outcomes. The model provides guidelines and practical considerations concerning the whole evaluation life cycle. It can be used to guide practical user experience evaluations.
• User experience evaluation method development. The Experiential User Experience Evaluation Method (Section 2.2.3) was developed to assess the experiential user experience of interactive display systems in public environments. It is based on knowledge and approaches from two separate fields: the SUXES method (Turunen, Hakulinen, Melto, et al., 2009) from the field of human-technology interaction and the Experience Pyramid model (Tarssanen & Kylänen, 2006), a theoretical framework meant for designing, analyzing, and developing tourism products, in particular. The created evaluation method was used in two case studies.
• Applying user experience evaluation methods in varying evaluation cases. The SUXES method (Section 2.2.2) was strongly utilized in six case studies, and at least some elements of it were employed in an additional two studies. Applying the method varied from using only some of the SUXES statements in the questionnaires, or pursuing the idea of gathering both user expectations and experiences, to following the method as a whole, i.e., gathering user expectations
……………
127
and experiences at least on the set of the nine SUXES statements regarding even individual interaction techniques.
• Taking the context, the user group(s), and other evaluation circumstances into consideration. Overall, the case studies presented have differed substantially due to context or user group, as well as interaction techniques. I have demonstrated a variety of practical user experience evaluation cases and ways to take the circumstances into account in the evaluation design. The evaluation contexts have varied from people’s homes to public and work environments, while the user groups have ranged from people with disabilities, schoolchildren, to professionals in certain fields.
• Transparent reporting of user experience evaluations. I have reported eight user experience evaluations of seven interactive systems in detail, including all the information requested from the participants. Transparent sharing of the evaluation designs and results allows other researchers and practitioners to utilize the information in their own work, and thus, advances the development of practical user experience research.
The contribution of this dissertation is of high practical significance. The dissertation will especially benefit beginning user experience practitioners and young researchers by providing real-world examples of evaluation cases and a step-by-step process model for user experience evaluation that can be utilized as a guideline for practical work. The scientific significance of this work cannot be neglected, either. Strict methodological issues, such as a strong theoretical basis, validated methods, or statistically significant results—something studied by Wechsung (2014), for instance—are out of the scope of this dissertation. However, this work contributes to the research gap of how to evaluate user experience in practice. The practical evaluation work is still a crucial part of the academic research. This dissertation challenges academic researchers to share their evaluation designs, data collection methods, and results transparently with the human-technology interaction community.
Evaluation methods could be more systematically developed, and ultimately also validated, if knowledge and practical considerations would be distributed more openly. For example, a research group focusing on interactive displays operated with hand gestures runs a user evaluation with only 10 participants because of their limited resources. They gather user experiences with a self-constructed questionnaire consisting of 15 statements and a couple of open-ended questions. They report the collected data, their findings, and practical considerations on what went well and what seemed to be the pitfalls—also considering the subjective data collection. Then, another research group focusing on the same matter, but having largely better resources, finds the first article. They notice that the first group has actually used some interesting statements and approaches
……………
128
that they themselves had not thought of. They decide to combine their previously used approaches and questionnaire with the ones presented in the article. Then they run a large-scale user evaluation with 100 participants and analyze the results. The results reveal clear clusters in the answers of different statements. Finally, they report the evaluation procedure, data collection, and their findings in detail. A third research group finds this article, decides to operationalize the clusters into statements, and compares the results gained with the whole set of statements against the set of cluster-statements. They report their research transparently to the community, and again, someone utilizes the knowledge in their own work, and so on.
Ultimately, the original, self-constructed approach has gone through an enormous number of developmental iterations and starts to produce consistent and truly beneficial results. Having gone through critical comparative studies, it has become a method commonly acknowledged and approved within the scientific community.
It is rather hypocritical to demand using validated user experience evaluation methods in the still constantly expanding world of interactive systems. One cannot use such methods if they do not exist. Because of varying systems and evaluation circumstances, and the lack of readily suitable and applicable evaluation methods, researchers may have no other option than to create evaluation approaches of their own. Forcing existing methods to certain evaluation circumstances just because they are validated, and thus, treated as acceptable, may be fatal and lead to the absence of any truly useful results. Väänänen-Vainio-Mattila, Olsson, and Häkkilä (2015) conducted a literature review on empirical user experience studies related to ubiquitous computing systems. They found out that the methods and approaches used in studies are rather lightweight and, in fact, do not enable deep understanding of the experiences, which would be important to develop the systems further. It would be crucial to share knowledge achieved through different evaluations. Then, information and methods from separate, but similar kinds of, evaluations could be combined and the methods developed and eventually also validated. The shortage of empirical user experience research is raised (e.g., Hassenzahl & Tractinsky, 2006; Vermeeren et al., 2010; Bargas-Avila & Hornbæk, 2011). However, I believe that empirical user experience research does occur, but is just not reported as openly as it should be—perhaps due to the unwritten requirement of utilizing validated evaluation methods.
The presented case studies and approaches serve as a starting point for evaluations in similar circumstances, but they need further investigation and improvements. In my future work, I will hopefully be able to systematically develop further the evaluation approaches presented here.
An ideal outcome would be to have fixed, validated user experience evaluation methods for interactive systems with varying purposes, use contexts, and target user groups. The methods should, however, allow
……………
129
customization, such as including or excluding certain statements or elements based on the circumstances existing in the evaluation case.
Modifying the methods should be thoroughly studied and clearly instructed in the end. Developing such methods would require years of work from several persons and is out of the scope of what one person can do alone. A feasible way to accomplish these kinds of evaluation methods would be to study the individual elements step-by-step in separate studies with plenty of participants and then to combine the results into a flexible, yet exhaustive, set of methods for different evaluation circumstances.
Furthermore, as all of the evaluation cases presented in this dissertation have also dealt with novel interaction techniques, I would be interested in investigating and developing interaction-technique-related user experience evaluation approaches. However, this more detailed focus of research seems to remain a secondary aim for me at the moment compared to the wider, more societal issues of the environment and the user group as evaluation circumstances and how they affect not only the evaluation design, but also the user experiences per se. In conclusion, my aim for future work is to contribute to the investigation and development of practical user experience evaluation research that takes into account the three main factors of user experience—the system, the context, and the user.
……………
130
……………
131