User Experience

9

2 User Experience Evaluation

This chapter introduces background on user experience and its evaluation.

My work has a highly practical emphasis instead of a strong theoretical basis. Thus, this background description is kept compact, and its main idea is to present the stance taken regarding the topics. As user experience is defined and understood in varying ways, here I explain what I mean by the term. Then, I briefly discuss existing methods for user experience evaluations. Finally, I describe two methods, the SUXES (Turunen, Hakulinen, Melto, et al., 2009) and the Experiential User Experience Evaluation Method (Publication IV), which have been utilized in the case studies presented later in Chapter 3.

2.1 U

E

There are numerous definitions for the term user experience. According to the ISO (2010), it is “a person’s perceptions and responses that result from the use or anticipated use of a product, system or service.” This definition takes into account user and system, but ignores context, which is seen as one of the three main factors that user experience is built from by Hassenzahl and Tractinsky (2006) and by Roto, Law, Vermeeren, and Hoonhout (2011). In this respect, the ISO (1998) definition for usability would be more suitable, as it says, "the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” This definition, however, disregards more or less the subjectivity of the matter and also highlights effectiveness needlessly.

Unlike the more traditional usability, user experience is something purely subjective and thus cannot be evaluated by observation or expert evaluation alone. “Usability” and “user experience” are still used almost synonymously surprisingly often, especially in industry (Hassenzahl, 2008).

……………

10

According to, e.g., Roto et al. (2011), a clear, fit-for-all-fields definition for

“user experience” is still missing. Perhaps due to this, the terms “usability”

and “user experience” are constantly interchanged, especially among people who are not directly working with the issue. When it comes down to individual questions asked from users, however, it is undeniable that even the most experienced user experience or usability expert is not always able to say whether the question concerns usability or user experience.

Furthermore, in many cases, such an exclusive separation is simply impossible to make. Sometimes, a single measure can be seen to concern both usability and user experience depending on the point of view, roughly objective or subjective.

To demonstrate the challenge in dividing measures strictly to usability and user experience, case SymbolChat (III) (Publication III) provides a good real-world example: Objectively, we researchers observed and measured the communication with the system to be slow, but the users with intellectual disabilities subjectively rated the communication to be rather fast. Although the measure speed of communication might appear to be a matter of pure efficiency and thus a usability-related measure, here, it was also a matter of added value to the users. Therefore, it can be seen as a measure of user experience. Furthermore, even though an objectively assessed usability property of a system might be poor, its subjective user experience rating may still be good and vice versa—beauty is in the eye of the beholder. Roughly speaking, any usability-related measure can be a measure of user experience as well when asked from the users themselves, but many times, not vice versa: e.g., the user experience measure comfort of a pillow cannot be truly assessed objectively, i.e., by anyone other than the actual user of the pillow.

User experience is something more, then. Its core is on how the user feels, not on how he or she performs, or would be able to perform with a system of a certain “usability level.” While usability can, to some extent, be evaluated in a more objective manner by experts, e.g., user experience is something only the users themselves can evaluate and determine.

Obviously, something about user responses can be said based on observing the users. For example, whether the users seem extremely happy or very disappointed when interacting with a system indicates if the system is well received. However, observation data alone can lead only to educated speculation and cannot be used as the basis for evaluating user experience, as the truth of user experience is only within the user. A better term for observed reactions could be simply “user response” or even “user reaction.”

Still, the term “user experience” is used in studies where, in fact, nothing has been asked from the users themselves (e.g., Vajk, Coulton, Bamford, &

Edwards, 2008). Considering observation, for instance, Roto, Obrist, and Väänänen-Vainio-Mattila (2009) also raise the question “How can we observe how users feel, i.e., observe the user experience?”

……………

11

Alongside several models describing user experience, a number of definitions for the term have been constructed, many of them trying to define almost exactly the same thing with only slight differences in, e.g., wordings and emphasis. Merely to point out a few definitions, Alben (1996), e.g., refers to experience and quality of experience in the context of the ACM/interactions Design Award as “all the aspects of how people use an interactive product: the way it feels in their hands, how well they understand how it works, how they feel about it while they’re using it, how well it serves their purposes, and how well it fits into the entire context in which they are using it.”

Without her explicitly stating this to be a definition for user experience per se, it can, and also has been, interpreted as such (e.g., All about UX—

definitions, 2014). Hassenzahl and Tractinsky (2006), conversely, define

“user experience” as “a consequence of a user’s internal state (…), the characteristics of the designed system (…) and the context (…) within which the interaction occurs (…).” This definition highlights the three core elements affecting user experience—user, system, and context. According to Mahlke (2008), the influence of the user and the context—in addition to the system only—have been recognized as an influential part of usability already by Shackel (1991), for example. The idea of all three components—user, system, and context—having an effect on user experience is highly relevant for my research and this dissertation: User experience evaluation cannot be designed disregarding any of these factors.

Hassenzahl (2008) later states simply that user experience is “a momentary, primarily evaluative feeling (good–bad) while interacting with a product or service,”

but restricts his flexible definition by continuing: “Good UX is the consequence of fulfilling the human needs for autonomy, competency, stimulation (self-oriented), relatedness, and popularity (others-oriented) through interacting with the product or service (i.e., hedonic quality). Pragmatic quality facilitates the potential fulfilment of” these ”be-goals.” The second part of the definition suggests that all of the listed human needs demand to be fulfilled to achieve good user experience, and thus, sets high standards for user experience. From the viewpoint of the practical evaluation work done for this dissertation, Hassenzahl’s (2008) definition is overly complex and perhaps too accurate.

Furthermore, this definition overlooks context.

As this dissertation is not attempting to solve theoretical issues in user experience research, but instead has a highly practical perspective, the definition for user experience is kept simple and flexible. Here, “user experience” means:

A user’s subjective opinion about (or answer to) a certain statement (or question) about the system (or modality, interaction, or any other specified target) in a certain context at that time.

I kept the definition loose so it will not restrict the kinds of users, opinions, statements, questions, systems, or contexts it can deal with. The definition

……………

12

may be used with a range of agendas, be it user experience, usability, or consumer satisfaction. In fact, this definition does not exclude non-interactive or even non-computer-based “objects,” but instead, can be used concerning anything that can have a user in the first place—be it an interactive public display or a watering can. Furthermore, to maintain simplicity, the abbreviation UX is not used in this dissertation. The complexity around the term and its definition seen in literature is probably only increased by using the buzzword-like abbreviation “UX.” Thus, the term “user experience” is interpreted literally here: an individual using an object (user) + his or her feeling about the object (experience) = user experience.

Furthermore, the concept of user experience comprises different aspects or focus areas. For example, Wright, Wallace, and McCarthy (2008) talk about aesthetic (user) experience, and they identify several aspects of experience:

sensual, emotional, spatio-temporal, and compositional. While specified aspects of user experiences may be particularly relevant for certain studies, such special nuances of user experience are out of the scope of this dissertation. Here, the core is on how to evaluate user experience, not specifically what to evaluate. Apart from some exceptions, the focus here is mainly on short-term user experiences. The user experiences gathered are rather general-level experiences, one might say even usability-like aspects, such as pleasantness or easiness to learn. However, each case study has its own characteristics—more detail can be found in Chapter 3.