As mentioned before, Molich et al. (1999) carried out research indicating that even though usability groups may find several usability issues while testing an application, these will broadly defer from the results of a different usability testing group. This conclusion does not mean that a particular group had a lack of skills or knowledge, but instead that it does not exist a standard framework that a usability group could follow to reach some clear statements.
For this purpose the Common Industry Format (CIF) (ISO 25064 2013) was developed to establish a framework that might help for creating consistent reports while executing usability tests. CIF emphasizes on the incorporation of the usability as a part of the decision-making process for computer software.
In this work, one application that may assist on the creation of CIF for usability and user interface test reports is proposed: the CAUTE framework introduced first by Seffah et al.
38
The application Web Based CAUTE Tool (WBCT) aims to help managers and testers during the whole testing process; from the planning until the final report offering a relatively simple and intuitive user interface accessible from a web navigator.
Figure 11 shows the main screen of WBCT. A simple web interface has been used in order to ease the testing procedure for the companies; thus, a high level of previous testing knowledge is not required. The main screen gives access to the different sections of the application; this is the list of projects that some user of the application is currently testing and the list of possible audience for the testing. It also contains graphics about the performance of the testing and notifications with useful information and possible problems.
Figure 11. WBCT: Main screen.
The first stage for using WBCT consists of creating an initial population for the database by means of persona (Wolpert et al. 2011) representing the people who might participate during the testing process: final users, stakeholders, managers and professionals. Figure 12 displays a populated data base that includes some of the profiles defined by Seffah et al.
for the CAUTE process.
39
Figure 12. WBCT: List of possible audience.
On the second step, one of the projects should be selected or the user should create a new one. As shown on the figure below, as one clicks on “Projects”, the system lists all the existing projects.
Figure 13. WBCT: Selection of project.
40
To ease the process, we have divided the eleven steps proposed by Seffah et al. in three stages: “Prepare test”, “Gather data” and “Analyze data”. Each one of these stages is decomposed in a set of smaller phases. “Prepare test” stage includes “Plan”, “Design”,
“Acquire” and “Setup” phases. “Gather data” stage includes “Preview”, “Conduct” and
“Debrief and compile” phases. “Analyze data” stage includes “Analyze”, “Report” and
“Capitalize”. In addition, each of the phases is divided on several sub-steps that request different information to succeed on the testing process. Figure 14 discloses these different stages that should be performed while using the application.
Figure 14. WBCT: Main screen of a product. Starting point.
During the preparation of the test, the general information about the project should be introduced, the testing method, the audience of the testing process, materials (hardware and software) that will be used and the environment where the testing will be executed. At
“Gather data” stage, the WBCT suggest to conduct a pretest and a formal inspection, to prepare the test and to give some recommendations on how to perform the testing method that the user selected on the previous stage. At last stage “Analyze data”, the application invites to search for possible failures and patterns, creating a categorized list by priority, severity and frequency of occurrence. Next, the possible improvements based on the
41
detected failures should be developed and prioritized. At last, one can download a CIF report with all the detailed information from the project screen.
Once the process starts, the first stage “Prepare test” and on the first phase “Plan”, basic information of the project should be introduced: name, version and a short description of the application, the purpose and goal of the test and the frame for the whole process as stated on figure 15.
Figure 15. WBCT: Plan phase 1. The user introduces general data about the project.
Next, the testing method that will be applied during the process should be selected as shown in the image below. Once the methods is selected, the application gives a short description of the testing method, the stages of the development where it can be applied, the typical required personnel, what can be measured, the type of data that can be collected, how to perform the test, the possibility for remote testing, and the weakness of the method.
42
Figure 16. WBCT: Plan step 2. The user selects the testing method
At last, the widgets and scenarios that will be tested should be set introducing a title for the action and an adequate description. Since these scenarios might change before starting the process, the information can be stored and modified if necessary. After introducing the data, one should press “Next” to continue with the second step, “Design”.
Figure 17. WBCT: Plan step 3. The user introduces the UI widgets and scenarios that will be tested.
On the “Design” phase, the application suggest some requirements categorized in three sections, indispensable elements for the selected method, useful elements that would help to perform a better test and elements that are recommendable to use as good practice.
Figure 18 displays the requirements of the thinking-aloud method related to the location and the environment where the test will be performed. On this phase, the available requirements should be identified by marking the checkboxes placed close to the options.
43
Figure 18. WBCT: Design phase 1. The user fills in a form about the requirements related to the location.
Figure 19 shows the requirements connected with the necessary equipment while testing by using the thinking-aloud method. As in the previous step, the resources should be selected for performing the test.
Figure 19. WBCT: Design phase 2. The user fills in a form about the requirements related to the equipment.
The “Design” step continues with the selection of the audience. The application offers the different roles defined by Seffah et al. on the CAUTE process, briefly explaining the
44
functions of each role. The necessary roles for the testing process should be selected using the assistance provided on the previous step while selecting the testing method.
Figure 20. WBCT: Design phase 3. The user selects the roles that will participate during the test process.
On the last step, the amount of people for each of the previously selected role should be set as shown in the image below. Once this action is performed the “Design” phase is completed and is possible to proceed with the next phase, “Acquire”.
Figure 21. WBCT: Design phase 4. The user selects the amount of participants of each role.
On the “Acquire” phase, the individuals who will participate on the process should be selected, as shown on the image below. The system displays the persona of the people who belong to any of the previously selected roles. In this view, people can be included or excluded from the process and the dates when they will perform the tests or meetings
45
should be set. In addition, one should include the role of the user while using the software, the amount of years that the user has been performing the role, the user’s experience using the platform or operating system, where the application will be deployed, the experience of the user in application related to the same domain and the user’s experience with previous versions of the product or similar products. After the individuals who will participate on the process are selected, the “Acquire” step can be considered as complete.
Figure 22. WBCT: Acquire phase. The user selects the individuals who will participate.
The last phase on the “Prepare test” stage is “Setup”. On this phase, the system analyzes the introduced data and displays failures, in red, and recommendations, in blue, considering the testing method selected on the “Plan” phase.
46
Figure 23. WBCT: Setup phase 1. The system highlights possible failures during the preparation phase and provides some recommendations.
At last, the system requires the supervision of all the options included on the review step in order to guarantee that all the requirements have been meet before proceeding with the test itself. As shown on the image below, the hardware and software should be verified, install and configure the product that will be tested, prepare the physical environment completing the requirements settled on the “Design” phase, review and assemble the test materials and prepare the legal documents that the audience must sign to proceed with the test.
Figure 24. WBCT: Setup phase 2. The system requires that the user ensures that all the requirements have been met before starting the test.
After the “Prepare test” stage is completed, the web browser is directed to the product dashboard, as shown on the image below, where one can download a partial CIF report with a summary of the current information and continue with the next stage “Gather data”.
47
Figure 25. WBCT: Main screen of a product with a completed "Prepare test" stage.
The “Gather data” stage starts with the “Preview” phase. In this phase, the WBCT suggests to conduct a pretest and a formal inspection in order to find possible failures on the designed plan for the testing process. The application also suggests how to perform these activities recommending a set of steps that the test manager may follow. Figure 26 stands how the pretest and the formal inspection should be executed. The test manager should mark the checkboxes after the activity has been finished.
48
Figure 26. WBCT: Preview stage 1. System recommendations for conducting a pretest and formal inspection.
Figure 27 shows the last step of the “Preview” phase. WBCT suggest which type of skills are required and on which field the usability testers and observers should be trained to reap the maximum possible benefits of the procedure.
Figure 27. WBCT: Preview stage 2. Suggested training for using thinking-aloud method.
The next phase within the “Analyze data” stage is “Conduct”. In this phase, the system suggests to prepare the test and the observers and observers’ positions as stated on the
49
figure below. The hardware and software should be verified, the test manager should check that they operate as predicted and that the observers are in their positions.
Figure 28. WBCT: Conduct stage 1. System suggestions for test and observers preparation.
On the “Conduct” phase and before starting the test with the audience, the application offers some recommendations about the selected testing method and how to proceed with some required tasks as shown in the image below. After this, WBCT proposes a list of actions that might help on conducting the test since the audience arrives until they finish with the assigned tasks.
50
Figure 29. WBCT: Conduct stage 2. System suggestions on how to proceed using thinking-aloud method and the recommend steps for performing the test.
Figure 30 shows the “Debrief and compile” phase. WBCT suggests to collect the last information from the users with different methods (open-mind opinions, interview the users, post-test questionnaires) that are specific for the diverse testing methods that the application offers. As well, one should perform the same actions with professionals in order to avoid a possible loss of some information prior to the “Analyze” process.
Figure 30. WBCT: Debrief stage 1. System recommendations for debriefing audience after performing the test.
51
On the last step of the “Gather data” stage, the application advocates to organize and clean the raw results and annotate where the data is going to be stored as can be seen in figure 31. In this case, since the thinking-aloud method had been selected, the WBCT proposes that the path for the video, audio and screen capture files should be included so that they are easily accessible to anyone who is using the application or the CIF generated by the WBCT.
Figure 31. WBCT: Debrief stage 2. The system suggests how to organize the results.
After the “Gather data” stage, WBCT displays the information related to the stages that has been completed as shown in the image below. To complete the testing process, “Analyze data” stage should be finalized.
52
Figure 32. WBCT: Main screen of a product with a completed "Gather data" stage.
The last stage of the testing process starts with the “Analyze” phase where one should identify and prioritize preliminary user problems using different methods such as data mining, discovery or prediction. With such information, WBCT creates a list of potential problems as shown in the image below.
Figure 33. WBCT: Analyze stage 1: The user introduces preliminary problems.
53
The second step on the Analyze phase, the required values related to usability metrics and satisfaction should be included for every task and user so the test manager can stablish some comparisons using the data.
Figure 34. WBCT: Analyze stage 2: Introduce usability metrics and satisfaction measurements
After this, the application suggests to prepare some comparisons in order to find new potential problems that were not detected by the users and the professional testers, as shown on the image below.
Figure 35. WBCT: Analyze phase 2: Compare data and introduce patterns and findings
On the “Report” phase, the application gives some recommendations on how to study patterns and recommendations and suggests creating a list of recommendations that might
54
be included in further versions of the product as can be seen in the image below. WBCT also gives some guidelines for the internal report and the process that one should follow after the testing process.
Figure 36: WBCT: Report phase. Introduce recommendations and create internal report
The last phase on the testing process is named “Capitalize”. WBCT encourages the test manager to create, organize and send some questionnaires for the audience in order to collect possible feedback of the testing process that might be used in further testing procedures. Figure 37 shows how the capitalize phase could be presented in a CAUTE tool.
55
Figure 37. WBCT: Capitalize phase: Analyzing the feedback of the participants.
At the end of the “Analyze data” stage, WBCT shows all the information of the testing process and facilitates a complete CIF in a printable and understandable format that can be stored or used as final report of the whole procedure as shown in the image below.
Figure 38. WBCT: Main screen of a product with a completed "Analyze data" stage.
As commented before, at the end of each stage WBCT offers a partial CIF related to the testing process. It is possible to consult the output results, in CIF format, for the case study
56
testing process that have been shown on this section, in the appendix attached at the end of this thesis.
57
SUMMARY
As stated during the document, there no exist common agreements on how to perform a whole test process for a UI. Several scientists have tried to categorize the level of automation of the different user interface testing method and they have found that the methods lack of automated process either on gather, analyze or critique data. Other scientist demonstrated that different testing groups examining the same user interface could obtain diverse results.
The usage of a new process called CAUTE has been suggested. This might ease the testing procedure along with one web application that may guide test managers on planning, gather, analyze and creating conclusions of a test process.
58
REFERENCES
1. Abran, A. et al., 2004. Guide to the Software Engineering Body of Knowledge.
Engineering, 19759(6), p.204.
2. Albert, B., Tullis, T. & Tedesco, D., 2010. Beyond the Usability Lab.
3. Andrews, A.A., Offutt, J. & Alexander, R.T., 2005. Testing Web applications by modeling with FSMs. Software and Systems Modeling, 4(3), pp.326–345.
4. Astels, D., 2003. Test driven development: A practical guide.
5. Balbo, S., 1995. Automatic evaluation of user interface usability: Dream or reality. In Proceedings of the Queensland Computer-Human Interaction Symposium.
6. Barr, E.T. et al., 2015. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5), pp.507–525.
7. Battle, L. et al., 2005. Human-Centered Software Engineering — Integrating Usability in the Software Development Lifecycle.
8. Bell, T. & Thayer, T., 1976. Software requirements: Are they really a problem? ICSE
’76 Proceedings of the 2nd international conference on Software engineering, pp.61–
68. Specification - Business Process Model and Notation. [ONLINE] Available at:
http://www.bpmn.org/. [Accessed 10 March 2016].
12. Câmara, A., 2011. Natural User Interfaces. Human-Computer Interaction - INTERACT 2011, p.1.
13. Case A.F., 1985. Computer-Aided Software Engineering (CASE): Technology for Improving Software. Data Base, 17(1), pp.35–43.
14. Charfi, S. et al., 2014. Widgets Dedicated to User Interface Evaluation. International Journal of Human-Computer Interaction, 30(5), pp.408–421.
15. Cugini, J. & Scholtz, J., 1999. VISVIP: 3D visualization of paths through web sites.
Proceedings. Tenth International Workshop on Database and Expert Systems Applications. DEXA 99, pp.259–263.
16. Dijkstra, E.W., 1970. On The Reliability of Mechanisms.
17. Dix, a et al., 2004. Human-Computer Interaction. Human-Computer Interaction, Third(December), p.834.
59
18. Fidopiastis, C.M., Rizzo, A. a & Rolland, J.P., 2010. User-centered virtual environment design for virtual rehabilitation. Journal of neuroengineering and rehabilitation, 7, p.11.
19. Fisher, A.S., 1991. Case: Using Software Development Tools.
20. Hartson, H.R. et al., 2006. User experience - a research agenda. Foundations and Trends® in Human–Computer Interaction, 33(4), pp.59–70.
21. Hartson, H.R., Andre, T.S.T. & Williges, R.R.C., 2001. Criteria for evaluating usability evaluation methods. International journal of human-computer interaction, 13(4), pp.1–35.
22. Hartson, H.R. & Hix, D., 1989. Toward empirically derived methodologies and tools for human-computer interface development. International Journal of Man-Machine Studies, 31(4), pp.477–494.
23. Helms, J.W. et al., 2006. A field study of the Wheel-a usability engineering process model. Journal of Systems and Software, 79(6), pp.841–858.
24. Hornbæk, K., 2011. Some Whys and Hows of Experiments in Human–Computer Interaction. Foundations and Trends® in Human–Computer Interaction, 5(4), pp.299–
373.
28. ISO, 2010. Human-centered design for interactive systems. Ergonomics of human system interaction Part 210 (ISO 9241-210). Iso 9241210, 40(4), pp.759–68.
29. ISO, 1991. Information technology — Software product quality. ISO/IEC FDIS 9126-1, 1999126-1, pp.1–26.
30. ISO, 1998. ISO 9241-11, Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability.
31. ISO 25064, 2013. Systems and software engineering - Systems and software product Quality Requirements and Evaluation (SQuaRE) - Common Industry Format (CIF) for usability: General framework for usability-related information.
32. Itkonen, J. & Rautiainen, K., 2005. Exploratory testing: a multiple case study.
IEEE/ACM International Symposium on Empirical Software Engineering and Measurement - ESEM, 00(c), pp.82–91.
33. Ivory, M.Y., 2000. Web TANGO: Towards automated comparison of information-centric web site designs. In Conference on Human Factors in Computing Systems - Proceedings. pp. 329–330.
60
34. Ivory, M.Y. & Hearst, M.A., 2001. The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4), pp.470–516.
35. Johnson, J., 2010. Designing with the mind in mind: simple guide to understanding user interface design rules.
36. Kerlinger, F.N. & Lee, H.B., 2000. Foundations of behavioural research Fourth Edi., Harcourt College Publishers.
37. Kohavi, R. et al., 2009. Controlled experiments on the web: survey and practical guide. Data Min Knowl Disc, 18, pp.140–181.
38. Mann, S., 2002. Intelligent image processing.
39. Mayhew, D., 1999. The usability engineering lifecycle. CHI’99 Extended Abstracts on Human Factors in Computer Systems, pp.147–148.
40. Meszaros, G., 2004. xUnit Test Patterns.
41. Molich, R. et al., 1999. Comparative evaluation of usability tests. CHI’99 extended, (May), p.83.
42. Nielsen, J., 2003. Usability 101: Introduction to Usability. Nielsen Norman Group.
43. Nielsen, J., 1994. Usability inspection methods. Conference companion on Human
43. Nielsen, J., 1994. Usability inspection methods. Conference companion on Human