UNIVERSITY OF VAASA FACULTY OF TECHNOLOGY SOFTWARE ENGINEERING
Petteri Siponen
AUTOMATED TESTING VIA GRAPHICAL USER INTERFACE
Master´s thesis for the degree of Master of Science in Technology submitted for inspection, Helsinki, 20 April 2016.
Supervisor Prof. Jouni Lampinen
Instructor Prof. Jouni Lampinen
Inspector M.Sc. EBA, M.A. Laura Lappalainen
PREFACE
This thesis has been a great challenge and it has taught me a lot. I want to thank my family and friends, my professor Jouni, and my girlfriend Kirsi supporting and giving feedback.
TABLE OF CONTENTS page
PREFACE 1
TABLE OF CONTENTS 2
PICTURES 4
TABLES 5
TIIVISTELMÄ 6
ABSTRACT 7
1. INTRODUCTION 8
2. SOFTWARE TESTING AND TEST AUTOMATION 11
2.1. Software Testing 11
2.1.1. Black-Box Testing 15
2.1.2. Regression Testing 17
2.1.3. End-to-End Testing 18
2.1.4. Acceptance Testing 19
2.1.5. GUI Testing 20
2.2. Test Automation 21
2.2.1. The Objectives of Test Automation 24
2.2.2. GUI Test Automation 25
2.3. Graphical User Interface 26
2.4. Approaches to GUI Test Automation 28
2.4.1. Framework Use in GUI Test Automation 28
2.4.2. Pattern Recognition and Computer Vision in GUI Testing 29
2.4.3. GUI Ripper 31
2.5. Testability and Quality 33
2.5.1. Testability 33
2.5.2. Quality in Software 34
3. LITERATURE RESEARCH 38
3.1. Overview 38 3.1.1. Statistics of Android GUI Test Automation Literature 39 3.1.2. Statistics of Mobile GUI Test Automation Literature 41 3.1.3. Statistics of Other GUI Test Automation Literature 43 3.2. Literature Findings for Mobile GUI Test Automation 45
3.2.1. Capture/Replay 45
3.2.2. Test Case Generation 45
3.2.3. GUI Rippers and Crawlers 47
3.2.4. Automated Mobile Testing as a Service 49
3.3. Other GUI Test Automation Related Literature Findings 50
3.3.1 Test Case/Test Script Maintenance 50
3.3.2. Visual GUI Testing 52
4. CASE STUDY - PREPARATION AND PLANNING 54
4.1. Definition of the Case 54
4.2. Case Execution Definition 55
4.3. Objectives of the Case Study 58
5. CASE STUDY - EXECUTION AND RESULTS 59
5.1. Execution of the Case Study 59
5.2. Observations and Obtained Results 63
5.3. Analysis of the Results 64
6. RESEARCH FINDINGS 66
6.1. Literature Research 66
6.1.1. Mobile GUI Test Automation Findings 66
6.1.2. Other GUI Test Automation Related Findings 68
6.2. Case Study Findings 70
7. CONCLUSIONS 72
REFERENCES 75
PICTURES page
Picture 1. Fixing the bugs can increase dramatically over time (Patton 2006:18). 12 Picture 2. These are the main states of the bug life cycle (Patton 2006:302). 17
Picture 3. Different mobile GUIs. 27
Picture 4. Scopus search result with search terms: “GUI test automation Android". 40 Picture 5. Scopus search result with search terms: “GUI test automation mobile". 42 Picture 6. Scopus search result with search terms: “GUI test automation mobile". 44 Picture 7. GUI Test App’s original view and GUI objects are presented. 56
Picture 8. GUI Test App mutations. 61
Picture 9. Pointer comparison between failing and succeeding case. 63
TABLES page
Table 1. Test cases for executing GUI functionality tests in GUI Test App. 57 Table 2. Different mutations of the GUI Test App. 57 Table 3. Results for the test run of each mutation. 64
VAASAN YLIOPISTO Teknillinen tiedekunta
Tekijä: Petteri Siponen
Diplomityön nimi: Testausautomaatiota graafisen käyttöliittymän kautta
Valvojan nimi: Prof. Jouni Lampinen Ohjaajan nimi: Prof. Jouni Lampinen
Tutkinto: Diplomi-insinööri
Koulutusohjelma: Tietotekniikan koulutusohjelma Suunta: Ohjelmistotekniikka
Opintojen aloitusvuosi: 2007
Diplomityön valmistumisvuosi: 2016 Sivuja: 83 TIIVISTELMÄ:
Diplomityö käsittelee testausautomaatiota graafisen käyttöliittymän (GUI) kautta testattaessa Android-sovellusta käyttäen mustalaatikkometodia. Työn laajuus on rajattu GUIn testaustekniikoihin, jotka käyttävät mustalaatikkometodia ilman, että käytetään lähdekoodia. Myös muita GUI-testaustekniikoita ja ehdotettuja lähestymistapoja tutkitaan, erityisesti mobiilitestaamisen osa-alueella. Työn tavoitteena on löytää ratkaisu testiskriptin ylläpitämisen ja testiautomaation ajamisen tehostamiseksi. Toisena tavoitteena on löytää parhaita käytäntöjä testausautomaatiolle testattaessa sovellusta sen käyttöliittymän kautta.
Työ suoritettiin kaksivaiheisena. Ensimmäinen vaihe koostui kirjallisuustutkimuksesta keskittyen kolmeen tutkimusalueeseen: Android GUI-testausautomaatio, mobiili GUI- testausautomaatio, ja muu GUI-testausautomaatio. Tulosten selvittyä, ne jaettiin kahden otsikon alle: mobiili GUI-testausautomaation löydökset ja muut GUI-testausautomaatio löydökset. Työn toisena vaiheena oli tapaustutkimus, jonka aiheena oli Android- sovelluksen GUI-mutaatioiden testaaminen visuaalisen GUI-testauksen avulla.
Työn tuloksena löydettiin parannus graafisen käyttöliittymän kautta tehtävään testaamiseen käyttäen mustalaatikko-ympäristöä. Visuaalinen GUI testaus –metodin käyttö tuo tavoiteltua tehokkuutta verrattaessa lähtötilanteeseen, jossa Record and Replay -tekniikkaa oli alunperin käytetty. Myös havaitut lasilaatikkotekniikat antavat potentiaalisia tapoja kehittää testausautomaatiota graafisen käyttöliittymän kautta tehtävään testaamiseen hyödyntäen esimerkiksi GUI Ripping -tekniikkaa mallintamaan GUI sekä automaattisesti luomaan testitapaukset sekä -skriptit. Jatkon kannalta olisi tärkeää tutkia hahmontunnistuksen sekä koneoppimisen mahdollisuuksia kehittää GUIn kautta tehtävää mustalaatikkotestaamista. Myös sitä tulisi tutkia, kuinka GUI-muutokset voisi jakaa testityökalulle, jotta se voisi oppia ja ylläpitää oppimaansa tietoa. Lisäksi visuaalisen GUI-testaamisen ja GUI Ripping -tekniikan sulauttamista tulisi tutkia lisää, koska näiden yhdistäminen toimivaksi kokonaisuudeksi mahdollistaisi tukevan tavan testata sovellusta GUIn kautta.
AVAINSANAT: Testausautomaatio, GUI, Mustalaatikko, Android, Mobiili
UNIVERSITY OF VAASA Faculty of technology
Author: Petteri Siponen
Topic of the Thesis: Automated Testing via Graphical User Interface Supervisor: Prof. Jouni Lampinen
Instructor: Prof. Jouni Lampinen
Degree: Master of Science in Technology
Degree Programme: Degree Programme in Information Technology Major of Subject: Software Engineering
Year of Entering the University: 2007
Year of Completing the Thesis: 2016 Pages: 83 ABSTRACT:
The subject of the thesis is automated testing via graphical user interface (GUI) when testing Android application by using black-box method. The scope of the thesis is limited to GUI testing techniques using black-box method without using the source code. Also other GUI testing techniques and proposed approaches will be studied especially in the field of mobile testing. The objective for this study is to find solution for making the maintenance of test script, and test automation runs more efficient. Other objective is to find best practices for test automation via GUI.
The research of this study was executed in two divided phases. The first phase was literature research focusing on three different areas: Android GUI Test Automation, Mobile GUI Test Automation, and Other GUI Test Automation. After found the results, they were arranged into two titles: Mobile GUI test automation findings, and Other GUI test automation findings. The second phase was a case study of using Visual GUI Testing technique for testing Android GUI mutations.
The outcome of this study was the found improvement for testing via graphical user interface in black-box environment. Using Visual GUI Testing method gives efficiency in testing via GUI compared to the initial situation where Record and Replay technique was initially used. Also the found white-box methods are giving considerable best practices for improving test automation via GUI e.g. using GUI ripping for modelling the GUI and generating automatically test cases and test scripts. The future studies could focus on how the pattern recognition and machine learning could help GUI testing in the field of black-box techniques. Other future study aspects could focus on how the GUI changes could be informed to the test automation tool to learn and gain knowledge of the changes. Also the implementation of Visual GUI Testing and GUI Ripping should be studied more for making a robust method for GUI testing.
KEYWORDS: Test Automation, GUI, Black-box, Android, Mobile
1. INTRODUCTION
Quality plays a big role in software development and it has risen to a significant role.
The reason behind this growth is that users are very quality driven and it is easy to switch the software or application to another if you aren’t happy to its content or quality. The product needs to be tested and quality assured properly to meet the product specification and also the consumers’ expectations. Mobile markets have given a full range of different choices for the user and this is why it is easy to switch to using a
“better” application.
There has always been awareness of testing and examples have shown what can go wrong if something hasn’t been tested properly. Testing is needed mainly to save the loss of money. In mobile markets, as in any other markets, testing is needed to secure that everything is working properly. If your product isn’t, the user will find another application that will work and the company will loose the customer and their money.
The supply of mobile device on the markets started the rapid growth of mobile applications markets at the same time. This phenomenon grew the competition in the application markets. Latest report from Ericson shows that at the end of year 2015 there was 7,3 billion smartphone subscribers globally and growing yearly around three percentage (Ericson Mobility Report 2016). This tells the size of the force there is to move the markets and it has become this big very quickly. It also enables the possibility for the customer to switch the application if it doesn’t satisfy.
Now that we have electronic devices and different kind of gadgets around us has made us very quality driven consumers. People are keen on the quality and also very instantly changing the product or application to the next one if they are not happy with it. As said earlier this leads us to testing and why it’s important to ensure the level of quality of the product. To reach the level of preferred quality, the product needs to be tested properly and it takes time and money. To cover fully all the main areas and most urgent testing tasks, it can be costly and on most of the products the overall costs need to be cut to the minimum. This means also cutting the costs from the testing. As it is commonly known the testing is needed earliest as in the beginning of the specification of the software development and throughout the whole development life cycle. Reason why testing should be paid attention in the very beginning of the process is because of the costs of the found bugs are low and better quality can be reached easier. All this and the
repeatable manual tasks have led the testing into the automation, the next level of testing. With automation the testing can be done in a more efficient way and this also saves money and time. Normally test automation is used for example to speed up the testing, to break the software more quickly or more effectively in a repeatable manner, or help execute complex test cases in a more timely manner. Of course the test automation isn’t the solution to minimize the entire manual testing, but it is a great effort to lower the budget in one way. And if we focus on test automation and one of the issues that brings the costs up is the maintenance of the scripts and test cases. Test cases need to be checked and updated accordingly, if there are any changes in the product or design documents and some parts of product have been developed and changed. The maintenance of test cases and test scripts takes a lot of time and it needs to be done continually as the product will change and develop rapidly.
The premise for this Thesis is to find best practices to reduce the need of script maintenance and make the test automation more automated and intelligence than it is currently. In this thesis the focus will be on Android application’s Graphical User Interface (GUI) test automation maintenance focusing on functional regression testing using black-box testing without instrumenting the source code. The object of this research is to find a more intelligent way to use test automation when testing GUI in a repeatable manner. The automated GUI testing can be done recording the actions for the test case by using the application in test and then modifying the scripts. Normally there are several different test cases recorded by using test automation tools. When some update has been made to GUI and it normally changes the GUI layout some how, for example one main button has changed its position. In this case the recorded test cases using this button are no longer valid and they need to be updated to match with the current new version of the GUI. The scripts need to be updated manually or the whole recording needs to be done again. This iteration of maintenance doesn’t end after one update and it continues as long as the development is running. This means a lot of work is needed for maintenance. Making the maintenance of scripts more efficient would also enhance the whole development process. The main focus will be studying the existing studies to improve test automation via GUI when using black-box approach. Also non- black-box methods will be considered whether they are proven to increase the GUI test automation efficiency.
The scope of this thesis will be limited to Android application’s GUI’s functional testing with test automation. In the section of Software Testing And Test Automation,
the basic information of the areas of testing, test automation, GUI and GUI testing, and Testability and Quality will be presented.
Research will be carried out in two sections and these are: Literature research and Case Study. Research focus is on literature research and it will be complemented with case study of a test automation tool and bring the available information together to give the best solutions for making the maintenance of scripts in test automation more efficient when testing Android application. These results will provide guidelines for test automation to make the test automation maintenance more efficient. This makes the maintenance easier and reduces the manual work. The whole study will be explored from the perspective of GUI test automation when functional testing Android applications in black-box environment.
2. SOFTWARE TESTING AND TEST AUTOMATION
This chapter will present the basis for this research of test automation via graphical user interface. The main areas in this work are Software Testing, Test Automation, Graphical User Interface, Approaches to GUI Test Automation, and Testability and Quality.
Software testing chapter will present all the information needed from the point of view of this thesis. The main areas to be focused are black-box testing, regression testing, end-to-end testing, acceptance testing, and GUI testing. In the chapter of Test automation the objectives of test automation will be presented and GUI test automation is also studied. Graphical User Interface chapter presents the basics of what is graphical user interface and what it is consisted of. In the chapter of Approaches to GUI Test Automation some of the approaches or techniques are presented and studied a bit more in detail. The last chapter in the section of Software Testing And Test Automation will be Testability and Quality. This chapter presents the ground for what is quality and testability in overall.
2.1. Software Testing
Information systems and software systems have extended their range widely to business and consumer products and as part of the daily process. Nowadays people are used to use different kind of systems or applications and quite many have experienced the situation when software is not working the way as expected. In case of malfunctioning software it can lead to several problems for example loss of money, time, or reputation.
Worst case could be injury or death caused by software error. A human being with a mistake can cause these program defects. (International Software Testing Qualification Board 2011: 11–12.)
To avoid this kind of issues it is important to bring the testing as a part of the software development process and test as thorough as possible already from the start of the process. Before going any deeper to the testing, let’s start from the history. Where the testing started and how the name “bug” has established its role in software development.
The computer error has got its nickname, the bug, early in the 1947 when the computers were very big as filling the whole room. This occasion happened in Harvard University where the greatest computer at that time, Mark II, was built. The computer was on its run of different phases and it suddenly stopped. Technicians tried to find the problem and after a while they found a moth deep inside the computer. It had died to the high voltage and caused the computer to malfunction. This little moth was the reason to call computer errors as computer bugs. (Patton, Ron 2006: 11.)
Software bug can be defined with the specification of the product. If the software doesn’t do something the specification has declared, or the software does or doesn’t do something it shouldn’t or it hasn’t mentioned in the specification. Then of course, if the software has some usability issues like difficulties to understand, it is processing slow or hard to use then these actions can be also called software bugs. (Patton 2006: 15.)
Picture 1. Fixing the bugs can increase dramatically over time (Patton 2006:18).
It is important to start testing early in the software development process to avoid the big costs of a bug found in a later state of development, or worst, after the release. Bugs can be found at any state of a development process and early found bugs are cheaper to fix as it is shown in the Picture 1. If the bug is found while writing the specification it could cost $1. If the same bug were found in test round, it would cost from $100 to $1000.
After release this bug could be from thousands to millions of dollars depending on the bug and at what state it would be found. (Patton 2006: 18.)
When software is being specified and the development starts, there are different states the product goes through in the software development life cycle. When the software has been defined and the specification has been made the testing starts also. During this cycle there are several of different type of testing done and here are couple of examples what kind of testing it is.
One of the first tests is unit testing, which will be done to different units e.g. software modules, subprograms, subroutines, classes or other procedures in the program that are smaller parts of the whole software. This way the software will be tested starting from the smaller units rather than testing the whole product at once. This way of testing also helps debugging when the bug has been found and specified to exist in a specific module or unit. The main purpose of unit testing is to test if the function of the unit is contradicting a specification of another function or interface defining the unit in test.
Most of the unit testing is white-box oriented because it’s feasible when you have a smaller set to focus on. Also you only need to find certain specific type of bugs while focusing on one unit than a bigger part of software. (Myers, Glenford J., Corey Sandler
& Tom Badgett 2011: 85–86.)
White-box testing is one approach to test the software. White-box is opposite to black- box testing and is used for unit testing as mentioned previously. White-box testing may not be feasible if executed to bigger modules than smaller units of software. In white- box approach the tester knows what happens after giving an input to the unit or module and can observe how the outcome is produced. Whit this access to the code the tester can examine it and take some clues to his testing and this way find some weak points in the code. When executing the white-box approach it is possible to become biased and perform the tests to match the code’s operation. This way you would loose the objective approach to test the software. (Patton 2006: 55–56.)
After unit testing the software by testing it in pieces, these pieces will be put together into larger ensembles. At his point the units have been tested and shouldn’t bring up any surprises. Next step from the lowest level testing is integration testing. Integration testing is done against group of units that have been integrated to work as a bigger set.
This integrated set will be tested properly so that they work together and no bugs have been found. This process will continue incrementally putting together the units of the software and test that they work together. After all the units have been integrated and
tested the entire software has been put together and it is ready for next step, system testing. (Patton 2006: 109.)
After all the unit and integration testing is done the system testing can start. During system testing the software will be tested against its functional and structural stability and also the non-functional requirements will be tested, such as performance and reliability. If there is requirement for the software to interoperate with other software systems in collaboration, this will be tested during this phase. There can be organized a system integration testing, in case there are several systems to interoperate with and the interoperation may be very complex. System integration test is part of the integration testing. (Watkins, John 2001: 59.)
No we have presented some of the most used and most important parts of testing software. The field of testing contains many different phases and approaches of testing.
And it depends on the project and budget what testing is explicitly needed and afford to run on the software. Here are couple of more phases and approaches to be mentioned.
One of the most important test areas is the software security. This area needs to be focused well by planning the testing very carefully to cover every corners of software security. Testing needs to use risk-based approaches so that they are heavily studying the architecture of the software and at the same time having the mind set of and attacker (Potter & McGraw 2004). Other important areas to test are compatibility and performance of software in order to function on different environments.
In compatibility testing, it is needed to make sure the software is interacting the way the user would expect when using software on a device that is supported by the software.
So the compatibility means the software is interacting and sharing correct information with other software or system. This can be anything from copying text from text edit software and pasting it to another software, or mobile application working on different versions of the same operating system. The scope of software, platforms and devices is huge and to compatibility test all these can be a big task to achieve. It is normal to limit the scope to be more efficient. The key things, when limiting the set to be most effective, are popularity, age, type and manufacturer. With these criteria it should be easier to limit the scope of software, platforms and devices to be more reasonable.
(Patton 2006: 141.)
Software performance is one of the important measures of quality software. If the software is functioning well and there aren’t any errors with performance, the user doesn’t notice anything and the software can carry out all the tasks without any delay or irritation of the user. In the opposite situation the user experience can be very irritating or even causing financial loss to the company. When testing the application performance all the different levels of application need to be tested. The higher level is the client, the software visible to the user, and hosting services. At the lower level are servers that run the software and network infrastructure. And if there are third-party service providers integrated, those need to be taken into account as well. Software performance can be measured with key performance indicators, service-oriented and efficiency-oriented indicators. Service-oriented indicators are availability and response time and efficiency-oriented are throughput and capacity. Availability is measuring the time of the software is available to the user. Response time is measuring the time to respond to the user request. Throughput is a rate of events occurring within a given period of time. And the capacity is presenting the used theoretical resource capacity.
(Molyneaux, Ian 2014.)
Here was explained the testing in software development in a higher level and without going into the details and explaining all the methods and approaches of testing. Next subchapters will open more of the testing from the point of view of the research are of this thesis and explaining also more of the methods needed in GUI testing.
2.1.1. Black-Box Testing
Black-box and white-box testing was mentioned previously in this thesis and white-box was explained briefly. Here we are going to dive more into the black-box testing, because it is the main approach of GUI testing in this thesis. There are also two types of testing the black-box approach can be executed and these will be also explained later in this chapter.
Black-box testing approach is opposite to white-box testing. When using black-box approach the tester knows only what the software is supposed to do and what is the output if given a certain input to the software. Black-box testing doesn’t give an opportunity to view the code to see how the software is operating. This limits the testing to test the software more from the user’s point of view. (Patton 2006: 55.)
As mentioned there are two types of testing that can be used with black-box testing.
These are static and dynamic testing. Static testing is a way of testing something that isn’t running, e.g. examining and reviewing. Dynamic testing is then the “normal” way of testing software by using it in action. (Patton 2006: 56.)
Static black-box testing is the software specification testing and examining it for bugs.
This static testing is performance of a high-level review of the software specification. It is not meant to find any specific bugs. Finding large fundamental problems or oversights are the main goal in static black-box testing. For static testing it is important to do background research on who will be the software users and research on similar software to have some knowledge of the competitive software. (Patton 2006: 56–57.)
Now that we have covered the static black-box testing we can move on to dynamic black-box testing. Dynamic testing is basically testing of a running software and acting as an end user. As mentioned previously black-box approach doesn’t give the opportunity to look into the code. The dynamic black-box testing approach is leaning on these requirements. This approach is also named behaviour testing because it is testing the behaviour of the software while it is used. For testing with dynamic black-box approach the software documentation or specification is needed to know how the product will behave. If given an input A, you need to know the output B so that you know the software is working properly. With the specification you can start creating the test cases for the testing. Black-box testing has two fundamental approaches that are test-to-pass and test-to-fail. With test-to-pass is meant to test the software in its normal behaviour and with out pushing it to its limits. Test-to-fail then tries to break the software and pushes the limits. (Patton 2006: 64.)
International Software Testing Qualifications Board’s Foundation level syllabus presents techniques of black-box testing. First of the techniques is Equivalence Partitioning. With equivalence partitioning the inputs to the software are evaluated by their behaviour and then put into the groups that are expected to behave in a similar way. Other technique presented is Boundary Value Analysis. This analysis is useful technique with equivalence partitioning to test the boundaries of every partition. The boundaries are the maximum and minimum of a partition. Decision Table Testing is one technique to test system requirements that include logical conditions. Also the internal system design can be documented this way. The strength of this technique is that it creates the variety of different combinations of conditions that wouldn’t normally be
executed during normal round of testing. Two last techniques presented are State Transition Testing and Use Case Testing. The state transition testing is a technique to test the different states of the software and the transition between these states. A state transition diagram can be created from the software behaviour. This technique is very useful and used mainly in embedded software development and in technical automation.
Use case testing is then testing the software, as it would be actually used in real-world.
Use cases can be described in abstract level and test cases are derived from the use cases. This technique is very useful in acceptance testing and it can also discover integration bugs. (International Software Testing Qualifications Board 2011.)
2.1.2. Regression Testing
Regression testing is a big part of test automation when you are running the specific test cases multiple times during the application life cycle. The application runs trough a great amount of test automation and these runs are checking the feature, or other content updates hasn’t created any errors in the application i.e. running the regression tests.
Picture 2. These are the main states of the bug life cycle (Patton 2006:302).
Before going into the regression testing we need to explain a bit what is the life cycle of a bug. In Picture 2 it has been presented in a very generic way and it can vary a lot in different companies how the bug life cycle runs. Main steps for the bug, after it has been found and reported, are Open, Resolved and Closed. During these steps the bug
will be reported and fixed and the fix will be regression tested to be able to push the bug to the Close state. There are also two additional states in the picture: review and deferred. In Review state the bug will be reviewed if it needs to be fixed. The deferred state is for the bugs that are review to be fixed but are not urgent and can be fixed later.
(Patton 2006: 302.)
Regression testing can be divided in two “approach” where regression testing is needed.
In overall the regression testing is always testing by using the tests that have been already executed at least once before (Tamres 2002: 223). The first one to be mentioned is regression testing a fixed bug. After a bug has been found, it will be reported and assigned to the programmer and the bug will be fixed. Before the bug can be pushed to closed-state it will be regression tested to assure it has been fixed correctly and it’s not creating any new bugs. (Patton 2006:303.) Mainly the regression testing is known to use when e.g. an existing feature has been updated or modified and it needs to be tested whether it has created any errors or it is malfunctioning with other features integrated with it. This way it needs to be ensured the function or feature is working as it was in earlier versions. (Tamres 2002: 223.)
Ideally the regression tests should be run after every change made in the application or to its environment. This way the regression testing could be implemented to be part of the development and maintenance processes. The multi-level regression testing could be executed by first regression testing on the unit level and after that on the integration level. There are benefits when using this type of approach. First of all, when run the same tests on different levels it can reveal new bugs. The components can be run at the same time and the time used to detect the errors can be minimized. (Holopainen, Juha 2014: 8.)
2.1.3. End-to-End Testing
End-to-End (E2E) testing is one of the testing techniques as the unit and integration testing are. E2E testing is focusing on black-box testing and executing tests from the user point of view. Functional testing, which has been proven to detect effectively faults, is the main method on E2E testing. (Paul, Ray 2001: 211.)
When comparing E2E to unit and integration testing, E2E is not focusing on any single units as in unit testing. E2E testing is closer to integration testing but as said previously
E2E testing is application testing from the end user point of view and acting like a real user without focusing on testing any specific subsets of the application. When proceeding to E2E testing, the unit and integration testing should have been already executed and approved. E2E testing can be applied to any development processes because of its ability to be independent from any development process models and it can be also implemented in early states of the process to meet the requirements. (Paul 2001: 212.)
E2E testing follows the same steps of executing the tests for example in unit and integration testing. The steps are test planning, test design, test execution, and result analysis. (Bai, Tsai, Paul, Shen & Li 2001.)
2.1.4. Acceptance Testing
Acceptance testing is the one of the last tests done to the product and is ran after system testing. If some company has ordered the product, the real end user often executes these tests or at least Acceptance testing is done on the customer’s site as a showcase the product is developed according to the requirements. (Tamres 2002: 223–224.)
Acceptance testing is the validator for the given requirements and the final check to verify the product is ready for release. Test cases in Acceptance testing are covering all the main functionalities and requirements that an expected user will proceed when using the product. (Tamres 2002: 223–224.)
Agile testing is one approach to whole idea of testing being a part of the agile development environment. This means that everyone is part of creating the test plan, even the customers by defining use cases and attributes for the test cases. In agile development cycle the customer starts acceptance testing the product already in early states of the cycle. This way the developer can get valuable information and testing becomes more integrated part of development cycle. In agile development process named Extreme Programming the testing has been divided in to two phases: unit and acceptance testing. Also tests need to be written before coding starts. As the customer designs the tests for acceptance testing he or she also executes the tests. This is because of the objective point of view and for assuring that the product meets the requirements.
(Myers et al. 2011: 178–187)
Acceptance testing is the last checkpoint for the product either before its release to live environment in mobile applications or if delivering a product to a customer, then this check is for ensuring the product is in line with the specification and requirements.
2.1.5. GUI Testing
Graphical user interface (GUI) can be tested in couple of different ways. For example the focus can be either on unit testing the GUI or testing the system and its logic by focusing on the functionality via GUI. Both are important and complex tasks to perform.
GUI is combined of several different components that are also units. These units can be tested individually. Also GUI needs to behave correctly and for this category there’s also a list of different kind of behaviors. Units inside the GUI are usually called elements or objects and they include all the buttons, windows, text boxes, menus etc.
Examples of behavior in GUI are mouse clicks and movements, value displaying, disable, etc. All of these can be tested as different units before integrating them all to build up the GUI. (Hamill, Paul 2004.)
Other GUI areas to be tested are graphics, responsiveness, usability, and scalability.
These are normally tested at least on low- and high-end devices to ensure the product is running correctly on different hardware and behaving as expected with the provided environment.
As mentioned, functionality testing is another way of testing GUIs. GUI is based on event-driven functionality and every input given to the GUI creates an event flow to the event listener and it will perform an action that GUI will display. Functional testing the GUI will give confirmation that the product is behaving correctly when given user inputs to the system. (Ruiz, Alex & Yvonne W. Price 2008.)
Automation is rolling in with a big force. But manual testing is still needed, at least on GUI testing. Manual GUI testing is useful way of revealing bugs. Manually found bugs can increase adaptability by leading to another similar bugs that haven’t been found.
Another thing that automation cannot measure or test is the usability and user experience of the product. For catching the usability issues might need an experienced tester, but still this is a task for manual testing. Manual testing is still needed, but the
automation has taken its place. It has been developed different kind of tools and methods to run test automation tests. GUI testing is taking a great amount of testing effort and that is why there are several different tools available. The tools aren’t just for executing test, but also e.g. generating test cases, GUI model builder, and other supportive tools for test execution. (Yang, Xuebing 2011: 28.)
2.2. Test Automation
This chapter of test automation will present what is test automation and how it can help testing applications. Different tools for test automation will be presented. Objectives of test automation and GUI test automation have their own subchapters and these titles are looked into in a more thorough manner.
As we know, no testing or test automation is for free because of the amount of work to be done to achieve related benefits. To success in test automation you need more than just the tool acquisition. Robust background for test automation can be build with good test process, stable environment and realistic time budgeting. Here are some points to achieve success in implementing test automation: (Tamres 2002: 226.)
• Significant amount of time is needed first to program the test scripts and then to maintain the test suites.
• Programming experience is needed when programming test automation scripts.
• Tester needs training with tools to create right tests for automation.
• Good management is required to control the content related to tests and all files included.
• Automation is not guarantee of finding all existing bugs.
• New tests are needed while software develops and existing tests can get old and can’t find new bugs.
• Awareness of that test tools themselves are software and can have their own bugs. (Tamres 2002: 226.)
Testing can be time consuming and repeatable work and automation is a great effort to make these tasks more efficient and less time consuming. Not to mention that with automating testing you reduce the risk of not finding the bugs. One other benefit of test
automation is that the regression testing can become easier. The main principals of using test tools and automation are speed - running test with tool can do the job many times faster, efficiency - while you set your tool to run tests you may also use your time to focus on cases the automation can’t handle, accuracy and precision – human can make mistakes and focus can loose after running hundred of test cases, but a tool will perform the same work as planned without loosing the focus, resource reduction – with a tool it is possible to expand the resources needed by simulating the real environment, simulation and emulation – e.g. software or hardware can be simulated by a test tool to have a normal interference with the product in test, relentlessness – test automation and tools have all the energy to run tests you assign to them. Even this sounds very promising, but automation isn’t always the right call. The manual testing can be in some situations the right choice and test automation shouldn’t be applied. The whole test automation environment needs also a lot of time to be built and it doesn’t come up in a second. Test tools are for making testing more efficient and faster, it’s not a solution to put a computer to do your work while you’re resting. (Patton 2006: 231–233.)
Patton (2006: 233–239) presents some tools for test automation. These tools are viewers and monitors, drivers, stubs, stress and load tools, interference injectors and noise generators, and analysis tools. A brief presentation of these tools will be given in this section next.
Viewers and monitors are tools for seeing the events behind the software user interface.
White-box testing tools that help to see which part of the code or functions are being used and are they behaving as they should. One example of a viewer is a communications viewer, which enables to see the raw protocol data from i.e. the network. It works by observing the information and presenting it to another computer where the tester can observe and analyse the possible bugs. (Patton 2006: 234.)
Patton (2006: 235.) says that: “Drivers are tools used to control and operate the software being tested.” With a driver it is possible to create a sequence of wanted actions to be executed. Drivers can execute programs or act as a mouse or keyboard for example by adding another computer to be the driver and give the needed inputs. (Patton 2006: 235–
236.)
Stubs are also one type of drivers. They can receive or respond information and stubs are known as a white-box testing method. Patton gives an example that a printer can be
replaced with a computer, which will act as a stub and then the output can be analysed thoroughly. (Patton 2006: 236.)
Stress and load tools are for exercise of the software ability to handle great amount of stress and load. The system resources are playing a big role and available memory, disk space and files are affecting how the application is surviving. This could be done manually, but the tools are more efficient and helpful when testing this kind of part of the product. Load tools are more known in web environments by testing for example the servers load ability by simulating a situation with required set of connections and hits.
(Patton 2006: 237–238.)
Interference injectors and noise generators have similarities to previous tools presented.
For example the viewer tool can be modified to get an interference injector by switching the viewer with a computer that can view and alter the communication line’s data. This interference injector can simulate communication errors of all types. (Patton 2006: 238.)
The last tool category to be presented is analysis tools. This category holds in different kind of everyday tools that makes it easier to proceed with daily tasks and can save time. Patton mentions couple of software types that are in this category: word processing, spreadsheet, database, file comparison, screen capture and comparison software, and for example debugger. (Patton 2006: 239.)
Common tasks in test automation are also capture-playback tools, comparators and test execution tools. The idea in capture-playback tools is to record events for example keystrokes, mouse activity and output on the display. This and other GUI test automation tools will be presented in chapter 2.3.2. Comparators help in determining if the test case is passed or failed by comparing two data sets. Test execution tools are for running the whole execution of the test. This includes application’s initialization, data sending to the application, output capture that will be evaluated by comparator and finally results are filled in to the log. (Tamres 2002: 225–226.)
In cases that test is relying on the manual interaction, tests are plan to run only once or infrequently, or test evaluation is hard to program and it could be easier for a person to evaluate. These are examples of situation when test automation might not be good to be taken for consideration. Efficiency is not reached and automation isn’t offering the excepted additional value in these cases. (Tamres 2002: 226.)
By test automation, regression testing can be made easier with possibility to run already existing tests whenever it is needed and quicker than human to perform it. With automated tests are manually made errors avoided for example typing mistakes and lapses of concentration. To fulfil requirements of the good test automation is an experienced tester needed for defining and creating good test cases. (Tamres 2002: 226.) 2.2.1. The Objectives of Test Automation
Objectives of test automation can be seen as cost and test execution time reduction, greater test coverage, and quality of testing can improve by the action of test automation. And when these objectives are reached it also can indicate greater product quality, product is more reliable and therefore can be shipped to market faster. But at the end, the main objective for automating testing is most of the time the cost reduction.
(Mulder, D. L., & Whyte, G. 2013: 168.)
Moving from manual testing to test automation can improve the overall process of software development. Automation is also a great way to reduce the time on regression testing because the test automation can execute tests in a faster pace than manually executed (Rauf, E. A., & Reddy, E. M. 2015: 949). Rauf’s and Reddy’s research (2015:
955) also stated other benefits of test automation that are for example improvement in productivity, scripts were able to run on different platforms, and generally the test automation can be said to be fast, reliable, reusable and comprehensive.
As mentioned earlier automating testing can reduce the time used in testing. Tests can be set to run independently and tester can use the time for example to run more complex manual tests or increase the test coverage by adding more test cases for test automation.
The provided possibility to run different tasks simultaneously by the test automation saves time and makes testing more efficient and test automation can run the tests without any break. (Cervantes, Alex. 2009.)
One thing that speaks for the test automation is the major possibility to make errors when testing and verifying results manually. Test automation can limit the human factor out. Other fact is the software development is mainly maintenance of existing code and for testing this means repeating the tests, i.e. running regression testing. This needs time
and effort and test automation can be very useful in this task. (Beizer, Boris 1995: 233–
235.)
2.2.2. GUI Test Automation
There is a different kind of tools for automating GUI testing and each tool can provide different kind of ways to approach making the testing more efficient. In this chapter these approaches will be presented.
First approach to be presented is Record and Playback, also known as Capture/Replay.
This method is one of the simplest ways to automate GUI testing. As the name can tell, you first need to record the wanted actions on the screen and then the recorded test case can be played over and over again, when testing is needed. Record and playback tools can record keystrokes or mouse actions and then play the actions in wanted playback speed. Also the mouse movements and clicks can be adjusted to be either absolute or relative to a specific window, this can be helpful because of the window position can change. (Patton 2006: 240–243.)
Next type of GUI test automation is to use test monkeys. This approach is a way to simulate what users could do. Test monkeys are tools that randomize the inputs given to the GUI. This is also why test monkeys aren’t for running planned test cases. There are different states of monkeys: dumb, semi-smart, and smart monkeys. The dumb monkey is the most simplest and most straightforward tool to simulate random user inputs without knowing the software or its state at all. Next one of the monkeys is semi-smart monkey. Compared to dumb one, the semi-smart can e.g. maintain a log file of the actions. This would make it easier to find the cause of a crash bug if needed. The last type of monkeys is a smart monkey. This type of a tool can take the random testing to a more intelligent level. Smart monkey can act more like a user by knowing the states of the software and perform real action without randomly clicking around. This can give an enormous benefit while a big amount of time can be compressed to a fraction of it.
(Patton 2006: 245–250.)
Model based GUI testing is one method of GUI test automation. With this approach it is needed to know the GUI states and events and models will be created based on this information. Model based GUI testing tools are able to run random test cases or then specific known cases if wanted. (Yang 2011: 31.) The GUI test models, which embody
the GUI’s behaviour, are created either manually or by using an automatic tool for this purpose. The test cases can be created with the information of this model that gives the event sequences. A great advantage of model based approaches is the ability to generate concise test cases. (Bae, G., Rothermel, G., & Bae, D. 2014.)
One model based approach is event-flow model, which gathers all the events and their interactions. This model’s behaviour can be compared to control-flow model where the execution paths are represented, or a data-flow model that is representing all the memory location’s uses and definitions. Event-flow model focuses on GUI events and event sequences. This way GUI can be seen as a hierarchy of events and all the different execution paths of events can be tested. (Memon, Atif. M. 2007: 138.)
Artificial Intelligence (AI) methods are taking more and more a bigger step in software testing. The techniques of AI can improve the quality and reduce the costs. Couple of examples of using AI in GUI test automation is test case regeneration after GUI has changed, use of an automated oracle for mapping the user interface behaviour, and AI planning techniques and genetic modelling have been used in improving the automatic test generation. (Rauf, Abdul M., & Alanazi, Mohammad N. 2014.) Also A. M. Memon presents a way using AI planning in GUI regression testing. The primary goal is to detect the changes in GUI and update the test cases by regenerating and rerunning them (Last, Mark, Kandel, A., & Bunke, H. 2004: 52–55).
2.3. Graphical User Interface
Instead of using command line to give input to the software, the user interfaces have been a while around by using graphical user interfaces (GUI) for presenting information and taking user input to the software. The GUI’s nowadays are consisted of different particles and features. These are usually presented as objects, widget or elements. These are common to our day-to-day behaviour for helping to use daily activities and the GUI has for example menus and buttons. The other main elements used in GUIs are windows, scroll bars, drop-down menus, and informative icons. Depending on the GUI, it can be accessed by using the either keyboard, mouse, or, like in mobile devices, using touch input. The exact definition of graphical user interface is presented next: (Xuebing 2011: 49–50.)
A Graphical User Interface (GUI) is a hierarchical, graphical front-end to a software system that accepts as input user-generated and system-generated events which are from a fixed set of events, and produces deterministic graphical output. A GUI contains graphical objects; each object has a fixed set of properties. At any time during the execution of the software, these properties have discrete values, the set of which constitutes the state of the GUI. (Xuebing 2011: 52.)
Picture 3. Different mobile GUIs.
Today’s graphical user interfaces are very intuitive and rich. As presented in Picture 9, the three different screenshots are showing the variety of the GUIs. The banking application (1) is using integrated keyboard. Google maps (2) is simplistic and presenting the map that can be zoomed and swiped by touching the screen. And Android OS 5.1 (Sony Xperia Z3 Compact) is also presented (3) and there can bee seen a basic home screen with application icons, and time and weather widgets.
GUI isn’t just the objects and widgets the end user sees. There is another aspect of GUI when developing a product or application. The GUI can behave differently if the environment changes. This is something to be focused on when developing and testing application. First of the main aspects that are affecting GUI is platform; every platform has its unique features to be taken into consideration. Different operating system versions of the platform are affecting also and on the mobile side they can have major differences that will affect the application. Also in mobile the range of devices is extraordinary, and different screen sizes and specification are affecting too. (Raut &
Tomar 2014: 4–6.)
The software is developed following some architecture and GUI based application, especially on mobile, are normally following Model-View-Controller (MVC) architecture and this architecture has been versioned a lot. The basic idea behind is that the model manages the data storage, view will present the data to the user, and controller is responsible of controlling the user input. MVC can be very adaptive and meet the demands of high usability. (Luostarinen, Manner, Määttä & Järvinen 2010:
51.)
2.4. Approaches to GUI Test Automation
Approaches to GUI test automation will give an exploration to three different approaches of automating GUI testing. Every approach will give example of using test automation via GUI. The first one is focusing on use of framework in GUI test automation. The second one is Pattern Recognition And Computer Vision in GUI Testing. The last one is presenting GUI Ripper –technique for traversing the GUI structure.
2.4.1. Framework Use in GUI Test Automation
Test automation frameworks are used in testing and the purpose of frameworks is to make testers work a bit more efficiency. With a framework you can provide tools and services to support for testing activities. Framework offers an infrastructure for the test automation environment and can help develop test automation solutions. (Cervantes 2009.)
The simplest framework is the Capture/Replay, also can be known as record and playback. In this approach the user gives inputs to the system and these will be recorded. The inputs can be e.g. mouse movements or keystrokes. The recording will create a script which can be played back to run the test. These tests are simple to be created without knowledge in programming. But on the other hand, regression and maintenance can be error prone if the GUI changes and makes the test cases obsolete and fail. (McMaster, Scott, & Memon, Atif M. 2009.)
Another approach is Data-driven scripts. This framework is normally application- specific and the scripts can be created either manually coding or capturing by running the application. Data-driven script will use the given external data to deploy it to the input fields in the application. Negative side of this framework can be the hard coded components or fragile recognition strings that are in the script and these can fail by paralyzing all the scripts or even the whole application. Data-driven scripts framework is quick and easy to implement, but needs dedication in maintenance. (Nagle, Carl 2000.)
Test automation framework can be keyword-driven or also known as table-driven. This approach can typically make the framework application-independent. The tests are executed by using data tables that contain keywords. These keywords are describing the actions wanted to perform. They can also contain data for the application input fields and this data can be used also as a verification data. Application-independent framework will ensure the maximum reuse of the scripts and they can be easily kept updated. All though implementing keyword-driven framework most probably will be hard and time-consuming, but it will show the effort. (Nagle 2000.)
Hybrid test automation framework is combination of data-driven scripts and keyword- driven testing approaches. Keyword-driven architecture provides libraries and utilities for data-driven scripts by making them less prone to errors and to be more compact.
(Nagle 2000.)
2.4.2. Pattern Recognition and Computer Vision in GUI Testing
Computer vision is trying to recognize shapes and details from the picture or object. The optimal case would be when the computer vision can analyse and fully understand the investigated object, like little child can name all the animals from the picture. Humans
can recognize people from the pictures and tell their feelings, or humans can separate and recognize the object from any background. This is partly due to the ability to perceive our surroundings in three-dimensional way, but there hasn’t been found any explanation to how our visual system works. The field of computer vision has been researching for long time and researchers have developed mathematical techniques to work with three-dimensional view and recognize objects in the perceived surrounding or an image. (Szeliski, Richard 2010: 3.)
Pattern recognition has got its foundation in the 1960s and 1970s. With the gathered knowledge the modern-day challenges are tackled with the existing theories, ad hoc ideas, intuition and guessing. With pattern recognition it is possible to point out wanted or searched patterns from the investigated target. Kuncheva presents a classic example of use of pattern recognition where mail needs to be sorted by the digits that are handwritten. Examined target or object will have a corresponding class, which will define more about the found pattern. (Kuncheva, Ludmila I. 2014: 1.)
For recognizing objects or patterns from the picture or video can be difficult and lot of research has been done for computer vision and pattern recognition. Luckily there has been lot of improvement and research has come up with different kind of solutions. One common thing in computer vision and pattern recognition is that there always needs to be a model or description of investigated pattern or image. Because of this Shapiro &
Stockman states that: “many experts will say that the goal of computer vision is the construction of scene descriptions from images.” Problem-centeredness and fundamental issues are still present in the field of computer vision research. (Shapiro, Linda G., & Stockman, George C. 2001: 1.)
Chang, Yeh and Miller (2010) have researched the use of computer vision in GUI testing. They are presenting an approach for automating GUI testing in their paper. This approach uses images to test the GUI by specifying the components to be tested. With given input events and images, the visual test scripts can be generated automatically.
Although the GUI test automation is very error prone task, this presented approach presents Sikuli, a tool that uses computer vision for creating GUI test automation scripts. After creating the test script and given the images, the Sikuli will look for the image from the GUI and execute the given actions. After this the tool will observe the outcome and verify the result or GUI feedback with the specified image. Sikuli provides
also a library of functions to help automating GUI tests. (Chang, Tsung-Hsiang, Yeh, Tom, & Miller, Robert C. 2010.)
Chang et al (2010) approach using Sikuli is a way to use computer vision for GUI test automation. In this approach the tool seeks the object either from the specified area or from the whole window. It is not explained how this tool processes the comparison of investigated object and image.
As Szeliski (2010) and Shapiro et al (2001) describes the computer vision is well researched and it still has taken only baby steps when compared to humans and animals ability to observe and perceive our surroundings. These baby steps are still giving a lot to the science and people in their daily tasks. In GUI test automation it has started to prove some advantages. Most of the comparison tools are comparing the object and investigated image by pixel operation and this can be quite inefficient and needs some level advanced logic behind the comparison process.
2.4.3. GUI Ripper
As known, testing needs test cases, which need to be maintained. Test cases can be created either manually or automatically. A tester does the manual work and automatically created test cases can be done with the model derived from the system specification. This isn’t though that common to use the automatic test case creation because of the overall form of system specifications. For generating a test case to test GUI you can use Capture/Replay tool to interact with the system. It can take 20–30 minutes to generate a normal GUI test case. This kind of test case typically includes 50 different events. (Memon, Atif, Banerjee, Ishan, & Nagarajan, Adithya 2003: 260.)
Memon et al (2003) have described an approach to automatically generate test cases for GUI testing. This technique is called GUI Ripping: “Reverse engineering the GUI’s model directly from the executing GUI”. This dynamic approach can be used to a system’s GUI that is currently active. From the launch of a system GUI ripping starts and it walks through all the windows and paths. It gathers all the available information, like properties and values, from the GUI elements in the current window. GUI’s structure and execution behaviour will be formed with this technique. GUI’s execution behaviour will be represented as event-flow graph and integration tree, and structure as GUI Forest. The information got from the executed system will be verified by the test
designer and after that the model is ready to be used in test case generation. (Memon et al 2003: 260–261.)
GUI Forest is the first ripping outcome of a system’s GUI. Window structure of the GUI is now presented as hierarchic nodes in the forest. Each node contains window’s state and it’s widgets with values, and properties. (Memon et al 2003: 262.)
To be able to use the GUI forest for creating test cases, it is needed to have modelled structures for representing execution behaviour. These structures are called flow of events. GUI is consisting of different components and events are triggering different components. Flow of events is representing this behaviour and a flow graph is indicating a GUI component’s flow of events. Event-flow graph is then presenting all the possible communication between events in a component. Integration tree is then describing the event-flows of GUI components and their communication with each other. (Memon et al 2003: 263–264.)
The approach that Memon et al (2003) described was strong act for creating new technique for generating GUI test cases automatically. Their focus was still quite narrow in the study and didn’t cover the major platforms. Now there as been new development and research, and GUI ripping has reached Android platform as well.
Amalfitano, Fasolino, Tramontana, Carmine and Memon (2012) have presented AndroidRipper, which is extending the work done on ripping. (Amalfitano, Fasolino, Tramontana, De Carmine & Memon 2012.)
With automated technique, AndroidRipper, an Android application can be tested via graphical user interface. In developing this technique the work of model-based GUI testing has been exploited in the study. AndroidRipper uses the GUI ripping technique, which was mentioned previously in this chapter. By traversing the application’s GUI it systematically and automatically gathers the information from new events faced for generating test cases and executes them. This extended, configurable GUI analysis technique can examine the graphical user interface and notice crashes while application is running. (Amalfitano et al 2012: 258–259.)
In their study Amalfine et al (2012) compared the results of AndroidRipper to Monkey, which uses random testing approach, and AndroidRipper showed to be more feasible and cost-effective. Also compared to in detecting defects and covering the code,
AndroidRipper was more successful. The application tested was WordPress for Android, an open-source application. (Amalfitano et al 2012: 260.)
2.5. Testability and Quality
People are nowadays expecting a lot from the devices and software they are using. They have become very quality driven and aware and they are expecting that the software is working as intended. Other point of view is the requirements for the software or application that need to be implemented. When these are filled the product can be considered a quality product.
In following chapters the quality and software testability is presented. These chapters are not focusing on the end user point of view, it though will be presented, and focusing more from the technical side and in overall from the development life cycle side.
2.5.1. Testability
Software development has normally focused on filling the requirements of the version under development. This philosophy may lead to a point when testability decreases.
One way to make software easier to test is to focus on design aspects by maximizing cohesion and minimizing coupling, this will make it easier to test the software. By cohesion it is meant how much the functionality is affecting other modules or functions.
Coupling means how dependent, in number, the module is of others. If the software architecture and code is very coupled, it makes the tests also highly coupled. This brings it to the point where tests need to be updated continually and are fragile. The same situation can be considered when the code cohesion is low. When software has been designed and developed following best practices, less time will be used for unit and regression testing of the system. When the software has the cohesion and coupling on a proper level the goals of higher readability, maintainability, and testability can considered to be succeeded. (Alwardt, Mikeska, Pandorf & Tarpley 2009: 178–179.)
Other aspects of testability have been studied and next some of the main points are presented. The literature review has gathered the information and the data has been break down to four main categories, which are fault tolerance, exception handling, handling external influences, and miscellaneous issues. For fault tolerance there has
