Test automation versus manual testing

In modern software development, the main question is not as unambiguous as should one use test automation or manual testing? Rather one should think of the level of automation to be used and which levels of testing are feasible to automate [4]. There are many advantages of test automation, such as speed, accuracy, reproducibility, relentlessness, automated reports etc., however maintenance load of test cases, more complex designs and implementation of test cases can make test automation unfeasible to be used in every situation [22]. One should always consider the return of investment (ROI) when determining whether to automate some testing level and which level of automation to use [3].

The main advantages of test automation are almost always taken for granted when speaking on this subject. The following list explains these advantages in more detail.

• Speed is maybe the most important single reason to consider when deciding the implementation of test automation. In the time a human takes for writing a 25 digit command and pressing enter, a modern computer can do it hundreds, thousands or even millions of times. Result analysis of a single line response like prompt or command executed by a human can take about one second and by computer only milliseconds. This kind of advantage is emphasized in regression testing which can contain millions commands and responses. [3, 4]

• Accuracy, a computer will always execute the test case as exactly as it is written. It will not make mistakes. If an error does occur, it is either written in the test case or the result of some external disturbance. [4, 22]

• Reproducibility can be considered to be accuracy plus timing. Even if the test engineer would not make mistakes timing can be the significant factor when trying to reproduce a defect which is caused by some timing issue. Of course, variable timing can be also seen as an advantage in testing, but it should be controlled and not caused by contingency. [4, 22]

• Relentlessness is one of machines’ best advantage over humans and is defined as accuracy combined with reproducibility plus stubbornness. A machine will never give up or get tired and keeps performing its ordered tasks until it is finished or it breaks apart. [4, 22]

• Automated reports and documentation are always taken as by products of test automation, but provide a great advantage, since some cases, they can reveal what will or has been actually tested vs. the specification documentation.

Modern testing frameworks even recommend and encourage making test cases as human readable as possible [25]. After execution, test automation tools usually will generate reports and logs and make it easier to prove what was actually tested and what were the results. [4, 22, 26]

• Provides resources that are needed to test large test contexts simultaneously like thousands of connections to some server application [4]. This can be

extremely difficult or impossible to manage manually and to successfully open sufficient amount of connections [4]. These repetitive tasks call for test automation.

• Efficiency. A test engineer’s talent and capability is not fully used if one will be tied up in following test executions. Test automation frees up test engineers to do more valuable tasks, like test case design and result analysis etc. It also improves motivation and job meaningfulness, if routine tasks can be left for machines. [3, 4, 22]

Test automation is less advantageous with the more complex test case designs, which will need different points of view for testing, need for possible new investment for test automation equipment, and competence to operate new systems. Also maintenance of test cases is one field that needs more focus than with the manual testing, where maintenance is usually done within the testing routine [22].

The levels of automation can be divided into three distinct categories: full automation, semi-automatic and manual testing. In the full automation level, all procedures are automated, except for thedeeper analysis of failed cases which is done manually. With a successful outcome, the full automation level process does not require any human interaction. The setup, test execution, and teardown phases are all automated, and starting, ending and possible next phase triggering is handled by the testing system. In semi-automatic testing, at least one of the testing phases of the setup, execution or teardown must be done manually if too complex or fluctuating to be automated. The manual testing category contains test cases which are not in any way feasible to be automated. One-off, usability and exploratory tests are good examples of this category. One-off test cases are tests that are intended to be executed only once or very few times. This is because the functionality or feature under test is only a temporary solution and will be replaced or erased soon resulting in an ROI too low to justify its automation. Usability tests usually are included in this category, because user interface tests are difficult to automate due to results being based on empirical factors and can be extremely hard to describe in any deterministic way. User interface is a part of software which is constantly changing, with little

variation. Exploratory test descriptions already reveal reasons why those test are not practical to automate, based on exploring and testing software and designed tests not already applicable. If any defects are found through exploratory testing, there can be efforts to make automated test cases for those specific cases, but all preceding exploratory tests and results must be carefully documented in detail for reproducibility reasons. [22, 26, 27]

In designing a test automation strategy there is a good model, having a shape opposite the V-model, which is used to show the importance of ROI and the need of test automation in certain testing levels. The model is called Mike Cohn’s Test Automation Pyramid [22] shown in figure 7. The pyramid can be used as a guide on where to invest test automation resources. [22]

Figure 7: Mike Cohn's Test Automation Pyramid

As from the figure above, it can be observed that the unit and module tests are the base of test automation pyramid. After an acceptable level of automation is achieved at a certain level, one can start to automate next level test cases. Unit and module level tests should be automated as a first thing, and as much as feasibly possible.

Normally, tests are written with the same language as the software and by the programmers themselves. The next level, functional and system tests, is also important, but if the preceding base of unit and module tests are leaking, this level’s number of caught defects will skyrocket, and it will be come inefficient. This level should be only used to catch architectural defects. The user interface level should be the last level in which to invest resources, because ROI is the lowest at that level.

Manual tests are shown as a cloud, because there is always possibility that all test cases are not possible and feasible to automate. [22, 28]