Development-time defects of different projects

As established based on literature (see chapter 2.3.2) and chapter 5.2 above, it is beneficial to find a defect as early in the development process as possible and test automation helps find defects at the earliest possible moment during construction time. One of the main reasons for that is the early fault detection when CI system executes the automated tests and gives fast feedback to developers. It also ensures that the changes in the code don’t break any functionality. It raises a question whether the projects that use test automation at KONE SW Center with the practices of “green” projects (chapter 4.1.2):

 Have less reported defects than yellow or red projects and

 Spend less time on defect fixing activities

These two points are investigated and discussed at the following chapters 5.3.1 and 5.3.2. As stated in chapter 4.1.2, project X is considered to be in the “green”

category of test automation whereas project Y has, until April 2014, been in the

“yellow” category, i.e. there has been automated testing but not nearly enough and all of the automated tests haven’t been executed every time a new commit has been made. These projects are again measured against each other as they highlight different processes of testing, are somewhat comparable projects in terms of the project size (in the number of developers) and thus give a hint on test automation’s profitability. Project Z’s numbers of defects are not analyzed here, as their defect management process differs too much from projects X and Y. The results from a conducted survey are however considered from all of the projects.

Defects can be either reported or non-reported at SW Center. Reported defects are usually either the ones found by STL or the ones that have prolonged over a single sprint. The majority of defects that are found and fixed at SW Center don’t get reported, as they are corrected during the two-week sprints, because they are considered to be a normal part of every-day software development. In this section the numbers of reported defects i.e. the ones that are either a) found at STL or b)

prolonged over a single sprin,t are analyzed between the two projects, X and Y.

These defects are stored in the defect management systems QC and VersionOne.

These defects are thus, in this instance, considered to be avoidable rework of the projects or faults that “slip” through in the testing process (see chapter 2.3.2), even though all of the defects found by STL are not ones that could have been found earlier at component testing level of SW Center. The analysis of “faults-slip-through” (see chapter 2.3.2) was however considered too time-consuming in this thesis as it would’ve required the analysis of every single defect that has occurred by an expert of both of the software projects under study. This is why it is simplified that the time use of fixing reported defects is considered avoidable rework.

Reduction in avoidable rework can thus be considered a benefit of test automation, but it doesn’t completely answer the question, what exactly is the relationship between automated testing and the productivity of software development i.e. does test automation actually accelerate software development. It could make sense as test automation seems to lead to finding defects earlier in the process, having fewer defects (especially at STL’s manual testing) and increasing the integrity of the previously working software after changes in it – and have fewer defects overall (see chapter 5.2.2.1). In addition to this, the overwhelming majority of people at the studied projects considered test automation either somewhat important or really important to the productivity of software development (see appendix 2).

The best way of finding out the real impacts of test automation – or any other change in the software development – would be to have a controlled experiment where the velocity (weighed number of done stories) would be measured before, during and after a change. Velocity is already currently measured at software projects, but the velocities of the projects under study are not yet comparable with each other, which is why they cannot be used to evaluate the efficiency of current software projects. Reduction in avoidable rework is thus directly used as the

positive effect of test automation in this thesis work, with the lack of a better measure.

5.2.2.1 Numbers of reported defects

Figure 20 illustrates the cumulative amounts of defects at projects X (red line) and Y (blue line) since the late 2011 as found in defect management systems. These include all closed defects too, which of course are the vast majority of the cumulative defects in both projects. The thick arrow line is a linear trend line that highlights the rate of found defects in the software. As one can see,

1. the total number of defects is much larger at project Y, 2. the finding rate of defects is also much higher and

3. the curve of project Y’s defects had gotten steeper over the time, during the year 2013

Table 22 shows further analysis of defect findings between the two projects during year 2013. A weekly snapshot from every single Thursday was used, because project X uses that time of week as the point when the numbers of defects are being followed. The numbers of open, fixed and closed defects at every single Thursday of 2013 were used in the calculations. “Inflow” means the average number of new defects that were reported during the weeks; “outflow” means the number of defects that were fixed during the weeks. The average numbers of both open and fixed defects at every Thursday were also calculated.

The table shows that project Y’s inflow and outflow of defects are 3,6 / 3,9 times higher than with project X, i.e. there are a lot more defects found and fixed weekly at project Y than a project X. It raises a question whether project Y indeed spends roughly 4 times more time to defect fixing than project X. The numbers of both projects’ defect are adjusted so that project X’s defects are adjusted to 1,0, and the numbers are therefore comparative rather than exact.

Figure 20. Cumulative amount of reported defects at the two projects.

Table 22. Relative number of defects at the projects, when project X’s numbers are adjusted to 1,0. leading to earlier defect notifications and preventing much of the defects to emerge from STL’s manual testing and late verification testing phases (project X)

 Leaving much of the software testing to STL’s manual testing and fix defects as they emerge (project Y in the past)

A

The experience from literature (chapter 2.3.2) and the comparison of defects found at SW Center and at STL (chapter 5.2.1) would of course already suggest that earlier defect notification is a better way. A figure that further highlights this difference in KONE’s case is the number of open defects: project Y had 4,6 times more defects open during 2013 than project X on average. In fact, as stated in chapter 6.1.2, some of the project Y’s defects have previously got updated during the next release (~ half a year later), which is part of the reason why they have tended to accumulate. The number of “fixed” defects i.e. the defects that are fixed but still need to be verified is also two times higher at project Y.

These numbers can also be adjusted to the lines of code (LOC), as project X’s application’s size is roughly about 1,8 times bigger than project Y’s. If the open and fixed defects of projects were divided to LOC, the difference would be even greater: project Y would have had on average 8,4 times more open defects than project X, during 2013. Table 23 shows these numbers.

Table 23. Average number of reported defects during year 2013 based on the size of the code (project X’s numbers are adjusted to 1,0).

Project Y Project X

Open defects 8,4 1,0

Fixed defects 3,6 1,0

Besides the numbers of defects, another important reason for test automation is that the investment in automated tests (if they are maintained accordingly) can be used later on in the project, every time a new commit is made. This same kind of regression testing of the same kind of scope cannot be achieved with manual testing as the execution times differ greatly with automated and manual testing (see chapter 5.1).

There is however a difference in how much STL tests both of the software produced by these projects as there are currently more people focusing on project Y’s testing and project X doesn’t get as much tested manually at STL as project Y does. That means that if STL had more time to test project X’s software, its

number of reported defects would most probably be larger. The same can, however, be said of project Y’s automated testing: if a large number of automated tests could be instantly implemented to the system, they would reveal defects that the (manual) testing activities haven’t so far revealed. This is why the figures described above can be considered relevant in the assessment of the quality of these products.

5.2.2.2 Time use of defect fixing activities

Project Y has had a habit of keeping book of the total work hours that they’ve dedicated to defect fixing for several months. These are the defects that are added to defect management systems QC and/or VersionOne. Project X also kept book of defect fixing activities during the period of two weeks during February 2014.

The results are shown in table 24. They feature the results from four different Scrum teams with the percentage of time used to defect fixing, how many defects they fixed over the time and how many people worked in the team during the time period.

The numbers could allow making estimates of how much the project uses its resources to defect fixing and how much defect fixing costs per a single defect or a specific time period. The following facts and numbers are used to make these calculations:

 There are 24 sprints/year * 10 workdays/sprint = 240 workdays/year

 A single total workday of a developer = 7,5 hours

 The average cost of development: 20 €/hour (a made-up figure not based on reality)

 Number of developers:

o Project X: 14,5 o Project Y: 21

Table 24. Defect fixing at projects Y and X during a number of sprints.

The results are gathered from a small number of sprints (especially at project X) and their validity is thus something that can be questioned for a fair reason. Of course, the numbers of defects presented in chapter 5.3.1 already suggest that there is likely to be a difference between the rework activities of the two projects, and so do these figures as well. The time used to fix a reported defect between the two projects is also something that would be very interesting, but due to the lack of data it cannot be said that project X’s defect fixing is definitely easier than with project Y, even though in this instance it is calculated as 1,8 times less with project X.

Given the numbers of defects at the projects presented in chapter 5.2.2.1 and the figures of table 27 it however seems fair to say that project Y spends relatively more time to defect fixing than project X does. Besides the figures it also seems logical because there’s a difference at the use of test automation; project X is a green project at test automation whereas project Y has been a yellow project. As project Y has more defects emerging from STL and as the defects found by STL are considered more time-consuming to fix (see appendix 2), it can be said with great certainty that project Y spends more time on defect fixing (avoidable rework) than project X. It is estimated based on the figures above and software program managers’ estimates, that avoidable rework at project Y has so far been 33 % and 10 % at project X.

It must however be noted that it is impossible to reduce the time of defect fixing to 0. Defect fixing is unavoidable at any software projects, as defects (based on

their severity) must be fixed to ensure that the software works as it is wanted to, so to just dismiss the defect fixing activity isn’t a solution. It is also not attainable to reach 0 defects, at least when the software is still under development (pre-release), as there are bound to be defects in a software project of a large scale. It is however possible to improve in the rate of defects, and test automation is something that seems to lead to better software quality during pre-release time.

Reduction of avoidable rework in software development projects is something that should lead to acceleration in the software development process, i.e. time-to-market. The upper project management was contacted to make an estimate of the software’s time-to-market effects on profitability in the case of KONE’s software development. The question asked was, what effects a single software development project’s delays can have in monetary terms. It proved to be a hard question to answer as software is only part of the bigger projects regarding development projects of different machines. However, there was one noticeable instance in the near past, as one software project (not using test automation at the time) was indeed a bottleneck during 2010-2011 as a part of the bigger project for 0,5–1 years’ time. That is why the whole project got delayed, and the effects of it are estimated to be multiple times the yearly labor costs of TAT team. Test automation is something that can reduce the time-to-market, especially as it seems to lead to shorter stabilization times i.e. the time from the start of a release test to the point where no major defects appear. This viewpoint was discussed in chapter 2.3.2 and it is also something which has been observed at KONE SW Center.