Agile in Waterfall : Improving the Flexibility of Product Development

(1)

Sammy Loitto

Agile in Waterfall

Improving the Flexibility of Product Development

Helsinki Metropolia University of Applied Sciences Master’s Degree

Industrial Management Master’s Thesis

25 August 2012

Instructor: Marjatta Huhta, DSc (Tech)

(2)

Author(s) Title

Number of Pages Date

Sammy Loitto

Agile in Waterfall: Improving the Flexibility of Product Develop- ment

96 pages + 5 appendices 25 August 2012

Degree Master's degree

Degree Programme Degree programme in Industrial Management

Instructor Marjatta Huhta, DSc (Tech) / Principal Lecturer

This Thesis concentrates on improving the existing Waterfall-based product development model used in the case company, to meet the changing needs of product development.

Product development is more and more based on standardized and open source components and competition in any industry has increased as the entry barriers has been low- ered. Therefore adjustments were needed to enhance the flexibility of product development and shorten the time to market for new products and features.

The research approach applied in this study is action research. The model development is done in iterations, in two action research cycles. The data used for the development of the model are collected in the interviews, discussions, a brainstorming session, and a Kaizen workshop in the case company. The workshop participants were selected to get a broad perspective with people from different departments, including product business management, project office, quality management, system architects and system engineering. Al- together 21 persons give their view on product development and the new model creation.

The model development is done by creating and verifying two prototypes, Prototype 1 and 2, which were developed in several workshops (Workshop 1-4) with subject matter experts and key stakeholders. The prototypes and verification of the prototypes led to the proposal of the final model to the case company. The outcome of the Thesis is a proposal for a new product development model based on Agile development principles, combined with the required tools to meet the targeted levels of quality and management visibility applied in the case company.

Key words Product development process, Agile, Waterfall, software development, quality, management visibility, Scrum, Scrumban, Kaizen

(3)

Contents Abstract

Table of Contents List of Figures List of Tables Acronyms

1 Introduction 1

1.1 Business Problem 3

1.2 Case Company 3

1.3 Research Objective and Research Question 5

2 Method and Material 6

2.1 Action Research 6

2.2 Research Design 9

2.3 Research Process and Implementation 11

2.4 Data Collection and Analysis 12

2.5 Validity and Reliability 17

3 Current State Analysis 19

3.1 Standard Product Development Model 19

3.2 Capability Maturity Model Integration for Development (CMMI-DEV) 22

3.3 Release 1 Project 24

3.3.1 Value Stream Map 26

3.3.2 Lessons Learnt 28

3.4 Quality Requirements 29

3.5 Project Visibility 30

4 Best Practice for Product Development 33

4.1 Overview of Product Development 33

4.2 Agile Software Development 37

4.2.1 SCRUM 38

4.2.2 Scrumban 40

4.2.3 Extreme Programming (XP) 42

4.2.4 Crystal, DSDM and RUP 44

(4)

4.2.5 Agile and CMMI^® 47

4.3 Stage Gate Models 48

4.4 Agile to Waterfall Best Practices 50

4.5 Analysis of Vital Elements from Best Practices 52

5 Model Development 60

5.1 Overview of Model Development Process 60

5.2 Prototype 1 61

5.3 Verification of Prototype 1 66

5.4 Prototype 2 68

5.5 Verification of Prototype 2 74

6 PROPOSED MODEL 79

7 CONCLUSIONS AND RECOMMENDATIONS 86

7.1 Summary 86

7.2 Managerial Implications 88

7.3 Evaluation of the Thesis 90

7.4 Reliability and Validity in this Study 91

References 94

Appendices

Appendix 1. Inteviews and discussions Appendix 2. Lessons Learnt session

Appendix 3. Workshop 3 (Kaizen Workshop) Appendix 4. C1 process gate criteria mapping Appendix 5. Product Backlog

(5)

LIST OF FIGURES

Figure 1. The action research cycle.

Figure 2. The action research spiral.

Figure 3. Research design of this study.

Figure 4. Literature review in this study.

Figure 5. Practical flow of the research process in this study.

Figure 6. Iterative process of data collection and analysis in this study.

Figure 7. Simplified picture of the C1 process.

Figure 8. C1 process gates and phases.

Figure 9. Gates and reviews during Release 1 Project.

Figure 10. Value Stream map for Release 1 Project.

Figure 11. Example of a Kanban board.

Figure 12. Extreme programming process life cycle.

Figure 13. The Crystal family names.

Figure 14. DSDM process diagram.

Figure 15. The RUP process.

Figure 16. Example of a five-staged Stage-Gate model.

Figure 17. Prescriptive versus adaptive, in Agile models.

Figure 18. Product development models in a comparison matrix.

Figure 19. Model comparison matrix mapped to the project phases.

Figure 20. Prototype 1 of the development model against the C1 process Figure 21. A feature board example from Prototype 1.

Figure 22. Prototype 2 of the development in the C1 process.

Figure 23. The sprint cycle of Prototype 2.

Figure 24. The proposed new model, presented in gates and phases.

(6)

LIST OF TABLES

Table 1. Details of the data collection for the current state analysis.

Table 2. Details of the data collection in workshop 1.

Table 6. C1 process gates used during development projects.

Table 7. CMMI staged maturity levels.

Table 8. Results from lessons learnt session.

Table 9. Vital elements of the different models.

Table 10. Structure of Section 5.

Table 11. C1 process mapped against the proposed gates in Prototype 1.

Table 12. Issues discussed in workshop 2.

Table 13. Gate naming and their descriptions in Prototype 2.

Table 14. Principal differences between C1 process and the Prototype 2.

Table 15. Product development challenges versus Prototype 2.

Table 16. The 7 features of the new product development model.

Table 17. Issues solved by the new model and enhanced with managerial implications.

(7)

ACRONYMS

ASD Adaptive Software Development AR Action Research

BMS Business Management Suite CE Concurrent Engineering

CMMI Capability Maturity Model Integration

CMMI-DEV Capability Maturity Model Integration for Development DoD Department of Defense

DSDM Dynamic Systems Development Method FDD Feature Driven Development

GMR Gate Maturity Review

IIDD Incremental and Iterative Design and Development IPD Integrated Product Development

KPI Key Performance Indicator NPD New Product Development

OSSD Open Source Software Development PDB Product Decision Board

PDM Product Development Model PFU Project Follow-up

PRR Project Redirection Reviews RAD Radical Application Development

RFx Request for Information/Proposal/Quotation RUP Rational Unified Process

R&D Research and Development SE Simultaneous Engineering SFS System Functional Specification TDP Technical Data Package

TPS Toyota Production System UI User's Interface

VSM Value Stream Map WiP Work in Progress

WP Work Package

XP Extreme Programming

(8)

1 Introduction

This Thesis concentrates on adjusting the existing product development model in the case company to meet the changing needs of the product development process in order to comply with the modern Agile methods to meet the requirement of flexibility.

Product development is one of the basic business processes that can be defined in several ways. Therefore product development has to be defined. Before defining product development, a definition of a product is needed. There are various views on what distinguishes a product and how it differs from a service, and what is the relation of a service and a product. In this study, a broad definition by Ulrich and Eppinger is used which defines a product as something sold by an organization to its customers (Ulrich and Eppinger 1995: 2). As for product development we will use the following definition:

Product development is the set of activities beginning with the perception of a market opportunity and ending in the production, sale and delivery of a product. (Ulrich and Eppinger 1995: 2)

Waterfall is a term used to cover all the development models that are based on a stepwise product development approach. The steps are usually referred to as phases, for example Planning phase, Design phase etc. The different phases are separated by gates. The gates are used to grant access to the next phase, usually the gates include a set of goals or criteria to pass the gate. The main thought behind Waterfall is that a product development project proceeds through the phases and gates in one direction from the beginning to the end. As water flows down a waterfall. The criticism towards waterfall states that there is no room for flexibility and learning during the project.

Agile on the other hand is a term used to cover all the product development models that are based on iterative development. The focus in Agile is on learning during the project and adapting to what is learnt during the project. For example, implementing Feature 1, might expose something that completely changes the need for Feature 2, then Feature 2 can be adapted based on the learning from Feature 1. This is made

(9)

possible by using iterations. Where Waterfall defines design as something done in the beginning of the project, Agile has continuous design in every iteration. Flexibility and Agile are very often linked together.

In this study flexibility is considered in two dimensions. First of all flexibility in terms of shorter product development projects and therefore shorter time to market for new products. Secondly, flexibility in terms of product development project content, meaning adaptability to changing customer needs during projects. With improved flexibility the competitiveness of the case company can be enhanced, with the possibility to fast- er meet new customer needs and the possibility to develop products that are more inline with customer needs.

The nature of today’s business environment puts pressure on management to extend product and service offerings. Managers are aware of the tough competition and need to do all in their power to differentiate their products from other available offerings on the market. In many industries, however, openness in innovation and possibilities brought by new technologies make it nearly impossible to be more cost efficient than the competitor, or have a superior product. Even if the company can manage to achieve some advantage with better quality or by having a cheaper production, it is very likely that the competitors will soon copy the product, and even improve it in a short while. This puts enormous pressure on innovation and product development.

Product development is, thus, rapidly changing now. Models and processes for product development, such as the Stage Gate model where created decades ago. Today, business practice suggests that companies do not compete with inventions any more; they compete with innovating and developing efficiency, product development included. It means that the focus is now shifting from the questions of “How to create inventions?”

and "How to be innovative?" to the challenges of "How to apply the invention?" and

"How to implement the innovation?"

The current economic situation has also led to smaller research and development (R&D) budgets for many companies, a smaller R&D budget makes it difficult to justify early investment decisions. A reduced R&D budget puts considerable pressure on the cost of product development. This fact, coupled with the difficulty of influencing deci-

(10)

sion makers to make an early investment decision, is the reason for this study to focus on improving the product development model. The new economic situation calls for the new model that can provide a shorter time to market interval, avoid delays and better match the product and customer needs. Therefore this Thesis investigates the evolu- tion from traditional Waterfall based product development models to Agile product development models.

1.1 Business Problem

The business challenge of this study is to adapt the currently dominant Waterfall based product development model to become more flexible. This means incorporating elements of the Agile development mode, and by avoiding the negative impacts of the total Agile model.

To address these challenges, this Thesis focuses on the product development model needed for the case company. It starts with the investigation of the standard product development model and also a variation of this model that has evolved from the standard model to meet challenges of adaptability and time to market requirements. It then compares the models used in the industry’s best practices to common Agile development models widely known in the software industry. Currently, the case company uses a standard Waterfall-based product development model in all its development projects, with minor adaptation based on the size of the project. The current model brings certain visibility to the project management, and that visibility should not be reduced when adapting the process. In addition, the current model sets a number of distinct criteria for quality, and it has tools to follow up on cost and schedule targets.

The Thesis, therefore, aims to suggest a new model that would include the best practices from Agile product development while retaining the visibility and quality needs set by the case company.

1.2 Case Company

The case company is a worldwide leader in global security solutions and systems, providing Lead Systems Integration and value-added products and services to civil and

(11)

military customers around the globe. In addition the company develops products for the public safety industry, with the company projects ranging from airplanes to small software orders.

Being a large industrial company, the case company has years of experience in product development, with established processes following the main frame of the traditional Waterfall model. In the case company, this model has been used for every development project, with minor adaptation based on size of the project. The adapted Water- fall model has been successful and, so far, satisfied the needs of the case company due to a number of reasons, including the fact that it has provided a required level of visibility for the management. In addition, the Waterfall model has managed to set distinct criteria for quality, and suggested effective tools to follow on the cost and schedule targets.

Presently, the company uses one process model for all the projects, which needs to be adjusted to better suit particular project types. Although some guidelines have been created for how to adapt the model for smaller projects, there are still no guidelines available for how to deal with Agile development models, which is a present need for improving the case company product development processes.

The case company has earlier tried Agile development models, but the results have been unsatisfactory. The main challenge was the lack of management visibility and control of the product development. Therefore, this Thesis develops a new model that would addresses the management visibility and quality requirements from the case company, in addition to meeting Agile requirements.

This Thesis uses an opportunity to apply the newly developed model to an ongoing software development project in which the software is being developed by a subcontractor using Agile methodology. The project follows the existing product development processes, and the in-house parts of the project also follow the standard procedure, including, for example, Documentation, Testing and Verification, Service Creation and other standard process stages.

From the beginning of this project, it has been identified that the Agile method would provide a fast way to create the required product. The challenge in the project is,

(12)

however, that the Agile method, which addresses very well the way to work within the project, does not bring the same level of visibility for the management to follow the progress of the project. Therefore, the problem of improving the visibility for the management is articulated as one of the foci of this Thesis.

1.3 Research Objective and Research Question

The objective is to bring more flexibility to the current product development model, in terms of shorter time to market and more adaptability to changes during the product development cycle. In addition the objective is to maintain or improve the current level of quality and management visibility.

The research question for the Thesis is formulated as this:

How to meet the targeted levels of quality and management visibility while utilizing Agile development practices in an improved product development model?

This research question results will be applied to develop a model for a pilot project.

The pilot project is a software development project. The first two releases of the product have applied standard processes practiced in the case company, and the goal of this Thesis is to propose how to be more flexible and efficient in the next release of the project while meeting the specific company requirements.

The plan to meet the Research objective is: a) to analyze the shortcomings of the currently used product development model (Waterfall-based), especially in small to mid- sized project focused on software development; b) identify areas where improvement is needed; c) analyze the possibility of adapting the current model to meet the Agile development principles; and d) to propose a model that would benefit from the flexibility of Agile development while keeping the Quality metrics of the currently used model, such as quality requirements and management visibility. As the company already has strong processes and guidelines in place for product development projects the current process has to be taken account.

(13)

2 Method and Material

This section overviews the research method used in this thesis. This section will also present the research design of the Thesis and the data and material collection process.

In addition the criteria for reliability and validity for the thesis are presented in the end of this section.

2.1 Action Research

The research method selected for this Thesis is Action Research (AR). The principles of action research are based on problem solving in cycles where the problem is solved together with the peoplee involved in it. Another definition of action research is research in action, compared to research about action, aiming for the problem to be solved. (Coghlan and Brannick 2010)

The term Action research was first introduced by Kurt Lewin, an American educator and social psychologist. Kurt Lewin used the term action research for work that did not separate the investigation from the actual action taken to solve the problem. Lewin introduced a cyclical process which involved a non-linear pattern of planning, acting, observing and reflecting on the changed incurred in the social situations. (Ferrance 2000: 7) Another educator, Stephen Corey at Teachers College at Columbia University was one of the first to use action research in the pedagogical field. He believed that the value of action research lies in the changes that occurs everyday. He argued that by studying the consequences resulting from taken actions would more likely change and improve existing practices than reading about what others have discovered. (Fer- rance 2000: 7-8)

The cycles of action research are divided into a pre-set and four main steps. The action research cycle starts with the pre-step, context and purpose, investigating the reasons for the research unfolded, at which the context of the research problem and key stakeholders are identified. After the pre-step, the actual action research cycle starts, with the four main stages rotating, namely: diagnosing, planning action, taking action and evaluating the action taken. (Coghlan and Brannick 2010: 22)

(14)

Figure 1 illustrates the stages of the action research cycle.

Figure 1. The action research cycle (Coghlan and Brannick 2010: 22).

As seen from Figure 1, the pre-step sets the scene, by defining the context and the issues to be solved. When the context is defined and the impacting internal and external forces identified, what is left for the pre-step is to define the desired future state.

(Coghlan and Brannick 2010: 21-22)

After the pre-step, the action research process enters the Action research spiral which consists of two or more consecutive research cycles.

The Action research spiral is illustrated in Figure 2.

(15)

Figure 2. The action research spiral (Coghlan and Brannick 2010: 24).

As seen in Figure 2, the first step of the main steps is the diagnosing. The purpose of the diagnosing is to name the issues at hand and to define the actions that need to be done. As the diagnosing will most probably change in later iterations of the research cycle, it is very important to document thoroughly the initial diagnosing as well as the changes that are done in the later iterations. (Coghlan and Brannick 2010: 22-24)

After the diagnosing comes the planning action step. The planning of actions is done based on the outcome of the pre-step and the diagnosing, and is consistent with them.

After that, the plan is executed and the planned actions are performed in the taking action step, which is followed by the evaluating action step. The performed action is evaluated to assess whether the original diagnosis and the taken actions were correct, and what should be feed into the next cycle. (Coghlan and Brannick 2010: 22-24)

Although there are many different variants of definitions and implementations of action research, the fundamentals of actions research stay the same and include these main steps and iterations of them. In this Thesis, the three first steps – setting the research question, conducting the current state analysis and analysing literature review – corre- spond to the pre-step of the action research cycle. The data collection and analysis phase followed by taking and evaluating actions, represent the main steps in the action

(16)

research cycle. Considering the business problem of this Thesis, action research was selected as it best meets the needs of the Thesis.

2.2 Research Design

The research design implemented in this Thesis is illustrated in Figure 3.

Figure 3. Research design of this study.

As seen in Figure 3, the research starts with setting the research question and conducting the current state analysis, followed by the literature review and data collection and analysis. The purpose of the current state analysis is to establish the reasons for

(17)

the research being conducted and to analyse the needs for change. The current state analysis is composed of interviews, discussions and several workshops. The participants to the current state analysis events where selected from different departments and roles, to get as broad perspective as possible. The participants were selected from Product Business Management, Project Management Office, System engineering, Sys- tem Architects, Quality Management, Customer documentation and Verification. By including all of the important stakeholders the new model is easier to implement and the acceptance of the new model will be easier.

Secondly, the results of the current state analysis are compared to the industry’s best practices derived from the literature review. Finally, the results of the current state analysis and the literature review are synthesized to create prototypes for the data collection and analysis part.

As for the first part, the literature review, it investigates various frameworks selected after generic analysis of Agile development principles. To be able to make a comparison between the Agile models and the traditional Waterfall models, theoretical research is made on Waterfall models. One traditional model based on Stage Gates, clos- est to the model used in the case company, is analysed in detail to make comparison to the Agile development models.

The literature review is illustrated in Figure 4.

Figure 4. Literature review in this study.

(18)

As seen from Figure 4, the literature review selects and analyses the development frameworks using 5-10 theoretical sources per framework to understand, make comparison and create prototypes based on industry best practices.

2.3 Research Process and Implementation

The research process in this study is based on the actual action research cycles. It takes the current state analysis and literature review as input to the diagnosis step, which is performed with the members of the project team. The data collection and analysis lead to the prototype creation and verification. The project team creates the prototypes based on output from the diagnosis step. The prototypes are then verified in workshops with the key stakeholders from across different functions of the product development process. The key stakeholders consist of persons directly working with the development process and of persons accountable for business results based on new products and project efficiency. After the verification stage, the results are evaluated in evaluation workshops. Three prototypes are created based on theoretical search and workshops with subject matter experts. Each prototype is then separately verified with the experts and refined in further workshops. The final prototype is verified by the product decision board and, based on the decision board feedback, the proposal for pilot project is created.

Figure 5 presents the whole research design process.

(19)

Figure 5. Practical flow of the research process in this study.

As seen from Figure 5, the research process includes the pre-steps (with current state analysis, literature review, decision points) which end up in creating Prototype 1; and the actual steps of the action research spiral, ending with the proposal for the new product development model.

2.4 Data Collection and Analysis

The process of data collection starts with the current state analysis including a series of case company interviews and discussions, as well as a lessons learnt session conducted by the Quality Manager of the currently ongoing development project.

The details of the data collection for the current state analysis are specified in Table 1.

(20)

Data from (event)

Participants Data, dura- tion

Topics, questions

Documents Interviews,

discussions

The case company experts (4 persons):

• System Architect

• Project Manager

• Quality Manager

• Head of Operational Ex- cellence

3 x 1 hour sessions

Appendix 1 Field notes, minutes and PowerPoint presentations

Lessons Learnt Ses- sion

The project team members (8 persons):

• Quality Manager

• Project Manager

• Product Business Man- ager

• Verification Manager

• Documentation Manager

• Project Manager, for subcontractor company

• Development Manager

1 x 1 hour Data collection session 1 x 1 hour presentation of the results collected

Appendix 2 Field Notes, Minutes and PowerPoint presentations

Table 1. Details of the data collection for the current state analysis.

As seen from Table 1, the participants at the lessons learnt consisted of the project team. The outcome of the current state analysis, from both the interviews and discussions and the lessons learnt material, was used together with findings from the literature review as input to Workshop 1.

At Workshop 1, Prototype 1 was developed. The participants of Workshop 1 consisted of a limited number of the project team members.

The details of Workshop 1 are shown in Table 2.

Data from (event)

Topics, questions

Documents Workshop 1 Selected project team mem-

bers (4 persons):

• Quality Manager

• Project Manager

1 x 4 hour

workshop Prototype 1 creation.

(see Appendix 1)

Field notes, minutes, Pow- erPoint presentations and illustration figures of the model Table 2. Details of the data collection in Workshop 1.

(21)

Prototype 1, which was created in Workshop 1, was presented at Workshop 2 for the company's key stakeholders for product development models.

Table 3 shows the details of events in Workshop 2.

Data from (event)

Topics, questions

Documents Workshop 2

(= Decision Point 1)

Selected members (8 persons):

• Selected project team members (from Work- shop 1)

• Head of Operational Ex- cellence

• Head of Project Manage- ment office

• Head of R&D, software development

• Senior Project Manager

1 hours presentation of Prototype 1

1 hour dis- cussion 1 hours data collection for Prototype 2

Prototype 1 verification (see Appendix 1)

Field notes, minutes and PowerPoint presentations

Table 3. Details of the data collection in Workshop 2.

The end of Workshop 2 also became Decision Point 1. At Decision Point 1, the Head of Operational Excellence gave his approval for continuing to Workshop 3 and his support for starting the preparation for the new project development model to be used at the pilot project. His approval was needed for securing resources for development of the new project development model. At Workshop 2, the list of invited participants to Workshop 3 was also agreed.

Workshop 3, in addition to the main stakeholders within the organisation that were affected by the project development model, also included external mentors from sub- contracting company.

The details of Workshop 3 are presented below in Table 4.

(22)

Data from (event)

Topics, questions

Documents Workshop 3,

Kaizen workshop (= Decision Point 2)

Selected members (19 persons):

• Selected project team members (from Workshops 1 and 2)

• Head of Operational Excel- lence

• Head of Project Management office

• Head of R&D, software development

• Senior Project Manager

• Manager of Verification team

• R&D Project Manager

• Manager of test laboratory

• System Architect 2, not part of project team

• PBM representative 2, not part of project team

• Development Manager

• Senior Quality Manager

• Participants from Subcontrac- tor

• Lean Mentors from Subcon- tractor

2 x 8 hours 2 hour of the pre- paratory event (mentors, subcontractor partici- pant and selected project team members) 1 hour of the retro- spective (mentors, subcontractor partici- pant and selected project team members)

Release 1 Project issues, General projects, Values stream map, root cause analysis and Solution brainstorming (see Appen- dix 3)

Field notes, minutes, PowerPoint presentations, photographs and VSM

Workshop 3 was performed as a Kaizen workshop facilitated by experienced Kaizen mentors. The purpose of Kaizen workshops was to achieve a state of continuous improvement and identify small steps for improvements. This workshop was focused on synchronizing people, finding a common goal, uncovering problems behind problems (root causes), identifying long term solutions as well as the next small steps. The word Kaizen means "Continuous improvement", and the Kaizen philosophy is based on small changes for the better. Instead of looking for a dramatic big innovation the purpose of Kaizen is to identify the small steps that can be taken immediately. The results of following the Kaizen philosophy can lead to significant changes and big improvements, the difference being that the improvements are done in small steps and they never end. (Masaaki 2012: section 1)

As the outcome from Workshop 3, Prototype 2 was created. The Product Business Manager presented Prototype 2 as part of C-1 milestone presentation to the Product Decision Board. C-1 milestone is the first milestone for any project. At C-1 milestone,

(23)

the resources are secured until the next milestone. C-1 milestone also serves as Deci- sion Point 2 for the development process to create a new project development model.

The outcome from the Decision point 2 was used as input for the Workshop 4. At Workshop 4, Prototype 2 was finalized with the recommendations given by the PDB.

The outcome of Workshop 4 was the proposed new model to be used in a pilot project.

The details of Workshop 4 are presented below in Table 5.

Data from (event)

Topics, questions

Documents Workshop 4 Selected project team mem-

bers (4 persons):

• Quality Manager

• Project Manager

4 hours Prototype 2 finalized. Crea- tion of the proposed model

(see Appendix 1)

Minutes, Pow- erPoint presentations, Visio chart, product backlog Excel and quality criteria Excel

Between Workshops 1-3, qualitative interviews were conducted with System Architect, Project Manager, Quality Manager and Head of Operational Excellence in the case company. All the workshops and interviews were documented; with interviews questions and answers collected, and the workshops minutes and PowerPoint presentations created during sessions kept in the project team and researcher’s archive.

Figure 6. Iterative process of data collection and analysis in this study.

(24)

Overall, the process of data collection and analysis for this study can be represented by an Action research cycle and its main stages, as illustrated in Figure 6.

2.5 Validity and Reliability

To be able to correctly measure the results of research, it should be evaluated from the point of view of validity and reliability. Validity measures the appropriateness, the meaningfulness and the usefulness of the research. Whereas the reliability reflects how free from errors and how consistent and repeatable the research is. (Dooley 1995: 77- 78)

One of the measures for improving validity of the research is addressing the question of "Was what was found as a response to the questions originally asked?". This measure of validity is sometimes called internal validity or face validity. In business research and qualitative research such as this Thesis, the internal validity is usually not a matter of concern, as during the research a lot of data is collected about the subject of the study.

The matter of internal validity should reflect if the actual research question(s) was answered. External validity as another aspect of validity answers whether the results of the results would applicable in other contexts or situations. In qualitative research with small samples of data, the applicability to other contexts can be difficult. In this Thesis, for example, the appropriate measure of external validity will be the question of "How transferrable the results are to other projects in the case company?", as the research focuses on only one project. (Quinton and Smallbone 2006: 126-129)

Reliability can be explained with addressing the question of "Would we get the same results if we would the research were done again?". This means that reliability deals with consistency of the research. The reliability of the research can be strengthened by applying the following methods: using different data sources, using different data collection tools, applying well-established theory from one area to another, collecting data

(25)

at different time points, involving different researchers at different points of the research. (Quinton and Smallbone 2006:129-131)

In this study we plan to increase validity by first of all focusing on the fact that the Research Question is answered. In addition the validity is planned to be increased by involving external experts to the model creation phase and by validating if the new model is also applicable to other projects in the case company. The Reliability is planned to be increased by studying a wide range of data from different sources in the Section Best practice for product development. In addition, it is planned to include a broad range of experts from many different fields and from other projects to the model creation phase. The experts will rotate their participation to the different workshops, while some key participants will participate to all workshops and interviews

(26)

3 Current State Analysis

This section describes and analyzes the product development model currently used in the case company, stressing the importance of quality, cost and schedule criteria of the existing model. In addition, the visibility of the project progress is described, as well as the main challenges and shortcomings of the currently used model.

This section also shows how CMMI^® (Capability Maturity Model Integration) is used to guide the current processes and projects in the case company and how the CMMI level is audited to set the criteria for product development process. CMMI defines the quality requirements used at the case company.

The most important quality metrics of the case company studied in this thesis are quality requirements and management visibility. Where the main part of management visibility is provided by the C1 process, the main part of quality requirements are derived from the CMMI.

Finally, this section also discusses how the standard model is used in the current case project.

3.1 Standard Product Development Model

Presently, the case company uses a waterfall based product development process with clear and distinct guidelines. This process includes several sub-processes; for example, for product development projects in the case company a process model called C1:

Product Management process is used. The general process policy document (Van- cayzeele 2010) defines the scope and objective of the C1 process as follows.

The objective of Solution and Product Management Process is to provide a clear framework for the solution and product portfolio management including, requirement development, road mapping and business performance, the governance, and all aspects to develop, maintain and remove products or solutions on time, within budget, at the level of quality and with the level of profitability as in defined business case compliant with Line of Business requirements. (Vancayzeele 2010: 8)

(27)

The C1 process is a model for following up and managing the whole product lifecycle.

The C1 process is illustrated in Figure 7.

C0 C1 C2 C3 C4

C-1 C5 C6 C10

Ramp down

C9 Maintenance Maintenance

Field Validation Verification

Validation Integration

Implementation Definition

Analysis

C1.2 Product Project Management

Figure 7. Simplified picture of the C1 process (Case Company BMS 2012).

As seen from Figure 7, the C1 process starts with Milestone C-1 and ends at Milestone C-10. The Gates from C-1 to C5 are used to follow up on the development project, while Gates C6 to C10 are utilized for phasing out products from the portfolio. (Case company Business Process Management Suite 2012)

In the C1 process, for every milestone there are well-defined criteria set for passing the milestone and get the approval to continue to the next phase of the project. At every milestone, the project specific validity for each gate criteria is evaluated. Al- though this process does allow a certain level of tailoring in the beginning of the project, the case company basically uses the same model for every project. (Case company Business Process Management Suite 2012)

The process policy deployment is ensured by Product Decision Boards. The Product Decision Board reviews the gate criteria and makes sure that the C1 process is implemented in a consistent way. (Vancayzeele 2010)

Since this Thesis is limited to the analysis of the product development part in Gates C-1 to C5, only this part of the model will be described.

The C1-process gates and their descriptions are listed in Table 6.

(28)

Gate Description

C-1 Content proposal ready C0 Project commitment

C1 Commitment confirmation and planning ready C2 Implementation ready and ready for integration C3 Ready for verification and permission to tender C4 Ready for customer deliveries and verification ready C5

Ready for volume deliveries and Field validation ready

Table 6. The C1 process gates used during development projects.

The lag between Gates C-1 to C5 corresponds to the actual phases of a development project. Figure 8 illustrates these gates and the phases between them.

Analysis C0 C-1

C-1

Definition

Implementation

Integration

Field Validation Validation

C5 C5 C1

C2 C2

C3 C3

C4 C4 C1 process, project development gates

Figure 8. C1 process gates and phases.

As seen from Figure 8, the lag between Gates C-1 to C5 includes the following phases:

a) Analysis, b) Definition, c) Implementation, d) Integration, e) Validation, and f) Field Validation. The two phases left out of the scope of this Thesis are Maintenance (after Gate C-5) and Ramp Down (after Gate C-6), which go beyond the product development process. The C1 process is a classic version of waterfall, where product development lifecycle starts with an analysis or discovery phase and continues stepwise

(29)

through several gates and phases to end up in a product release. Figure illustrates how the waterfall based models works, starting from top left and then sequentially pro- gressing towards the bottom right.

3.2 Capability Maturity Model Integration for Development (CMMI-DEV)

This sub-section gives an overview of CMMI and some understanding of the levels of CMMI. It also shows how CMMI and Agile development fit together. CMMI as such lies outside the scope of this Thesis, but as it sets requirements for the product development process, it has to be considered, and the proposed model has to meet the criteria set by the targeted CMMI level.

Capability Maturity Models (CMM) are used to give a simplified view of the world, the goal of CMMs being to provide the essential tools to describe an evolutionary improvement path from immature ad hoc processes to disciplined mature processes with improved quality and effectiveness (McMahon 2010) To follow up and ensure proper processes, the case company uses CMMI-DEV to manage and measure that the processes are properly implemented and used. CMMI-DEV consists of the best practices and development activities for developing products and services; and it addresses the whole product life-cycle from development to maintenance. The purpose of the CMMI- DEV is to help organisations improve their development and maintenance processes for both products and services. (Chrissis et al. 2011: 3-9)

CMMI for development consists of practices for project management, process management, systems engineering, hardware engineering, software engineering and other supporting processes used in product development and maintenance. The CMMI-DEV consists of exactly 22 process areas, with 16 being core processes, 1 shared and 5 development specific processes. The five development specific processes are addressing the requirements development, technical solution, product integration, verification and validation. Chrissis defines the process area as:

A cluster of related practices in an area that, when implemented collectively satisfies a set of goals considered important for making improvement in that area (Chrissis et al. 2011: 20).

(30)

The process areas are made of different components, these components divided into Required, Expected and Informative. The CMMI-DEV does not suggest using every component of every process area; instead it provides the means to tailor the processes for the project and use the applicable components. It is the task of every organisation to map its processes to the process areas in the model. (Chrissis et al. 2011: 19-31)

The CMMI-DEV uses several levels to describe the evolutionary path recommended for the organisation. Levels are also used for rating and appraisal of an organisation or a smaller group inside an organisation. In CMMI, there are two ways to represent levels, continuous and staged. The continuous level makes it possible to achieve capability levels and follows the achievement of maturity levels. (Chrissis et al. 2011: 31-33) Both ways of representing the levels provide means to improve processes, and both provide the same content and use for the same components.

In its practice, the case company uses the representation of maturity levels. In Table 7, the maturity levels of CMMI are illustrated.

Maturity Level Description

5. Optimizing Continuous Process Im- provement

Organizational Innovation & Development and Causal analysis & Resolution

4. Quantitatively Managed

Quantitative Management

Organizational Process Performance and Quantitative Project Management

3. Defined Process Stan- dardization

Requirement Development, Technical Solution, Product Integration, Verification, Validation, Organizational Proc- ess Focus, Organizational Process Definition, Organiza- tional Training, Integrated Product Management, Risk Management, Integrated teaming, Integrate supplier Management, Decision Analysis & Resolution and Organ- izational environment for integration

2. Managed Basic Project Management

Requirements management, Project Planning, Project Monitoring & Control, Supplier Agreement Management, Measurement & Analysis, Product & Process Quality Assurance and Configuration Management

1. Initial Heroic efforts Design, Develop, Integrate and Test Table 7. CMMI staged maturity levels. (Koch 2005)

(31)

As seen from Table 7, the CMMI maturity levels range from 1 to 5. Maturity Level 1 is called initial, and this level is automatically achieved if an organisation can design, develop, integrate and test. Maturity Level 2 is called managed, and it contains 7 process areas, all related to management. According to CMMI, disciplined management is needed before technical processes can have any value. Level 3 is called defined. This level includes 14 process areas. According to CMMI, the 14 process areas, together with 7 process areas in Level 2, provide a full set of disciplined processes for the organisation. Level 4 is called quantitatively managed. With the process areas in place at levels 2 and 3, the organisation is capable of providing statistical data of its performance. The two process areas in Level 4 use the statistical data to give some understanding of the organisation’s performance and the quality of the products the organisation produces. Level 5 is called optimizing. Level 5 has 2 process areas used to guide the organisation to continuous improvement by finding and correcting the root causes of problems. (Koch 2005)

Presently, the case company is certified to CMMI Level 2 and is striving for CMMI Level 3 certification in the near future. Thus, the current project is followed up according to CMMI Level 3 metrics and any new process model also has to ensure that the CMMI Level 3 criteria are met.

3.3 Release 1 Project

Currently, a development project is going on (we call it Release 1 Project) in which the case company is using a standard development model internally and the subcontractor is using Agile methodology for software development. The subcontractor started by using Scrum but after the beginning has changed to Scrumban. At the case company, Release 1 Project is still reviewed using the standard process (C1 process).

Release 1 Project has a challenging set-up which is important to understand before proceeding with the current state analysis. The actual project is led from Helsinki, Finland, and involves two main subcontractors and one smaller subcontractor. The actual product is developed by a subcontractor in Tampere, Finland, with a smaller part of the product being developed in Vienna, Austria. The User's Interface design is done

(32)

by a design company in Helsinki, Finland; while the final verification of the product is done by the case company's office in Élan court, France. In addition, the lead customer for the product is situated in Germany where the case company has set up a customer project to manage the roll-out and the field validation of the product. The customer project also acts as the interface towards the customer and has facilitated workshops between the case company and the customer (from Gates C-1 to C4), where the customer has had the opportunity to see the product grow and comment on the look and feel of the product. The project also included heavy User's interface (UI) development and the lead customer wanted to participate to the development of the UI. The customer workshops therefore also included agreeing on UI concepts.

When comparing the execution of Release 1 Project to the standard process gates, it was discovered that the gates and their reviews did not fit together. The Agile development methods used by subcontractor lead to a situation where the criteria for the gates were not reached before the end of the project.

Figure 9 illustrates the unfolding of Release 1 Project.

Figure 9. Gates and reviews during Release 1 Project.

As seen in Figure 9, Gate C-1 and C0 were reached in accordance with the case company’s C1 process guidelines, while Gate C1 was reached very late in the project. Gates C2 and C3 were reached just before Gate C4, though the time line between C4 and C5 were in line with the C1 process.

The gates were actually planned as shown in Figure 3, meaning there were no additional delays planned. The challenge in Release 1 Project was that the gate maturity did not match the processes, due to the way the software development was per-

(33)

formed. Software development was done using Agile methodology in 3-week iterations, with new features implemented at the very end of the project. The specifications of the last features were done just before the last iteration, just three weeks before reaching Gate C4. In the case company, however, the C1 process expects that the specification should be done already for Gate C1. This mismatch led to a lot of confusion and anxi- ety among the management. The fact the there was a long gap between Gates C0 and C1, the mismatch of project maturity versus Gate C1 criteria and the late arrival at Gates C2 and C3 lead to a considerable loss of trust from the management side. This loss of trust led to a lot of additional work for the project team to ensure the management of the good progress in the project. Altogether, the management required three additional reviews of the project status (shown in Figure 12). These additional reviews forced the project team to prioritise the reviews and put the project schedule at risk.

Moreover, the close co-operation with the customer, where customer could all the time see the progress of the product and refine their requirements, also led to additional nervousness for the management. The customer’s comments led to a sense of new requirements added to the product all the time. Sometimes, the customer’s comment actually led to new requirements. Although it was expected to help the final delivery, to get the comments early enough would help much more for the project team to adapt. Even thought the project team was confident that the customer would accept the delivery and be happy with the product, it was difficult to communicate it to the management. The Agile development model used by the subcontractor really helped to involve the customer to the development and led to the customer "speaking for the product" even before it was delivered to them. It also made it possible to meet customer expectations as some requirements were easier to understand while looking at the actual UI together with the customer.

3.3.1 Value Stream Map

To get a picture of how the actual work in Release 1 Project went on, the product development process was analysed using Values Stream Mapping technique (VSM) and root cause analysis.

(34)

With the help of VSM, it was possible to identify the top issues and challenges faced during the project. After the VSM for Release 1 Project was created, it was used to show and pinpoint the main issues of the project. The top two issues identified with VSM were investigated using the Root Cause analysis to find the reason for the challenges, which was later applied to Prototype 1 to create Prototype 2, both prototypes presented in Section 5.

In Figure 10, the Values Stream map of the Release 1 Project is illustrated.

Figure 10. Value stream map for Release 1 Project.

As seen from Figure 10, the project was divided into three different streams, the actual software development stream mainly managed by the project manager from subcontractor. The customer stream was mainly managed by the project manager of the customer project, and the main stream for all internal work managed by the project manager of Release 1 Project. To manage the three different streams, which evolved to have their own schedules, created a lot of additional work especially for the project manager, but also for the rest of the project team.

(35)

The Value Stream map was used to pinpoint pain points during the project. The actual process of value stream creation is described in Appendix 3. The main issues found were mapped to the two following issues: 1) C1-process does not support Agile development model and 2) Misunderstood requirements.

3.3.2 Lessons Learnt

At the end of Release 1 Project, a Lessons Learnt session was held with all the people involved in the project. The lessons learnt session is part of the standard process in the case company, and its results are used to improve the subsequent projects. The discussions conducted on lessons learnt session is given in Appendix 2.

The lessons learn session held after Release 1 Project identified several improvement proposals for Release 2 Project and generally for the company. The most relevant improvement proposals for this Thesis are listed in Table 8 (not in priority order).

Event Issues discussed

Improvement proposals (towards Release 2 Project) 1. A better tool for live net meetings.

2. A Wiki-page for communication between different sites

3. A new way to report project milestones when Agile software development is used

4. Better communication between different projects about new features in each release.

5. Shorter meetings, teleconferences with customer instead of face to face meetings.

Lessons Learn session

(after Release 1 Project)

Other top issues to consider

1. Open requirements in the contract with customer, requirements were discussed for a very long time with the customer

2. Communication with several parties in different locations requires a lot of time and effort

3. C1 milestone process does not suit well to Agile development model, visibility to management about the project’s progress and process not good enough

Table 8. Results from Lessons learnt session.

In addition to the improvement proposals listed in Table 8, the lessons learnt session identified other top issues during the project. For example, the C1 process and milestones issues mentioned in both lists indicated that the existing product development process did not support the way of work in Release 1 Project. Also in other parts of the project process (for example, in Customer Documentation and Quality Management)

(36)

and the mismatch with the C1 process and Release 1 Project was highlighted as an issue.

3.4 Quality Requirements

Quality requirements make up a significant part of the product development process and play an important role in product development projects and the quality follow up of projects for the case company. The management uses the quality requirements to ensure high quality of projects and the outcome of projects. The quality requirements are often used as key performance indicators (KPIs) in projects. In addition the quality requirements together with other targets are often used as basis for personal incen- tives in the case company.

The guidelines for quality requirements come from the case company process guidelines. The process guidelines are formulated according to CMMI-DEV and define very precisely what deliverables are expected at each gate, the validity of each deliverable decided in the beginning of the project.

The project specific requirements, metrics for them and Key Performance Indicators (KPI) are also defined in the beginning of the project. The usual KPIs utilized by the case company are, for example, the number of test cases performed, the number of test cases passed, reported faults, the number of Critical Faults, Major Faults and Mi- nor Faults, and other indicators. The KPIs are used to set gate criteria for the later gates (C2-C5). An example of a gate criterion for quality requirements is 90% of test cases passed and maximum 0 Critical Faults and 10 Major Faults.

Release 1 Project met the quality requirements according to the initial expectations.

One of the main findings was a relatively low number of bugs (bugs versus performed test cases) reported during the verification phase compared to other development projects. The reason for the low number of bugs was partly explained by the Agile development model used by the subcontractor, with a test automation running every night, finding all the most obvious bugs and partly by the high quality of the developed software (presumably also the result of the Agile development model used). In addition,

(37)

the fact that the specification work was done late, just before implementation, allowed the testing team to take part in the specifications and also the sprint demos to better understand the features to be tested. This fact probably also helped reduce the number of bugs reported earlier due to misunderstanding of the feature.

As a result, all the mandatory quality audits were passed before each of the C1 process gates. But as the audits are tied to the C1 process gates, they did not give the needed information about the project quality. As presented in Figure 9, we see that all the gates after Gate C0 came very late in the project. One of the reasons for the relatively late gates was due to the fact that the project wanted to meet the quality requirements for each gate.

To summarize, the standard quality requirements in Release 1 Project were met, but they did not serve their purpose to provide information about the project quality during the project life cycle, only giving the required information at the end of the project.

This is also one of the key reasons for the case company to either not use Agile development in its integrity and adapt the product development model used to serve the case company specific requirements.

3.5 Project Visibility

As the quality requirements is the tool to monitor the project progress and product quality, the project visibility is the tool to follow that the quality requirements are met and that the project progresses according to plans. Therefore, the required visibility is vital to ensure management that projects and products will meet the customer needs with the right quality.

Currently, the project visibility is achieved by passing the gates in the C1 process. In the case company process, the gates are usually reviewed, first, by the Gate Maturity Review (GMR) board and then, if approved, checked by the Product Decision Board (PDB). In addition, the project manager presents a project status report monthly at the Project Follow-Up (PFU) meetings. The gate reviews are evaluated based on their maturity compared to the C1 process gate criteria.

(38)

In the Release 1 Project, the project visibility was enhanced by additional management reviews as presented in Figure 9. The conclusion is that the standard tools for project visibility were not enough to provide the management with the needed information.

Even with the facts that product development quality was high throughout, the project and quality requirements were met, and software releases were on time, still the project got more management attention than the projects in general. This high attention of the project was partly due to the importance of the lead customer for the product developed, and partly due to the fact that the subcontractor was using Agile development, and not Waterfall-based development which the case company management was used to.

Summing up, the current state analysis described in Section 3 discovered that the Agile development model indeed enhanced the flexibility of the product development project. It helped to meet customer expectations and created a very close relationship between the project team and the customer, which helped communicate and eliminate misunderstandings in requirements and changes needed during the development.

Thus, the results of the Agile product development model were undisputedly positive.

For example, product quality was high, with a low rate of reported bugs; the development speed was extreme and met the extremely strict deadlines which were set by the customer; finally, customer satisfaction was high and created positive environment between the customer and case company and also between the case company and subcontractor. In addition, the created product was appreciated by other customers, which led to the fact that new releases of the product were planned.

Eventually, the positive impact of Agile development has initiated the case company to start looking for possibilities to adapt its current models to Agile development principles and also to start using Agile development models for projects which utilize in-house resources for the development. In addition, the challenges met with project quality requirements and project visibility have revealed the need to investigate how the C1 process could be adapted to better support Agile development models. The C1 process is proven to provide the required quality and visibility and it is adopted by all the organisations within the case company, therefore, it was decided that the use the C1

(39)

process should remain mandatory, and if Agile development is used the only option is to adapt it to the C1 process.