2 Research Methodology
2.2 Research Process
The research process of this study is based on action research and includes a series of steps required to solve the research problem. The flow includes both stage-wise and itera-tive process in the study. The flow diagram below presents an overview of the research process used in the study.
The process uses several steps defined as: Identification of Objective, Current State Analysis, Literature Survey, Conceptual Framework Design, Initial Model Evaluation (itera-tive process), and Final Proposal. All steps can be described in detail here.
12
Figure 3. Research Procedure of this study.
13
The first step of action research process is identification problem. In this step, the re-searcher identifies the problem of the case company. At this point, the rere-searcher defines the objective of this research and enlists problems for the research.
In the second step, the researcher does current state analysis of the case company. The researcher evaluates the current practices and processes used by the case company.
Based on data the researcher evaluates strengths and weaknesses of the case company.
The current state analysis justifies the need of research and supports research problem of the study.
The third step is Literature survey. Based on research objective, problems and current state analysis of the case company, the researcher gathers information from various lit-eratures and internal documents used of the case company. At this stage, the researcher gathers information related with processes used in the case company and steps required to solve research problem.
The fourth step of the research procedure is presenting a conceptual framework of this study. The conceptual framework is based on findings from both current state analysis and literature survey. This step forms the theoretical backbone of the research. This framework is discussed with selected team’s representatives and other stakeholders and used for study.
The fifth step is discussing the conceptual framework with selected team’s representatives and analyzing outcome. The conceptual framework is followed by selected teams with a cycle of planning, acting on the plan, evaluating the outcome and providing feedback. This process can run in several iterations depending upon the goal. The cycle ends when the team meets the objective. In this step the data is collected and analyzed at the final stage.
During the implementation of the process, teams are suggested to perform stages of plan, do, check and reflect. This will help teams to correct its mistakes. The researcher and other appraisal team members can help teams to correct their mistakes and adjust ac-cordingly.
The last step of the research is impact analysis after process implementation. Based on findings the researcher can propose some guidelines. these guidelines can be used by the case company to take further measures. Also, this study can be used as a case study for other companies while implementing the similar process.
14 2.3 Data Collection and Analysis
In this study the qualitative research technique is used. Qualitative data collection meth-ods e.g. semi-structured interviews, observation of documents, questionnaires, and dis-cussion with focus group can be used in this research. Some other techniques such as measurement of impact and consequences can be used for study also. The measurement of impact and consequences will be done quantitatively.
In this study, the data has been collected from various sources. The data has been col-lected from team’s representatives and members by semi-structured interviews, discus-sion and questionnaires. Also, observation of internal documents has been a part of this study. The researcher has selected three teams as part of this study. The data has been collected from teams at different times in this study. The teams are located both in Finland and India. Hence, the selection of teams adequately represents the functional and geo-graphical distribution across the globe.
The data collection of selected teams at first stage is presented in Table 1. At this stage data is collected for current state analysis. The first stage of data analysis was done by appraisal team and the researcher.
Interview Participants Date Duration Documented Case
Table 1. Data Collection Stage 1 for Current State Analysis
The second stage of data collection is presented in Table 2. At this stage the data is col-lected and analyzed after Final appraisal. The second stage of data analysis was done by appraisal team including the researcher.
Interview Participants Date Duration Documented
1 HK (PO) & MR (PM) 2.11.2020 1 hr Appendix 7
2 HR (PO) & MR (PM) 2.11.2020 1 hr Appendix 7
15
3 JG (PO) & JH (PM) 3.11.2020 1 hr Appendix 7
Table 2. Data Collection Stage 2 for Final Appraisal
The third stage of data collection is used for impact analysis post CMMI implementation in the case company. At this stage the data has been collected from the same team. The third stage of data analysis was done by the researcher individually.
Interview Participants Date Duration Documented 1 HK (PO) and his team
19.11.2020 1 hr (discussion Appendix zz
Table 3. Data Collection Stage 3 for Impact Analysis post CMMI Implementation
The following table presents an information about appraisal team which performed ap-praisal of CMMI implementation and workshop team that was involved in reviewing and approving conceptual framework proposed by researcher. The appraisal team was formed of people of EXTERNAL AGENCY, case company and the researcher. The appraisal team worked with selected teams of case company to implement CMMI. The appraisal team reviewed and gave feedback for improvement in CMMI implementation. The ap-praisal team used benchmarking apap-praisal for CMMI implementation. The workshop team reviewed and gave feedback on conceptual framework developed by the researcher. The workshop team was formed of people from EXTERNAL AGENCY and case company.
People Purpose Techniques used Duration
From EXTERNAL AGENCY:
16
The data collection, presented in Table 5, has been done in the form of feedback. The feedback has been presented in the form of recording and documentation of observations for different case projects.
Case Projects Duration Contents Observation Log Case Project 1 26.4.2020
2.11.2020 18.11.2020
Semi-Structured Interviews, Observation of internal docu-ments and artefacts used in the team, Discussion, Presentation and Questionnaires
audio, field notes, internal documents repository
Case Project 2 26.4.2020 2.11.2020 25.11.2020
Semi-Structured Interviews, Observation of internal docu-ments and artefacts used in the team, Discussion, Presentation and Questionnaires
audio, field notes, internal documents repository
Case Project 3 27.4.2020 3.11.2020 19.11.2020
Semi-Structured Interviews, Observation of internal docu-ments and artefacts used in the team, Discussion, Presentation and Questionnaires
audio, field notes, internal documents repository
Table 5. Observation of Data Collected from teams of the case company
In Table 5, the various processes used by different case projects is documented. This list presents an overview of Process Models with specific purpose used by different case pro-jects.
Case Projects Processes and Tools Used
Purpose Contents
Case Project 1 Scrum, Kanban, Version1, Jama, Confluence
Case Project 2 Scrum, Version1, Jama, Documentation, documents, notes
17
Confluence, SharePoint Sprint activities with Kanban board, Defect log-ging, Content management Case Project 3 Scrum, Version1, Jama,
Confluence, SharePoint
Documentation, Sprint activities with Kanban board, Defect log-ging, Content management
documents, notes
Table 6. Processes and Tools used by the case company in Different projects
The number of sources for data collection are enough for researcher to analyze the pro-cess implementation and its impact in the case company. The different techniques used in data collection also helped the researcher to have thorough investigation of the process implementation and its consequences. Since the number of case projects is limited to 3 but people participating in the data collection is significantly enough in number. Hence the data collection can be said appropriate.
The researcher is working in the company, but he is working with EXTERNAL AGENCY in data collection and analysis. Hence, the research outcome is not influenced by researcher being a part of the company. The researcher conducted various activities such as collect-ing data from key the personnel of the case company and relevant information from litera-ture.
In this study the impact of implementation of CMMI in the case company has been done with the same group of people from Table 1 in the form of questionnaires to make sure that the outcome can be truly represented.
2.4 Validity and Reliability
Validity and Reliability of data is an integral part of qualitative research techniques such as action research. It is required that research should be generalizable. It means that the outcome should be replicated anywhere and informative in all situations. Although in ac-tion research generalizability of data is not an important criterion hence the common tools to measure validity and reliability of data is not applicable. Mills (2000) has properly de-fined validity and reliability of data in action research.
18
Generally, Validity is associated with the measures which is used in data collection. It is concerned with the measure which is used in accurate measurement of data. The com-mon processes are followed while measuring validity of data are information gathering and data review by experienced personal. There are five criteria for validity of action re-search: Democratic validity, Outcome Validity, Process validity, Catalytic validity and Dia-logic validity. Democratic validity asks, “whether the various perspectives of participants have been accurately represented in the study”. Outcome validity asks, “whether the ac-tion as outcome emerging from study leads to successful resoluac-tion of the problem”. Pro-cess validity asks, “Is the study being conducted in a dependable and competent man-ner?”. Catalytic validity asks, “whether the outcome of the study works as a catalyst for future action”. Dialogic validity asks, “whether the research process and outcome have been reviewed by peers and experts”. Hence, validity points out the importance of the tool being used in measurement of study and the understanding the worthiness of the study.
(Mills, 2000)
Reliability always comes along with validity. It ensures that the data has been collected over a period and by researchers are consistent in nature. It answers the questions that the test or survey will produce the same result if done at different times with different set of teams. It helps in testing the measurement technique used in data collection and analy-sis. Generally, it is a challenge for any group of researchers because the data collection method cannot be retake. In case of action research, it is required that data collection techniques should give good, consistent and reasonable information from those who are using it. (Mills, 2000)
But ensuring validity and reliability of data collection and analysis is often a major chal-lenge and associated with some risks. Brink (1993) has pointed out some risks or errors associated with data collection techniques used in the study. There is high likelihood that researcher can be biased during research and can make some presumptions that can lead to erroneous data. Also, the participants can have presumptions which can result erroneous data. Also, the situation such as geography, culture, work habits, etc can affect the consistency of data. Finally, the poor and inappropriate data collection techniques can result in inconsistent data. So, the researcher participating in the study should focus on the risks associated with the study and plan to mitigate it.
19
3 Current State Analysis
This chapter focuses on current state analysis of the case company. The benefit of current state analysis is to nail research scope, focus and capture issues and priorities, visualize the work and uncover challenges of the business, and develop a baseline to measure improvements. According to McKay (2019), current state analysis is key of any research to understand the key steps to be taken in building the case company’s awareness for the need of change. It helps in identifying the gaps between the current state and future states. It is very difficult to embark on a journey of change without a baseline. In such case the solution does not address the key business challenges. The current state analy-sis helps in defining the problem correctly. As Steve Jobs rightly said:
“If you define the problem correctly, you almost have the solution.”
This chapter is divided into several sections focusing on current state of selected case projects. The emphasis is on understanding current processes, methods and tools used currently in case company. The selection of case projects is based on functional and geo-graphical distribution of various teams in the organization.
3.1 Overview of Case Company
The case company is an international engineering and service company headquartered in Finland. It was established in 1910 and currently employ more than 55000 people global-ly. The case company is engaged in building and servicing products related with urban life improvement. The product or services are used in building construction. It has operations in more than 50 countries with more than 1000 offices. It provides local services to house builders, owners, designers and architects. The company decided to move implement digital experience to its products and services.
The study has been conducted for software research and development division of the case company. Three teams have been selected for this study. All teams are operating both in Finland and India. The selection of teams is based on distributed location, work culture and team composition. These teams are CT Team (case project 1 from Finland and India), AP Team (case project 2 from Finland and India), and KCE Group Team (case project 3 from Finland). This research has been carried out in collaboration with EXTER-NAL AGENCY which is helping the case company in implementing CMMI model. The data collection and analysis for the current state analysis has been done in collaboration with
20
the EXTERNAL AGENCY. Data is being used in this study with the consent of the EX-TERNAL AGENCY and the case company.
The CMMI model implementation in the case company was started in January 2019. So far three rounds of implementation review have been completed. The implemented review gives an idea about teams achieving the percentage of objectives of CMMI implementa-tion. The success is measured in terms of progress made by teams in different practice areas. The current state analysis of this Thesis will be based on findings after third imple-mentation review. In this chapter, the researcher will discuss the findings impleimple-mentation review 3 (IR3). Hence, IR3 will be used as the basis of current state analysis of the case company in this study. In next subsections, the findings of implementation reviews will be discussed for selected teams describes as case projects.
3.2 Case Project 1
The team selected for Case project 1 is CT Team which is based in Finland and India.
Here, the researcher discusses the current state analysis of Case Project 1. This discus-sion focuses on practices, frameworks or tools used by the team. For this study, the data has been collected and analyzed using qualitative techniques e.g. interviews, group dis-cussion and observation of documents and activities. The findings are documented in Ap-pendix 1 and 4. The current state analysis has been performed based on Implementation Review 3 (IR3).
Current state analysis of this case project reveals that the team is currently using agile practices (both Scrum and Kanban). The team follows Scrum activities such as Sprint Planning, Backlog Grooming, Daily Scrum, Sprint Review, Sprint Demo and Sprint Retro-spectives. It follows Cumulative workflow for monitoring its performance. The team is try-ing to implement CMMI level 3 with agile practices to improve performance and brtry-ing pre-dictability. After third implementation review of CMMI implementation, it has been found that the team has not achieved success and need to improve in many areas of agile prac-tices. Hence the team is not able to achieve CMMI level 3. Still the team need to bring repeatability, maturity and stability in its activities. Hence, it cannot be said that the team has achieved CMMI level 3 capability and maturity. The team cannot ensure the business stakeholders that the team can achieve its business objectives repeatedly. The team lacks some capability to meet the criteria of both organization-centric global practice areas and team-specific practice areas. The current state analysis of the team can be explained with strengths and weaknesses of practices followed by the team. In the following paragraphs the strengths and weaknesses of the team are being discussed.
21
After third implementation review, the team shows capability and maturity in some practice areas. The team is strong in practice areas such as PLAN, RDM and TS. There are some reasons that helps the company to show capability and maturity in the above-mentioned PAs. The team uses a tool (Confluence) to store vital information such as integration test-ing process, Dev, Integration and Release environment details, and master links to all processes. The coding standards and guidelines to write test scripts has also been docu-mented and followed in practice. This information is shared with other teams. The focus of documentation is that all necessary information should be readily made available to all stake holders for reference and usage. It has been ensured that the requirement can easi-ly be traced back. The team has been advised to continue the above-mentioned practices repeatedly.
But the team does not show significant capability and maturity in some other practice are-as. The team needs to improve in the area of Sprint planning (PLAN), monitoring and Control (MC), managing performance and its measurement (MPM), and peer review (PR) of development particularly code. The team does not break user stories into tasks with corresponding efforts efficiently. Time estimation for task is also poor. Hence, remaining work to be completed at any point of time is not known. Also, significant delays are not identified early for appropriate actions. Hence, the team is not efficient in sprint planning.
Further, the team does not justify that its solution meets customer expectations. It does not produce any evidence of any mechanism that verifies coverage of test scripts to prod-uct functionality. The team has not demonstrated the current status, risks and impedi-ments in progress review meetings with management. The best tool in such case is KPI dashboard. But there is no evidence of inference drawn from KPI dashboard to identify areas of improvement. Henceforth, it may not be possible to identify significant delay in deliveries and take timely corrective actions. This way the team may not be ensure that performance objectives has been met in its case. The team also conducts Sprint Retro-spectives regularly. But there is no evidence of focused action items towards process im-provements. Also, the team does not initiate corrective actions for process improvements on time. Such feedback points out that the team lacks proper actions in certain process areas. The team has been advised to improve in the above-mentioned PAs in the follow-ing sprints.
3.3 Case Project 2
The Case Project 2 of this study is AP Team located in Finland and India. In this
The Case Project 2 of this study is AP Team located in Finland and India. In this