An evaluation on client vs. vendor perspectives for risk assessment in software projects : a systematic literature review on the main perspectives of bespoke software development projects' risk assessment

(1)

An evaluation on client vs. vendor perspectives for risk assessment in software projects.

A systematic literature review on the main perspectives of bespoke software development projects‘ risk assessment.

Sharon Clarissa Guardado Medina

Master's thesis

School of Computing Computer Science

February 2016

(2)

ii

UNIVERSITY OF EASTERN FINLAND, Faculty of Science and Forestry, Kuopio

School of Computing Computer Science

Student, Sharon C. Guardado Medina: Evaluation on client vs. vendor perspectives for risk assessment in software development projects.

Master‘s Thesis, 59 p., 4 appendixes (20 p.)

Supervisor of the Master‘s Thesis: Ph.D. Paula Savolainen.

February 2016 Abstract:

This thesis focuses on the research and evaluation on risk perspectives of the main parties involved in bespoke software projects, explicitly those involving clients and vendors. Over time software projects implemented by external vendors have grown in number and complexity, requiring companies‘ and vendors‘ active participation and commitment.

In order to corroborate the prevalent risk assessment perspective present in related literature, a set of twenty relevant articles was selected. The search protocol included guidelines and criteria for the articles search, selection and review.

The results confirmed that the client‘s viewpoint has primordially been used to assess risk factors associated with software project, whereas the vendor‘s one has been overlooked.

The review included most recent articles, in addition to some of the most cited on the topic. The data from each of those articles was extracted and analyzed. Later into the research the risk factors were re-evaluated in an effort to validate whether there would have existed any variance on the risk assessment body of knowledge having those risks been approached from the vendor‘s perspective. It could be concluded that while the vast majority of factors were identified from a client‘s perspective, most of them also concern the vendor directly.

Keywords: Software engineering, project risk, risk management, software develop- ment, bespoke software project, client and vendor perspectives.

CR Categories (ACM Computing Classification System, 1998 version): D.2.0, K.6.1

(3)

iii

Foreword

This thesis was done at the School of Computing, in the University of Eastern Finland during the course of two years, in which I had the opportunity to participate as a research assistant. I would like to acknowledge the contributions others have done directly or indirectly to support me in the completion of this Master‘s thesis.

First of all I want to thank God, for I believe each opportunity that has come to my life it has been by his grace. He has put wonderful people in my way, which have been key factors to achieve this goal I set many years ago.

I would like to thank my family that has been an integral part of my life, support- ing me unconditionally throughout this process. I specially thank my father for his mo- tivation, trust and insistence, even when at times I felt like giving up, he was there try- ing to encourage me to continue. To my loving husband, who has been my main support and who took care of our son during the nights and weekends I tried to make some writ- ing progress. I love you.

To the staff in University of Eastern Finland (UEF) who so gently assisted me during the time I studied there, it was an amazing experience to get to study in such a fine institution and in such a beautiful country.

I want to specially express my immense gratitude to my thesis supervisor, Ph.D.

Paula Savolainen, acknowledging her effort, interest and support. Not only did you helped me to find a wonderful place to carry out my IT project, and took the time to patiently go through all my thesis drafts, giving valuable observations and suggestions, but you gave me the opportunity to work with you and to dip into investigation, where I learned a lot and had the chance to put to practice so many different skills. Thanks for your genuine interest, for sharing your knowledge and time. I learned a lot from you.

Thanks to you all, February 2016 Sharon C. Guardado

(4)

iv

List of abbreviations

ERP Enterprise Resource Planning ISI Institute for Scientific Information IT Information Technologies

JCR Journal Citation Report

PMBOK Project Management Body of Knowledge PMI Project Management Institute

PRINCE2 Projects in Controlled Environments 2 SLR Systematic Literature Review

SME Small and medium-sized enterprises UEF University of Eastern Finland

(5)

v

List of Figures

Figure 1: The software acquisition problem ... 10

Figure 2: Parameters for articles search in Scopus. ... 21

Figure 3: Process of Scientific journals‘ selection ... 29

Figure 4: Search and Selection Process for articles ... 30

Figure 5: Professional background of the studied populations ... 44

Figure 6: Perspective of the studies ... 45

Figure 7: Context of the software development projects studied ... 45

Figure 8: Type of risks by party‘s perspective. ... 48

(8)

viii

List of Tables

Table 1: Distribution of articles by category. ... 22

Table 2: Inclusion and Exclusion criteria for studies. ... 25

Table 3: Summary of evaluation results for most cited and newest articles. ... 32

Table 4: Summary of evaluation results for Google Scholar group. ... 34

Table 5: Fields for data extraction from selected articles. ... 36

Table 6: Fields included in the matrix for risks extraction. ... 37

Table 7: Articles included in the SLR. ... 39

Table 8: Top risks identification by date of publication. ... 41

Table 9: Profile of articles introducing external sources for risk. ... 49

(9)

ix

List of Appendixes

Appendix 1 - Data extraction form 1: Search results by journal ... 60 Appendix 2 - Data extraction form 2: Preliminary list of candidate studies ... 62 Appendix 3 - Data extraction form 3: Preliminary list of candidate studies from Google Scholar ... 69 Appendix 4: Data extraction form 4: Selected studies description and analysis ... 72

(10)

1

1 Introduction

The present thesis focuses on the evaluation of software development projects‘ risks, which included primarily the identification process of risks within a software project, as well as the main perspective used for such identification in literature.

This research started as an initiative of the Kytkin Project, which concentrates on the study of software development projects from the suppliers‘ perspective and on software projects‘ startups and success.

Bespoke software projects have been carried out for several decades and even though they are executed in different contexts and industries, the vast majority of them ultimately share common risks that threat their performance and success. Those risks and their management approaches and strategies have been subject of multiple articles and the experts in the field have conducted studies with the aim of classifying and cata- loging such risks. For years the main focus of those studies has been mostly set on the client‘s side of the story, since traditionally it has been perceived to be the main protag- onist of such ventures. As Levina and Ross (2003) stated over a decade ago, while re- searching on Information Technologies (IT) vendors‘ value proposition in outsourced ventures, they were not aware about any empirical investigations being done at the vendors site, situation which limited the understanding of outsourcing outcomes as the vendors‘ perspective remained unexamined. However this state of affairs seems to have slowly started to shift direction in the last few years, as a couple of recent studies have noted that the customer and the supplier may have different perceptions of topics as risk, risk management, and project success (Savolainen et al., 2012).

For many companies the decision to develop bespoke software through an external vendor comes when they decide they do not have the time, the human resource or the installed capacity to perform the software development internally. At this point the company deciding to embark in a journey which is expected to produce a positive outcome and will allow the company to reduce and control IT related costs ((Levina and Ross, 2003), however there are key elements that might affect the way these projects are carried out and consequently their final outcome. These elements are associated to as-

(11)

2

pects as a sound planning and the involvement and commitment of the multiple stakeholders implicated in the project. Different stakeholders and parties might have or represent different interests within a project. Therefore diverse levels of commitment and involvement are expected to come into play, depending on the role they play and the benefits they are intending to gain from the project implementation; nevertheless, in theory all stakeholders and parties involved share the common interest of achieving a successful project, or at least avoiding a failed one.

Some of the points of views that are likely to converge during the project planning and effective live are those of project sponsors, end-users, senior management, suppliers, project managers, and the development team. Considering that every party involved will appoint its own project team, it is probable that more than one project manager, namely, one from the client‘s side and one from vendor‘s side, as well as multiple senior executives, developers, users and several more stakeholders from either side can coexist in a single project. Consequently, for the purposes of this review, it will be understood that henceforth the parties involved in a software projects, their stakeholders and their resources will be clustered as either client-customer‘s side or vendor- supplier‘s side.

Nowadays the trend of contracting external vendors for bespoke software development seems to be more widespread, bringing as a result more parties and stakeholders to the scene of software development projects. Whereas the risks identification and management are key features to guarantee that the expected outcome of a project will be reached, those processes have to be undertaken by both the client and the vendor, as the main actors involved in outsourced projects. Both parties might perceive and approach these processes in different ways.

In outsourced IT ventures both the client and vendor share the responsibilities for managing the project. In this regard, problems might arise from the differences between their goals and structures, which cause each side to feel vulnerable to opportunism or shirking of responsibilities by the other. The two sides may therefore pull project coor- dination in different directions. These factors, along with difficulties as obtaining quick feedback, meeting frequently, and building interpersonal relationships, make the man-

(12)

3

agement of outsourced development projects an arduous task for both parties (Sabherwal, 2003)

Nonetheless, it appears that the consideration on how the vendors identify and approach risks has been almost entirely overlooked in most of the relevant literature on the topic, as well as in most topics connected to projects and services outsourced to IT vendors. Moreover, while the outsourcing literature has extensively discussed subjects related to software development acquisition from the customer's perspective, little attention has been paid to research from the supplier's perspective (Taylor, 2007; Savolainen et al., 2012). Consequently, academics have started to recognize that the relationships between uncertainty, risk management and project performance, among others associated issues, need to be empirically examined from the vendor's perspective as well (Jun et al., 2011).

This thesis aims to confirm those affirmations with documental support and to weigh whether the lack of vendor‘s participation (if confirmed) in the body of research does represent a significant difference in the way software development projects risks have been approached and assessed.

Although bespoke software projects have been implemented by external suppliers for several decades, over time this type of bilateral endeavors have grown in size and number, becoming more and more common for all type of companies and organizations.

As a result, the general literature on the topic has as well increased, being possible to find several hundreds of results when making electronic searches for risks in software engineering related projects.

In my own experience in both private and public sector for several years in my home country I witnessed how a high percentage of bespoke software projects in those environments tended to fail, and how in our context, this phenomenon seemed to be particularly alarming in the government sector.

Empirically, it could be observed that numerous cases of projects failed, which after further inquire, seemed to have been destined to fail from the beginning. Some of those projects might have been pushed into the institution for political reasons; others

(13)

4

intended to replicate the successful experiences obtained in other countries, but without taking into consideration the cultural and contextual differences than originally contrib- uted to the model project success; In other cases multiple risks as lack of managerial support, staff instability and lack of commitment by users, among other, haunted the project during its lifetime. It is imperative to mention that in the previously mentioned context, all the failed software projects that I became acquainted, were bespoke software projects carried out by external suppliers. A great portion of the bespoke software projects‘ failures seem to have started in the conceptualization of the project and have got- ten worse as additional risks raised from all parties involved and their corresponding stakeholders.

Generally speaking, while the context in which software projects are conceived in most developing countries‘ environments might slightly vary from the context in industrialized regions, the overall execution process is fairly similar, being very much the same process described in literature, which have, for the most part, been written in developed economies context.

This research was proposed with two main objectives, the first of which was to identify relevant sources and information written on the subject of bespoke software projects and their associated risks. The second objective involves a deeper evaluation of the perspectives from which the software projects risk factors have been assessed.

It might seem a little improbable that experts on the subject, authors of studies and papers, have disregarded the importance of considering the vendor‘s involvement in the project risk‘s management process. However, the overall conclusion that can be drawn after reviewing available literature based on studies associated to software project, including those in outsourced or offshored contexts, is that the client‘s perspective has been the most widely studied of any of the involved parties‘ perspectives, while efforts in theory building that support vendor‘s context have been scanty, making it hard to find any empirical studies which contemplate subjects associated to the vendor's perspective (Savolainen et al., 2012; Aundhe and Mathew, 2009).

This thesis is organized in eight main chapters, which are distributed and described as follows:

(14)

5

Chapter 2 briefly presents basic concepts and a theoretical framework meant to of- fer an elementary background of the subjects being studied; among them general definitions on project‘s risk management, software projects risk management, risk factors associated to bespoke software projects and main project parties' perspectives, amog others.

Chapter 3 explains the identified research problem and the proposed research questions connected to it, which are meant to be answered using the results of the con- sequent research and analysis.

Chapter 4 and chapter 5 describe the methodology used for the research execution component of the present thesis, namely the conceptualization of the systematic literature review process. Chapter 4 explains the structure of the Search Protocol, which includes the selection and review criteria for sources and articles, as well as the articles search, inclusion and classification, along with their quality assessment. Chapter 5 sets forth the results of the sources and articles search and selection, through the application of the previously defined protocol.

Chapter 6 refers to the processes and tools that assisted in the process of the data extraction from the articles included in the literature review and the posterior analysis carried out using that information.

Chapter 7 presents the main findings, generated from the analysis of the data extracted from the 20 input articles. The analysis included actions as risks classification by significance, description of the original research populations‘ characteristics, elucidation of the identified parties‘ perspectives on projects risks‘ assessment and finally an evaluation on risk factors‘ primary origin.

Chapter 8 acknowledges the limitations found throughout the research and the results report process which might have affected the results or that could be considered and prevented in subsequent investigations.

Chapter 9 as the final chapter of this thesis concludes with a summarization of the main results of the research and the most relevant findings produced by the overall investigation process.

(15)

6

2 Theoretical framework

This chapter procures the clarification of the central notions related to the topics dealt within the present thesis, which will be used extensively throughout this document and in the same way involves the basic concepts on the field of action of projects risk management in software development ventures. A theoretical framework about the fundamental perceptions and definitions of project‘s risk management and risk management oriented to software development projects is laid out, as well as the general definitions of bespoke software projects and a brief portrayal of the major parties taking part in such endeavors.

2.1 Projects risk management

Although risks management is a practice that has been evolving over the last 50 years in the project management discipline, having connotations in a wide range of fields, it is pertinent to briefly revise the basic concepts on the subject.

According to the Project Management Institute (PMI), the risk management practice can be defined as: ―The act or practice of dealing with risk. It includes planning for risk, identifying and analyzing risks, developing risk response strategies and monitoring and controlling risks to determine how they have changed.” (Kerzner, 2013).

The practice of risk management has proven to be particularly vital in the field of project management, which is considered in the Project Management Body of Knowledge (PMBOK) as “the application of knowledge, skills, tools, and techniques to project activities in order to meet or exceed stakeholder needs and expectations from a project” (Project Management Institute, 2001), hence, in order to meet or exceed the different stakeholder‘s needs and expectation, especial attention should be granted to the actions designed to prevent threats that could negatively affect the outcomes of a project.

It can be assumed that, independently of the discipline or type of industry where a project is executed, all projects have the common characteristic of being unique, time-

(16)

7

constrained endeavors, which have fixed budget, with precise scopes. In this regard, Projects in Controlled Environments 2 (PRINCE2) defines it as: ―A temporary organi- zation that is created for the purpose of delivering one or more business products ac- cording to an agreed Business Case‖ (OGC, 2009).

Similarly, the PMBOK describes a project as ―a temporary endeavor undertaken to create a unique product or service”. Those same characteristics that make each pro- ject unique, entail risks that might as well produce deviations or variations that may considerably affect the aftermath of other characteristics, and even the final result of the project as a whole, altering its initially expected outcome or purpose.

2.2 Software projects risk management

The evaluation and classification of software project‘s risks has been the aim of previous studies in the past three decades. The first relevant articles on the subject started to be published in the 1980s and their production increased in the 1990s. As a result of the research done, several methodologies, frameworks and strategies have been developed and proposed, using diverse approaches to identify and analyze the risk factors in software development projects and its correspondent management.

In order to reduce the occurrence of possible risks that could affect a project during it execution, PRINCE2 proposed methodology for risk management contains three dimensions: risk management strategy to define how risk management will be embedded in the project management activities, what is the risk tolerance and when is the ex- ception triggered; risk‘s register as a tool for capturing and maintaining information of identified threads and opportunities; and a risk management procedure (OGC, 2009).

Implementing the previous methodology, or any other risk assessment and control methodologies, is a task that can differ from project to project, provided that the size, importance, complexity, type of project and level of risk for each venture will be different.

Software development projects, due to their inherent nature, seem to be particularly prone to failure. It has been assumed that up to one third of software development

(17)

8

projects end up in total failure or are otherwise abandoned before their completion (Ewusi-Mensah, 2003). According to the Standish Group¹ CHAOS Report of 2009 which is based exclusively on the initial estimations of the project scope, budget and schedule, only about 32% of all software project achieved to be completed successfully (Dominguez, 2009). The remaining projects, although managed to be completed, were challenged during their development, which probably pushed the decision making level to extend the schedule or budget or to settle for less features and functions than initially required.

In many of the failed projects cases, as postmortem analyses (Ahonen and Savolainen, 2010), post-abandonment reviews (Ewusi-Mensah, 2003) and other posteri- or studies have shown, there were significant symptoms or early warning signs of trou- ble which provided indication of manifesting risks (Kappelman et al., 2006), however they might have overlooked or not been properly identified or managed while the project was ongoing.

The lack of proper risk management is one of the leading factors why projects failed (Cerpa and Verner, 2009). A single risk that have not been properly identified and managed may affect a project, up to the point of turning it into a total failure. However, the action of identifying the possible risks that could threaten a project does not auto- matically guarantee that the project team will be able to successfully control, manage or mitigate them.

It is not possible to assure with all certainty that the projects which have failed due to inadequate management of risks would have otherwise been successful if the crucial risk factors would have been properly identified and managed. However, identifying the main causes of previous failures in similar projects can represent an advantage and lessons learned that might assist when managing related ventures in the future. In this regard, Ewusi-Mensah (2003) affirms that the chance of major software failures, is

1 Is an independent international IT research advisory known from their reports about information systems implementation projects in the public and private sector.

(18)

9

likely to be significantly reduced, or in some cases the failure may even be avoided, if organizations and individual members of project teams make a concerted effort to constantly re-examine the circumstances surrounding each failed project and make the lessons learned part of the organizational record of improvement in software practices.

2.3 Risks in Bespoke software projects

Bespoke software is also known as tailor-made, tailored, custom, customized or custom- developed or custom-made software, and the concept encompasses all varieties of software that have been specially designed and developed to meet the specific needs and requirements of a particular institution, organization or company, obeying the guidelines and requirements that have been dictated by the customer‘s organization specific preferences, needs and expectations. This type of software is typically selected when no packed software products are available, or when the business processes are highly specialized or when numerous systems need to be integrated. (COE, 2008).

Decades ago, most of the software development, habitually custom-made software, was performed in-house by internal development teams, since software was highly specialized and commercial off-the-shelf options where not so widespread, therefore for most companies, little or inexistent intervention of external vendors was observed in software development.

With time, platforms became more common and companies started to focus more on productivity software. This encouraged the requirements for software development to change. Instead of having a small dedicated group of programmers for particular projects, software development ventures soon required massive development teams as software requirement list became increasingly larger, and with them bigger project management teams also entered the scene. In many regions in industrialized countries, this led to a significantly increased demand on the skilled work force and a relative re- duction in overall profit for companies. This trend impacted directly in the growth of outsourced custom projects and engagement of companies with software vendors to complete the activities related to software development.

(19)

10

Figure 1: The software acquisition problem (Nelson et al., 1996)

In recent years the decision of how to acquire software is one numerous companies deal with on a regular basis. Such decision can be handled through different viewpoint; as an example Nelson et al. (1996) depict in a basic way the main elements that interact when a company ponders the decision of new software acquisition. In Figure 1 the software acquisition problem is presented as a quadrant where the acquisition team and approach are the fundamental edges of the decision. According to investigations done on software acquisition, coding and installation costs seem to be higher when software is developed internally, because external markets have more competitive sources of labor and technical expertise, and they are in a better position to take advantage of scale and scope economies.

A number of studies indicate that the leading reason behind outsourcing is the need to reduce and control IT operating costs (Levina and Ross, 2003). Consequently, in an effort to reduce cost and maintain competitiveness, the relationship with software vendors has become a daily matter for companies choosing to outsource their software development projects.

Though ascertaining the world market share of the bespoke software can be a dif- ficult task, since there are numerous variables that can be affected by the region, indus-

(20)

11

try or size of projects, among other aspect, the Forrester Research² indicates that in developed economies enterprises spend about the same on custom-developed business applications as on packaged business software (Ried, 2015). According to their findings, the share of custom software stands at 25.6% of the software market. However, while bespoke software market seems to have remained constant, frequently the types of projects falling in this classification have major associated risks due to the implications of their bilateral or multilateral relationships.

2.4 Differentiation on risk assessment perspectives

When software is developed through a contractual project there are parties which have different perspectives and different goals (Savolainen, 2011). For many companies the decision to outsource such projects comes when they ponder their options and decide they do not have the time, the human resource or the installed capacity to perform the software development internally. This condition alone might represents an initial chal- lenge for the project, considering that the contracted vendor staff might as well not have the level of expertise, previous experience on similar projects or the knowledge about the company procedures or specific tasks to be automatized, which are expected by the company; acquiring such knowledge and qualifications might, in some cases, take much more time than initially estimated by the company, the vendor and their project managers.

Furthermore, there is a key element that might affect the way the projects are carried out, and this element is associated to the involvement and commitment of the multiple parties and stakeholders related to the project. Different parties and their corresponding stakeholders might have or represent different interests that converge within a project. Therefore diverse levels of commitment and involvement are expected to come into play, depending on the role each play and the benefits they are intended to gain

2 Independent technology and market research company that provides advice on existing and potential impact of technology. It has five research centers in the US, four in European and four research centers in the APAC region.

(21)

12

from the project execution; however, in theory, all stakeholders and parties involved share the common interest of achieving a successful project, or at least to avoid a failed one.

Thus, during the project life several points of views can converge, such as those of project sponsors, end-users, senior management, suppliers, project managers, and the development team. Consequently, in this type of contractual arrangements more than one project manager, namely, at least one from the client‘s side and one from vendor‘s side, as well as multiple senior executives, developers, users and several more stakeholders from either side can coexist in a single project.

For the purposes of this review, it will be understood that henceforth the stakeholders and their resources will be grouped in two main parties identified as either client/customer‘s side or vendor/supplier‘s side.

Among all possible perspectives involved in a project, the fundamental one, and perhaps the most widely studied when it comes to software projects risk management, is the one of the client, which in one way or another, needs to be embedded in every single bespoke software project.

Moreover, it has been identified through research conducted on the subject by the Kytkin project staff, that little effort has been invested in consider a differentiation of software risks factors, based on the diverse parties‘ and stakeholder‘s perspectives, overlooking for example the vendor‘s perspective. Most authors have used a combina- tion of both the client‘s and the vendor‘s perspectives in their studies without establish- ing, either voluntarily or by omission, a clear differentiation among them.

The client, who is likewise identified as the customer, describes the institution or organization where the need for the software project originates, and is who provides the basic understanding of the processes and operations to be automatized; The client organization is also who would most probably be the final user of the project outcome.

The client is meant to provide orientation during the development process and indicates what priorities and traits should be emphasized during the project

(22)

13

The vendor‘s perspective in the second perspective that will be directly evaluated and considered during this thesis. The vendor, known as well as the supplier, in the context of software engineering is an individual or company that develops and implements determined software for the client. The vendor is usually held responsible for the execution of the project in itself; nonetheless, the client has its share in the responsibility of the execution, management and supervision of the implementation performed by the vendor. Some vendors might focus on particular operating system or platforms and others might as well specialize in a particular application area, such as accounting, e- commerce, and resources management, among other.

Both of the previously mentioned parties and corresponding stakeholders have their specific interests and evaluate the project and its related risks from a different point of view, whence considering some particular risk factors more significant to their organization, than its counterpart would for their own organization. Additionally, as proposed by Taylor (2007) in her research on vendor‘s perspectives, compared with in-house development, the software projects involving outsourcing may give rise to additional or different risks from the perspectives of both the client and the vendor.

For the vendor, the project is a way to do business, and therefore tries to maximize the profit of the project, possibly reducing costs on personnel and resources. For the client, the benefits to be gained by the implementation of the project should be worth the price, while the project costs should be minimized (Savolainen, 2011) and the company needs and expectations be met or exceeded. Nevertheless, in order to warrant the project adequate operation to some extent, it is essential that both sides can found con- verging interests between them.

Client and vendor often represent the two main implicated parties in bespoke software development project and hence, acknowledging the basic interest differentiation might lead to a clearer understanding of the actions to be taken on risks assessment, prevention and management, and which side, if either, should be held responsible for such actions. To make a clear distinction on the perspective from which the software project risk factors have been assessed could provide an enriched and deeper compre- hension of the results found on research performed on the subject.

(23)

14

3 Research formulation

This chapter focuses on a deeper description of the reasons why the research on this subject was undertaken, what were the decisive factors to delimit the specific research problem and how it prompted the corresponding research questions.

3.1 Research problem

Numerous articles have been written over the years by recognized experts pioneers in the field of project risk in the IT and Computer Science fields, with the objectives of pursuing, clarifying and expand the knowledge on the topic. In previous decades some authors have proposed main risks checklist (Boehm, 1991), analyzed classical risk management approaches (Lyytinen et al., 1998), others have suggested frameworks for as- sessing risk (Vitale, 1986) and methods for risk management (Heemstra and Kusters, 1996) as well.

In recent years there have been similar efforts by a new generation of researchers, which have proposed, for example, frameworks to improve effectiveness and increase the probability of success in IT projects (Alhawari et al., 2012), decision support systems for the modeling and management of project risks and risk interactions (Fang and Marle, 2012), contingency approaches for general IT projects (Taylor et al., 2012), risk- pricing methods for diverse type of software projects (Appari and Benaroch, 2010a) and integrative economic optimization approaches to manage systems development risks (Benaroch and Goldstein, 2009), among others.

Although different parties and stakeholders have been recognized during the aforementioned, and other related studies, it has not been common to make a clear differentiation between the client‘s and the vendor‘s perspectives on risk. However, different perspectives are partly taken into account in outsourcing literature, which has traditionally studied outsourcing from the client‘s perspective. (Savolainen, 2011). Even though research has been ongoing for some time on the topic of software development ventures and its risk management approaches, in general, it appears that for the most part, in such research efforts the acknowledgement of more than one risk assessment

(24)

15

and management perspective been involved in externally developed bespoke software initiatives has been overlooked.

As an example of this, Savolainen (2011) pointed out during her investigation on the subject of project success/failure definitions that, although some articles seems to consider the vendor‘s perspective on this topic, after further review of the available literature, only one article placed the main focus on such perspective.

The lack of deeper attention to this alternate perspective may be due to several aspects that could range from unawareness of such differentiation, absence of academic research presence in most vendors‘ firms, limited openness by vendors and confidential- ity of their projects information, to mention a few of them.. This consideration was con- ceptualized as the central research problem behind this thesis. Therefore to validate this assumption will occupy much of the present research.

3.2 Research questions

For the most part, this thesis will review the findings of experts in topic of risks in Software development projects. The focus will be mainly set on the investigation of risk factors associated to bespoke software development projects, and more specifically on the exploration of the available literature on the subject and whether it has in any way considered or included the multiple perspectives that correspond to different parties (client and vendor) and stakeholders involved in such projects.

Bearing in mind that an initial search proved to deliver plenty of literature related to risk management in software projects, it becomes necessary to embark on a further review of such literature, thus, the first research question arises as follows:

RQ1: What have been the most relevant empirical studies published in the last three decades, addressing the issues of bespoke software development projects and their inherent risks?

Whereas recent studies might have not yet achieved the relevance some older studies on the subject have reached, yet they provide an accurate insight of the practices in risk management at the time a particular study was carried out. The prospect of recent

(25)

16

articles will be considered essentially in order to deem the evaluation of risk management assessment in time, by comparing the earlier views of the pioneers in the field of software engineering risks and the current state of the art.

Since the development and implementation of bespoke software seems to continue having a substantial market share in the technological industry, it would be important to determine, based on the literature to be reviewed, whether there has been a shift in the main risks identified by prior researches and the current ones. Therefore, the second research question will be oriented as follows:

RQ2: Can the relevant studies on the topic of risks of bespoke software de- velopment projects show any plausible variation on the risk factors that have been important in the last decades, and those currently being consid- ered significant by contemporary researchers?

Another aspect that has been considered to be evaluation-worthy in the relevant literature related to bespoke software projects is associated to the potential differentiation between diverse parties and stakeholder‘s perspectives that well-known researchers might have considered when carrying out their studies. Hence the third research question will be oriented to that purpose:

RQ3: Can the relevant studies in the field of risk management of bespoke software projects provide evidence of any kind of significant differentiation between the vendor‟s perspective and the client‟s perspective?

(26)

17

4 Research Methodology

This chapter aims to describe the research methodology selected for the research and the reasons why such methodology was considered the most appropriate one, given the proposed research problem and questions. As the chapter progresses, the search protocol will be presented, including the selection and review criteria for sources and articles, as well as articles search, inclusion and classification, along with their quality assessment.

4.1 Selection of the research methodology

Since the ultimate goal of this research is to gather and analyze relevant information provided by significant and representative literature on the subject of risk management in bespoke software projects, it was decided that a Systematic Literature Review (SLR) approach would be the most adequate methodology to undertake the task. As Kitchen- ham (2009) proposes, SLR is the main method of synthesis of best quality scientific studies on a specific topic or research question, and as part of the goal of evidence- based software engineering, which is: „„To provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of soft- ware”, SLR serves in the aggregation of research results.

Although the software engineering research has relatively little empirical research compared with the large quantities of research available on other fields, and research methods might not be as rigorous as those used by researchers in other disciplines (Kitchenham, 2004), there are still several reasons for undertaking a SLR on the subject, the most common of which are: to summarize the existing evidence concerning a technology or practice; to identify any gaps in current research in order to suggest areas for further investigation and to provide a framework/background in order to appropriately position new research activities. (Savolainen, 2011).

This SLR was executed thought a carefully designed and planned process, which provided a well-defined procedure to execute the search and selection of the primary studies, as well as their posterior synthesis and analysis.

(27)

18

The primary studies chosen for the literature review were selected essentially based on the relevance of the article in which the study was presented. Although there is no official technique to determine the significance of scientific papers, there are methods that allow certain degree of judgment over an article‘s relevance. For the purposes of the present SLR, the relevance of scientific articles was judged by two decisive aspects, namely number of citations and quality of the study, which were considered as representative parameters to identify relevant publications on the aforementioned fields, both of which will be explained in depth in the subsequent sections. On the other hand, significant criteria was defined to assess the studies quality

4.2 Search Protocol

The research strategy used for the SLR was based on electronic searches of reliable and relevant empirical studies related to software projects‘ risks in the fields of Information Systems and Software Engineering.

In order to determine which articles were to be considered relevant, two decisive aspects were measured. First, the impact factor of the scientific peer reviewed journal where they have been published, given that such trait aims to describe both journal and author impact (Garfield, 2006). The second aspect evaluated was the number of times an article has been cited over time.

The protocol specifies the methods that will be used to undertake the systematic review. A pre-defined protocol is necessary to reduce the possibility of bias; hence, the definition and execution plan for the electronic searches was established as a protocol, which was elaborated in advance, according to the guidelines proposed by Kitchenham (2004) in her report ―Procedures for Performing Systematic Reviews‖. It is worth men- tioning that the protocol slightly evolved during the research process, in order to consider aspects that were initially disregarded.

The protocol was followed throughout the different stages of the research. Moreo- ver, given that a SLR is an iterative process, the original protocol was not a rigid document; instead it evolved as the study progressed. The whole search and selection pro-

(28)

19

cess, as well as the criteria used to include or exclude articles, were properly documented using the instruments and tools indicated in the protocol. Several data extraction forms were produced in order to properly document the research process. Those forms will be described during the protocol.

4.2.1 Selection of Scientific Journals

In order to collect relevant literature that would allow answering the research questions, it was pondered that the most suitable sources to provide the primary studies for this SLR, should be recognized electronic, peer-reviewed scientific journals in the fields of information systems and software engineering.

Different conditions were considered with the aim of assuring the selection of the journals (henceforth denominated ―sources‖) would be adequate, trustworthy and would include a wide range of recognized studies. The sources were selected following con- ventional criteria in order to ensure, as far as possible, that the most recognized journals in the field of software engineering and computer sciences would be included in the search, whilst dropping the possibility of exclusion of any relevant journal.

With the objective of registering the preliminary sources, the data extraction form 1 denominated ―Search results by journal‖ was designed. The considered criteria for the sources selection was:

1. Previous familiarity: To be selected, the journal had to be renowned in the field of Computer Sciences. For this purpose, a list of twenty-two familiar journals was provided by leading researchers in the field of software engineering projects in the department of Computer Sciences at the UEF. The list was used as the starting point for the sources search.

The Journals that were included in the aforementioned list will appear in the data extraction form 1 labeled as ―Previous familiarity‖ in the field called ―Main reason for consideration‖.

2. Five-year impact factor: The five-year impact factor is a value calculated by dividing the number of citations achieved by articles in a journal by the total

(29)

20

number of articles published in the five previous years. This element expresses the relative importance of a particular journal in its specific field. This factor can be estimated for most known journals using the Journal Citation Reports (JCR), which is an electronic tool available in the Institute for Scientific Information (ISI) Web of Knowledge site.

Journals that do not appear in the list of familiar journals, but are found to be listed among the first 100 journals ranked in the categories of information systems and software engineering in the JCR Science edition of 2013, should be included for consideration. These journals would be labeled as ―Impact factor‖ in the main reason for consideration.

3. Articles found: A final search should be conducted in the site of the ISI Web of Science to discard the exclusion of any relevant journals that might appear in the JCR, but have a less significant five-year impact factor. This search will be performed as follows:

 The keywords Software and Risk should be used as the search topic.

 The included categories should be limited to Computer Sciences - Software Engineering and Computer Sciences - Information Systems.

 The type of documents searched should be limited to articles.

The journals found through this search would be added to the data extraction form 1 labeled as ―Articles found‖ in the main reason for consideration.

4.2.2 Review of the Scientific Journals

After the inclusion of all possible sources in data extraction form 1, further requirements for the sources should be evaluated. The suggested requirements for the journals are:

1. Scope of the journal: The scope of the journals should be related to topics that concern programs, routines, and symbolic languages that control the functioning of the hardware and direct its operation, computer graphics, digital signal processing, and programming languages. Similarly topics that relate to acquisition, processing,

(30)

21

storage, management, and dissemination of electronic information that can be read by humans, machines, or both, resources for telecommunications systems and discipline-specific subjects information processing systems were included.

2. Subject Categories: The subject categories would be delimited to include only those journals whose main field was Computer Sciences, more specifically, those related to Information Systems and Software Engineering.

3. Number of articles published: Given that after reviewing the list of familiar jour- nals it was found that all of them had over 100 publications on the previous subjects. For this reason no journals with lower number of publications should be considered.

4.2.3 Criteria for the Articles Search in the Selected Journals

After a final list of sources is properly revised and approved by the supervision of this thesis, the search for the articles should begin. The bibliographic database Scopus would be used as the major tool to perform this search. Scopus claims to be the largest abstract and citation database of peer-reviewed literature containing articles from more than 20,000 peer-reviewed journals (ADAT, 2012), thus is considered to be an adequate op- tion to find possible articles.

Figure 2: Parameters for articles search in Scopus.

(31)

22

The search parameters that would be used to perform the articles exploration in Scopus are the name of the journal (source title); ―Risk‖ in the keyword for the article title, abstract and keywords; article as the type of document and Physical Sciences, So- cial Sciences and Humanities in the subject areas. An example of the search is presented in Figure 2.

The exact same search has to be executed for each one of the sources previously selected. The results obtained from each journal exploration should be documented in the data extraction form 1. The form would have previously been filled with general information from the selected journals during the sources search.

The remaining information required in data extraction form 1 will be recorded after the exploration for studies had been performed. Once a source had been explored, the following information needs to be collected according to the results obtained:

 Date of the oldest risk related article with citations.

 Total number of articles published by the journal until 2013.

 Number of risk related articles with at least ten citations.

4.2.4 Studies classification

This SLR will concentrate in the recollection and evaluation of data obtained from twenty relevant articles resulting from empirical studies, with a main focus on risk factors that threaten software development projects.

Table 1: Distribution of articles by category.

ID Name Description Number of

Articles Group 1 Most cited articles Articles from recognized electronic, peer-

reviewed scientific journals in the fields of information systems and software engineering.

10

Group 2 Most recent articles Articles published in the last 5 years from on the

same journals. 5

Group 3 Relevant papers from Google Scholar

Relevant papers found in a Google Scholar

search. 5

The sources exploration stage is expected to produce a broad spectrum of possible candidate articles for the review, however in order to assess the articles quality and nar-

(32)

23

row down the analysis, all articles retrieved using the search strategy proposed in the protocol, ought to be individually revised to verify that they meet the minimum requirements to be considered as preliminary studies for this review.

In order to maximize the evaluation of a high number of relevant articles in the field, and to be able to obtain a wide range of diverse sources that might contribute to answering the proposed research questions, it was decided during the development of the protocol, and in agreement with the supervision of this thesis, that there would be twenty articles take part in the review, which tentatively would be distributed in three groups, as proposed in Table 1.

4.2.5 Preliminary Selection from Articles in Groups 1 and 2

An overall review has to be performed on every candidate article found in each one of the selected sources. The classification of the articles retrieved from each journal will be done in two stages: First, the most cited articles and second, the newest articles.

1. Most Cited Articles: All the articles obtained through the searches ought to be sorted by their number of citations on a descending order. In this way, the articles with the highest number of citations would be the first ones to be evaluated. How- ever, all articles related to the topic and with at least twenty citations should receive a further revision. For each journal, the total number of articles found in this category should be registered in the data extraction form 1 in the column ―Risk articles with citations‖.

2. Newest Articles: For this category, the same group of articles will be sorted, never- theless this time in a descending order, by their date of publication. As a result, the newest articles will be evaluated first and only articles with less than five years (since 2008) of publication should be taken into consideration for a further examination.

4.2.6 Preliminary Selection of Articles in Group 3

Google Scholar was incorporated as a tool for this exploration given that it is a relatively new research instrument that works under the premise of ―aiming to rank documents the way researchers do, weighing the full text of each document, where it was pub-

(33)

24

lished, who it was written by, as well as how often and how recently it has been cited in other scholarly literature‖ (GS, 2015).

Given that Google Scholar does not separate its search by subject areas, it would be necessary to broaden the keywords, in order to narrow down the list of results to be retrieved.

The keywords for this group of articles should delimit the search to the main areas of interest of this thesis. Hence, the keywords would be Software, Project and Risk. The articles whose titles indicate at first sight that they are related to the topic of interest for this review should be selected for further examination. In this specific exploration there will be no limitations regarding the scientific journal, date of publication, number of citations and type of document to be searched.

Even though Google Scholar is likely to produce several hundreds of results under the proposed search schema, the possible list of candidates should initially include twenty articles for further revision. If after additional revision not enough articles fulfill the criteria to be selected, a second round search, following the same initial criteria would be executed, and so forth until the number of expected articles is reached.

4.2.7 Inclusion Criteria for Studies

In the three categories of articles, the title and the abstract would be the initial element to make a preliminary evaluation concerning the usefulness of the article for this review.

The articles should clearly state its relation to the topic of risks in bespoke software development projects. If the title is not totally clear about the subject of the article, the abstract might give a brief, yet deeper insight of the purpose and type of study that is being presented. It can also provide additional information such as the sample population and a glance to the results obtained in the study.

With the information obtained after the preliminary evaluation of the title and abstract of an article, it should be decided whether or not that particular article could be considered as a candidate for this research. If the study appears to have an adequate relation to the topic of risks in bespoke software development projects, it would join a

(34)

25

preliminary list of candidates and its general information ought to be entered in the data extraction form 2, denominated Preliminary list of candidate studies, which includes the following data:

 Title of the article

 Authors

 Journal of publication

 Date of publication

 Number of citations

Bearing in mind that the number of preliminary studies is not relevant, there was no need to set a strict limitation on the final size of the preliminary list in the first two groups. The preliminary list of candidate studies will be the starting point to select the twenty articles that will integrate the SLR process. At this point, the articles in the preliminary list of candidate studies should have been previously reviewed and are sup- posed to meet the minimal requisites to be included in the review; hence they were taken for further consideration.

Table 2: Inclusion and Exclusion criteria for studies.

Factor

Criteria for inclusion Criteria for exclusion Type of

study

Primary studies based on cases of bespoke software development projects or software development project managerial experiences.

Secondary or tertiary studies of any kind.

No empirical experiment was reported or performed.

Studies based on ERP implementations.

Software projects type

Development and implementation of bespoke software and/or bespoke software products for companies or organizations.

Systems security, maintenance or other associated IT ventures that do not explicitly involve bespoke software development.

Time span Studies should be related to risk factors inherent to finite schedule projects.

Studies based on continuous services have to be excluded.

Uniqueness The article should be based on a unique primary study.

Articles based on the same study than previously selected articles.

Language English Other languages beside English.

All the candidates in the preliminary list would have to undergo a secondary evaluation based on the set of inclusion criteria defined in this section. Only those articles

(35)

26

confirmed to satisfy all the inclusion criteria will be designated as final studies for this SLR. The inclusion criteria have been defined as presented in Table 2.

The articles based on project of Enterprise Resource Planning (ERP) systems implementation were not included in this review since they are considered as large scale projects which, beside software implementation, usually require a great deal of effort at all levels in the organization and implies changes in processes. ERP type of implementations touch almost every aspect of a company, therefore it involves a very specific set of risks that might not reflect the general risk factors and settings of most bespoke software development project.

Using these criteria, the evaluation of the candidates in the preliminary list of studies will begin with the articles with the highest number of citations (Group 1), followed by the newest articles (Group 2) and finally with the articles retrieved from Google Scholar (Group 3). The articles meeting all inclusion criteria will receive a final quality assessment.

4.2.8 Quality assessment of the final articles

In each one of the three articles‘ groups, the quality assessment will start from the articles with the highest values, either citations or date of publication, and will later continue in descending order until the expected number of articles in each group has been reached or until all articles in one category have been assessed.

The following factors were used to evaluate the quality and usability of the studies to be included in the final review:

1. Type of studies: The article must have been based on findings attained by the exe- cution of empirical studies. The settings, methodology and population used for such studies had to be clearly defined and explained in the article (Keele, 2007).

Articles based on claims, expert opinions or similar sources, not supported by corresponding empirical data, should not be included in this research.

(36)

27

2. Risk factors list: A list of risk factors on bespoke software projects or a categoriza- tion of bespoke software development risks is one of the expected outcomes of each article. If the primary study addresses development risks, but no list of risk was produced or introduced in the article, the study will not be considered suitable for the purposes of this review.

3. Uniqueness: Each article selected for the final review should be based on a unique study. If more than one article refers to the same study or to different stages of a longitudinal study, the findings presented in those articles should be compared and analyzed. In case the articles present no differences between their lists of risk factors, the article with the highest number of citations would be the one selected for review. In case one of the articles presents a new set of risk factors in its list of risks, in comparison to other articles based on the same study, the most cited article should be included; however the article introducing the new set of risk factors should also be included as a complement of the previous article.

4. Research methodology: In addition to a clear research methodology, each article should explain the process followed for the data collection, summarization and analysis techniques used to perform the study.

5. Conclusions: The article should report the results obtained from the study and pro- vide the author(s) conclusions.

In order to avoid research bias, after all the articles to be included in each one of the three groups had been evaluated, and the twenty articles originally expected had been selected and assessed, the final list of candidates should be revised and approved by the supervision of this thesis.

(37)

28

5 Results of the search process

The protocol was applied in the search and evaluation of the sources and articles. This chapter presents the iterative search processes that were executed. During the progres- sion of the searches the protocol was modified to some extent, correcting some of the initial suppositions and criteria that were not entirely accurate. This helped to depurate the original proposed protocol.

The process of elaboration of the protocol and the searches executions was carried out in the period between April and June of 2013 and the final search results are presented in this chapter.

5.1 Results from the sources’ search

Following the search protocol guidelines, well-known scientific journals were explored in order to find the most appropriate sources for the articles‘ search, based on characteristics as previous familiarity for researchers on the field and the journal‘s relevant impact factor. Each journal was searched for both most cited articles and newest articles in the topics concerning this review.

As a starting point, a list of twenty-two recognized scientific electronic journals was provided by specialized researchers in the department of Computer Sciences of the UEF. Each one of those journals was reviewed, in order to assess their relative impact, based on their 5-year impact factor.

With the aim of appointing a complete list of scientific journals, relevant in the fields of Computer Sciences and Software Engineering, an extensive revision of the JCR 2013 was performed, going through the first 100 journal with the highest 5-year impact factor. As a result of that revision, 7 additional articles were selected.

After a deeper revision of the scientific journals in the list of twenty-two familiar journals, they all were included in the potential sources, given that all of them had 5-

An evaluation on client vs. vendor perspectives for risk assessment in software projects : a systematic literature review on the main perspectives of bespoke software development projects' risk assessment