The research strategy used for the SLR was based on electronic searches of reliable and relevant empirical studies related to software projects‘ risks in the fields of Information Systems and Software Engineering.
In order to determine which articles were to be considered relevant, two decisive aspects were measured. First, the impact factor of the scientific peer reviewed journal where they have been published, given that such trait aims to describe both journal and author impact (Garfield, 2006). The second aspect evaluated was the number of times an article has been cited over time.
The protocol specifies the methods that will be used to undertake the systematic review. A pre-defined protocol is necessary to reduce the possibility of bias; hence, the definition and execution plan for the electronic searches was established as a protocol, which was elaborated in advance, according to the guidelines proposed by Kitchenham (2004) in her report ―Procedures for Performing Systematic Reviews‖. It is worth men-tioning that the protocol slightly evolved during the research process, in order to con-sider aspects that were initially disregarded.
The protocol was followed throughout the different stages of the research. Moreo-ver, given that a SLR is an iterative process, the original protocol was not a rigid docu-ment; instead it evolved as the study progressed. The whole search and selection
pro-19
cess, as well as the criteria used to include or exclude articles, were properly document-ed using the instruments and tools indicatdocument-ed in the protocol. Several data extraction forms were produced in order to properly document the research process. Those forms will be described during the protocol.
4.2.1 Selection of Scientific Journals
In order to collect relevant literature that would allow answering the research questions, it was pondered that the most suitable sources to provide the primary studies for this SLR, should be recognized electronic, peer-reviewed scientific journals in the fields of information systems and software engineering.
Different conditions were considered with the aim of assuring the selection of the journals (henceforth denominated ―sources‖) would be adequate, trustworthy and would include a wide range of recognized studies. The sources were selected following con-ventional criteria in order to ensure, as far as possible, that the most recognized journals in the field of software engineering and computer sciences would be included in the search, whilst dropping the possibility of exclusion of any relevant journal.
With the objective of registering the preliminary sources, the data extraction form 1 denominated ―Search results by journal‖ was designed. The considered criteria for the sources selection was:
1. Previous familiarity: To be selected, the journal had to be renowned in the field of Computer Sciences. For this purpose, a list of twenty-two familiar journals was provided by leading researchers in the field of software engineering projects in the department of Computer Sciences at the UEF. The list was used as the starting point for the sources search.
The Journals that were included in the aforementioned list will appear in the data extraction form 1 labeled as ―Previous familiarity‖ in the field called ―Main rea-son for consideration‖.
2. Five-year impact factor: The five-year impact factor is a value calculated by dividing the number of citations achieved by articles in a journal by the total
20
number of articles published in the five previous years. This element expresses the relative importance of a particular journal in its specific field. This factor can be estimated for most known journals using the Journal Citation Reports (JCR), which is an electronic tool available in the Institute for Scientific Information (ISI) Web of Knowledge site.
Journals that do not appear in the list of familiar journals, but are found to be listed among the first 100 journals ranked in the categories of information sys-tems and software engineering in the JCR Science edition of 2013, should be in-cluded for consideration. These journals would be labeled as ―Impact factor‖ in the main reason for consideration.
3. Articles found: A final search should be conducted in the site of the ISI Web of Science to discard the exclusion of any relevant journals that might appear in the JCR, but have a less significant five-year impact factor. This search will be per-formed as follows:
The keywords Software and Risk should be used as the search topic.
The included categories should be limited to Computer Sciences - Software Engineering and Computer Sciences - Information Systems.
The type of documents searched should be limited to articles.
The journals found through this search would be added to the data extraction form 1 labeled as ―Articles found‖ in the main reason for consideration.
4.2.2 Review of the Scientific Journals
After the inclusion of all possible sources in data extraction form 1, further require-ments for the sources should be evaluated. The suggested requirerequire-ments for the journals are:
1. Scope of the journal: The scope of the journals should be related to topics that concern programs, routines, and symbolic languages that control the functioning of the hardware and direct its operation, computer graphics, digital signal processing, and programming languages. Similarly topics that relate to acquisition, processing,
21
storage, management, and dissemination of electronic information that can be read by humans, machines, or both, resources for telecommunications systems and dis-cipline-specific subjects information processing systems were included.
2. Subject Categories: The subject categories would be delimited to include only those journals whose main field was Computer Sciences, more specifically, those related to Information Systems and Software Engineering.
3. Number of articles published: Given that after reviewing the list of familiar jour-nals it was found that all of them had over 100 publications on the previous sub-jects. For this reason no journals with lower number of publications should be con-sidered.
4.2.3 Criteria for the Articles Search in the Selected Journals
After a final list of sources is properly revised and approved by the supervision of this thesis, the search for the articles should begin. The bibliographic database Scopus would be used as the major tool to perform this search. Scopus claims to be the largest abstract and citation database of peer-reviewed literature containing articles from more than 20,000 peer-reviewed journals (ADAT, 2012), thus is considered to be an adequate op-tion to find possible articles.
Figure 2: Parameters for articles search in Scopus.
22
The search parameters that would be used to perform the articles exploration in Scopus are the name of the journal (source title); ―Risk‖ in the keyword for the article title, abstract and keywords; article as the type of document and Physical Sciences, So-cial Sciences and Humanities in the subject areas. An example of the search is presented in Figure 2.
The exact same search has to be executed for each one of the sources previously selected. The results obtained from each journal exploration should be documented in the data extraction form 1. The form would have previously been filled with general information from the selected journals during the sources search.
The remaining information required in data extraction form 1 will be recorded af-ter the exploration for studies had been performed. Once a source had been explored, the following information needs to be collected according to the results obtained:
Date of the oldest risk related article with citations.
Total number of articles published by the journal until 2013.
Number of risk related articles with at least ten citations.
4.2.4 Studies classification
This SLR will concentrate in the recollection and evaluation of data obtained from twenty relevant articles resulting from empirical studies, with a main focus on risk fac-tors that threaten software development projects.
Table 1: Distribution of articles by category.
ID Name Description Number of
Articles Group 1 Most cited articles Articles from recognized electronic,
peer-reviewed scientific journals in the fields of in-formation systems and software engineering.
10
Group 2 Most recent articles Articles published in the last 5 years from on the
same journals. 5
Group 3 Relevant papers from Google Scholar
Relevant papers found in a Google Scholar
search. 5
The sources exploration stage is expected to produce a broad spectrum of possible candidate articles for the review, however in order to assess the articles quality and
nar-23
row down the analysis, all articles retrieved using the search strategy proposed in the protocol, ought to be individually revised to verify that they meet the minimum re-quirements to be considered as preliminary studies for this review.
In order to maximize the evaluation of a high number of relevant articles in the field, and to be able to obtain a wide range of diverse sources that might contribute to answering the proposed research questions, it was decided during the development of the protocol, and in agreement with the supervision of this thesis, that there would be twenty articles take part in the review, which tentatively would be distributed in three groups, as proposed in Table 1.
4.2.5 Preliminary Selection from Articles in Groups 1 and 2
An overall review has to be performed on every candidate article found in each one of the selected sources. The classification of the articles retrieved from each journal will be done in two stages: First, the most cited articles and second, the newest articles.
1. Most Cited Articles: All the articles obtained through the searches ought to be sorted by their number of citations on a descending order. In this way, the articles with the highest number of citations would be the first ones to be evaluated. How-ever, all articles related to the topic and with at least twenty citations should receive a further revision. For each journal, the total number of articles found in this cate-gory should be registered in the data extraction form 1 in the column ―Risk articles with citations‖.
2. Newest Articles: For this category, the same group of articles will be sorted, never-theless this time in a descending order, by their date of publication. As a result, the newest articles will be evaluated first and only articles with less than five years (since 2008) of publication should be taken into consideration for a further exami-nation.
4.2.6 Preliminary Selection of Articles in Group 3
Google Scholar was incorporated as a tool for this exploration given that it is a relative-ly new research instrument that works under the premise of ―aiming to rank documents the way researchers do, weighing the full text of each document, where it was
pub-24
lished, who it was written by, as well as how often and how recently it has been cited in other scholarly literature‖ (GS, 2015).
Given that Google Scholar does not separate its search by subject areas, it would be necessary to broaden the keywords, in order to narrow down the list of results to be retrieved.
The keywords for this group of articles should delimit the search to the main areas of interest of this thesis. Hence, the keywords would be Software, Project and Risk. The articles whose titles indicate at first sight that they are related to the topic of interest for this review should be selected for further examination. In this specific exploration there will be no limitations regarding the scientific journal, date of publication, number of citations and type of document to be searched.
Even though Google Scholar is likely to produce several hundreds of results under the proposed search schema, the possible list of candidates should initially include twenty articles for further revision. If after additional revision not enough articles fulfill the criteria to be selected, a second round search, following the same initial criteria would be executed, and so forth until the number of expected articles is reached.
4.2.7 Inclusion Criteria for Studies
In the three categories of articles, the title and the abstract would be the initial element to make a preliminary evaluation concerning the usefulness of the article for this review.
The articles should clearly state its relation to the topic of risks in bespoke software de-velopment projects. If the title is not totally clear about the subject of the article, the abstract might give a brief, yet deeper insight of the purpose and type of study that is being presented. It can also provide additional information such as the sample popula-tion and a glance to the results obtained in the study.
With the information obtained after the preliminary evaluation of the title and ab-stract of an article, it should be decided whether or not that particular article could be considered as a candidate for this research. If the study appears to have an adequate re-lation to the topic of risks in bespoke software development projects, it would join a
25
preliminary list of candidates and its general information ought to be entered in the data extraction form 2, denominated Preliminary list of candidate studies, which includes the following data:
Title of the article
Authors
Journal of publication
Date of publication
Number of citations
Bearing in mind that the number of preliminary studies is not relevant, there was no need to set a strict limitation on the final size of the preliminary list in the first two groups. The preliminary list of candidate studies will be the starting point to select the twenty articles that will integrate the SLR process. At this point, the articles in the pre-liminary list of candidate studies should have been previously reviewed and are sup-posed to meet the minimal requisites to be included in the review; hence they were tak-en for further consideration.
Table 2: Inclusion and Exclusion criteria for studies.
Factor
Criteria for inclusion Criteria for exclusion Type of
study
Primary studies based on cases of be-spoke software development projects or software development project managerial experiences.
Secondary or tertiary studies of any kind.
No empirical experiment was reported or performed.
Studies based on ERP implementations.
Software projects type
Development and implementation of bespoke software and/or bespoke software products for companies or organizations.
Systems security, maintenance or other associated IT ventures that do not explicit-ly involve bespoke software development.
Time span Studies should be related to risk factors inherent to finite schedule projects.
Studies based on continuous services have to be excluded.
Uniqueness The article should be based on a unique primary study.
Articles based on the same study than previously selected articles.
Language English Other languages beside English.
All the candidates in the preliminary list would have to undergo a secondary eval-uation based on the set of inclusion criteria defined in this section. Only those articles
26
confirmed to satisfy all the inclusion criteria will be designated as final studies for this SLR. The inclusion criteria have been defined as presented in Table 2.
The articles based on project of Enterprise Resource Planning (ERP) systems im-plementation were not included in this review since they are considered as large scale projects which, beside software implementation, usually require a great deal of effort at all levels in the organization and implies changes in processes. ERP type of implemen-tations touch almost every aspect of a company, therefore it involves a very specific set of risks that might not reflect the general risk factors and settings of most bespoke soft-ware development project.
Using these criteria, the evaluation of the candidates in the preliminary list of studies will begin with the articles with the highest number of citations (Group 1), fol-lowed by the newest articles (Group 2) and finally with the articles retrieved from Google Scholar (Group 3). The articles meeting all inclusion criteria will receive a final quality assessment.
4.2.8 Quality assessment of the final articles
In each one of the three articles‘ groups, the quality assessment will start from the arti-cles with the highest values, either citations or date of publication, and will later contin-ue in descending order until the expected number of articles in each group has been reached or until all articles in one category have been assessed.
The following factors were used to evaluate the quality and usability of the studies to be included in the final review:
1. Type of studies: The article must have been based on findings attained by the exe-cution of empirical studies. The settings, methodology and population used for such studies had to be clearly defined and explained in the article (Keele, 2007).
Articles based on claims, expert opinions or similar sources, not supported by cor-responding empirical data, should not be included in this research.
27
2. Risk factors list: A list of risk factors on bespoke software projects or a categoriza-tion of bespoke software development risks is one of the expected outcomes of each article. If the primary study addresses development risks, but no list of risk was produced or introduced in the article, the study will not be considered suitable for the purposes of this review.
3. Uniqueness: Each article selected for the final review should be based on a unique study. If more than one article refers to the same study or to different stages of a longitudinal study, the findings presented in those articles should be compared and analyzed. In case the articles present no differences between their lists of risk fac-tors, the article with the highest number of citations would be the one selected for review. In case one of the articles presents a new set of risk factors in its list of risks, in comparison to other articles based on the same study, the most cited article should be included; however the article introducing the new set of risk factors should also be included as a complement of the previous article.
4. Research methodology: In addition to a clear research methodology, each article should explain the process followed for the data collection, summarization and analysis techniques used to perform the study.
5. Conclusions: The article should report the results obtained from the study and pro-vide the author(s) conclusions.
In order to avoid research bias, after all the articles to be included in each one of the three groups had been evaluated, and the twenty articles originally expected had been selected and assessed, the final list of candidates should be revised and approved by the supervision of this thesis.
28
5 Results of the search process
The protocol was applied in the search and evaluation of the sources and articles. This chapter presents the iterative search processes that were executed. During the progres-sion of the searches the protocol was modified to some extent, correcting some of the initial suppositions and criteria that were not entirely accurate. This helped to depurate the original proposed protocol.
The process of elaboration of the protocol and the searches executions was carried out in the period between April and June of 2013 and the final search results are
The process of elaboration of the protocol and the searches executions was carried out in the period between April and June of 2013 and the final search results are