4 CONCLUSIONS
4.3 R ELIABILITY AND VALIDITY
One important question is the quality of the research findings. The traditional way to assess this is through validity and reliability. Validity is concerned with whether the findings are really about what they appear to be about (David and Sutton 2004). This study divides validity into three parts, as suggested by, for example, Yin (1994) and Riege (2003). These are construct validity, internal validity and external validity. According to Trochim and Donnelly (2007) these may be defined as follows:
Construct validity refers to the degree to which inferences can legitimately be made from the operationalizations in your study to the theoretical constructs on which those operationalizations were based.
External validity is the degree to which the conclusions in your study would hold for other persons in other places and at other times.
Internal validity is the approximate truth about inferences regarding cause-effect or causal relationships
Reliability is also important for assessing the quality of the research process and findings.
Reliability is the degree to which the indicator or test is a consistent measure over time (Trochim & Donnelly 2007). This means that reliability and replication, as discussed above, are closely linked to each other. As the reliability and the three forms of validity introduced above are more positivist in nature, it could be argued that using these terms in a more constructivist study - as this dissertation - setting is not appropriate.
To answer this problem some researchers have replaced reliability and validity with the terms confirmability, credibility, transferability, and dependability, but as Riege (2003) notes, these correspond to construct validity, internal validity, external validity and reliability (in this order) so closely that the terms may be used interchangeably. As the positivist terms are more commonly used, they are also utilised here.
Riege (2003) introduces techniques that may be used to improve the construct validity, internal validity, external validity and reliability (and their “constructivist” counterparts) in qualitative case study research. His insights are listed in Table 9. The table is followed by comments about how his recommendations are dealt with in this study.
Table 9: Techniques for ensuring validity and reliability in qualitative case research according to Riege (2003) and in this research process.
Riege This dissertation
Construct validity
• Multiple sources of evidence
• Chain of evidence
• Have key informants review draft case report
• Confirmability audit (Examine the data, findings, interpretations and recommendations)
• 11 case sites
• The chain of evidence is introduced in this summary and the articles included.
• Industry partners in steering committee of the research project and other
researchers participated into the research process.
Internal validity
• Do within-case analysis, then cross-case pattern matching
• Do explanation building
• Assure internal coherence of findings and that concepts are systematically related
• Triangulation (sources, investigators, and methods)
• Steering group of the research project and colleagues commented on the issues related to internal validity during the process.
• Peer discussions
• Attention paid to internal coherence
• Different types of data gathering methodologies were applied.
• Several cases, methods and multi-disciplinarity are sources of triangulation.
• The theoretical model was used to monitor the empirical setting.
External validity
• Use replication logic in multiple case studies
• Define scope and boundaries of reasonable analytical generalization
• Compare evidence with extant literature
• Predetermined questions
• Thick description (develop case study data base)
• Cross-case analysis
• Specific procedures for coding and analysis
• Limits of generalization are dealt with in the introductory chapter.
• Findings are related to existing theoretical knowledge.
• The evidence was related to the theory in many phases.
• The research questions were defined at the beginning of the process.
• Content analysis procedures are used in analysis and coding.
• Predetermined themes in thematic interviews
• Replication logic was applied in case setting.
• Content analysis methods used for coding and analysis
• Cross-case analysis was limited.
Reliability
• Give full account of theories and ideas
• Assure congruence between research issues and feature of system design
• Develop and refine case study protocol
• Use multiple researchers
• Record observation and actions as concrete as possible
• Use case study protocol
• Record data, mechanically develop case study database
• Assure meaningful parallelism of findings across multiple data sources
• Use peer review
• Dependability audit (examine and document the process of inquiry)
• Clarify researcher’s theoretical position and biases
• The theories and ideas have been described in the first two chapters of this dissertation.
• Multi-disciplinarity
• Case study protocol was carried out in constant manner in all cases.
• Observation matrix and use of photos in recording the data
• The research process and the articles were carried out in co-operation with several researchers.
• The data was recorded and stored for further use.
• The articles went through a peer review process.
• Researcher’s biases and position are acknowledged in CV attached to this dissertation.
• The way conclusions are made may be understood by all.
As the study was carried out as part of big research projects, there was an active and ongoing confirmability audit, as both the colleagues in the research process and the industry partners in the steering committee of the project were auditing and directing the research process. This made it possible for the key informants to verify the findings.
Additionally, construct validity was obtained through the debriefing process. Further, having more than one case in each part of the study had an influence on construct validity.
This also made the chain of evidence visible in each case, as the central patterns became visible.
The debriefings, workshops, steering group and peer researchers have been actively affecting and commenting on the internal validity issues during the research process.
Further, a lot of attention has been paid to assuring the internal coherence of the findings and concepts so that they are systematically related.
One often mentioned way to increase validity is through several types of triangulation.
Researcher triangulation means that there is more than one researcher in the different phases of a study. In this study process, at least two researchers were working together in each phase of the process. Data triangulation means that several sets of data are used to analyse the problem at hand. (Denzin 1975.) This study utilised three sets of data: First, the interview material, second, the workshop material and third, the material from the walk-through audits. Methodological triangulation means that the same set of data is studied with more than one research method and theoretical triangulation involves studying the phenomena in question from several theoretical perspectives (Denzin 1975). These two forms of triangulation were not employed in this study.
As already discussed above, the replication logic has been an important feature of the study. This adds to its external validity. Other measures to improve external validity have included the fact that the themes in interviews and audits were predetermined based on theory, even though the interviews and audits were “only” semi-structured. The scope and boundaries of the generalizability were already established above. This helps the reader to assess the external validity of this research. The basic logic of this study – namely, comparing existing theory and literature to the cases – also adds to the external validity of this study.
Chapter 2 and article 1 present the theoretical background of this study and this side of the reliability issue. Other features that have added to reliability are that the process involved multiple researchers and that the data collection phase was recorded by taping the interviews, workshops and audits and by taking photos of the audits. This material has been added to the case study database of the Facility Services Research Group at the Helsinki University of Technology. The findings have been reported as articles that have gone through a peer review process.