Validity and reliability of the research

Silverman (1993, 156) suggests two forms of validation as particularly appropriate to the logic of qualitative research; triangulation and respondent validation. In triangulation different kinds of data and different methods are used to see whether they collaborate on another. The method that takes the researcher’s findings back to the subjects being studied is known as respondent validation.

According Yin (1994, 33) four tests are relevant to confirm the quality of the research design (Table 8.1). The tests are:

Construct validity establishing the correct operational measures for the concepts being studied.

Internal validity establishing a causal relationship, whereby certain conditions are shown to lead to other conditions, as distinguished from spurious relationships.

External validity establishing the domain to which a study’s findings can be generalised.

Reliability demonstrating that the operations of a study – such as the data collection procedures - can be repeated, with the same results.

Table 8.1 Case study tactics for four tests (Yin 1994, 33)

Test Case study tactic

Use multiple sources of evidence Establish chain of evidence Construct validity

Have key informants review draft case study report

Do pattern-matching Do explanation-building Internal validity

Do time-series analysis

External validity Use replication – logic in multiple-case studies

Use case study protocol Reliability

Develop case study database

The implementation of these tests in this study has been introduced in the following chapters.

8.3.1 Construct validity

Construct validity refers to establishing the correct operational measures for the concepts being studied (Yin 1994, 33). In the present study the construct validity was ensured primarily in three ways: using multiple sources of evidence, establishing a chain of evidence and obtaining the key informants’ comments and feedback on the report drafts.

Data source triangulation means checking out the consistency of different data sources, i.e. looking at the same case from different sources of data (Stake 1995, 112 - 113). In this study both qualitative and quantitative data collected from primary and secondary sources was used. Different methods of collecting data were used as another source of triangulation; informants were interviewed, questionnaires (structured interviews) were applied, and documents were studied.

Most of the data was collected by interviews from the case companies. The number of informants was three or more per one case company. The secondary data was

collected from publicly available sources, e.g. annual reports and annual accounts from the National Board of Patents and Registration. The reason for applying the rich triangulation in this research was to become more confident with the results.

The aim was to find both confirming and disconfirming evidence and to challenge the initial interpretations. Triangulation served well in doing this.

The key informants’ feedback was used. The report drafts were sent to the key informants and they were asked to comment on the descriptions and analysis. After that the researcher had a meeting, a telephone discussion or a discussion by email with the key informants. Details of the key informants’ feedback are given in Appendix 2.

Establishing the chain of evidence is related to the reader’s ability to follow the different phases of the research from beginning to final conclusions (Yin 1994, 102). This report is structured by the issues and in a chronological manner to make it easy to trace all the steps from the research questions, leading to the evidence and conclusions.

8.3.2 Internal validity

Internal validity establishes a causal relationship proposed by the research. In the present study pattern-matching and explanation building were used to establish internal validity. Each case description should provide a detailed picture of the business development effort implemented in the case company to justify the interpretations made.

8.3.3 External validity

External validity is about determining whether the findings are generalisable beyond the specific study. Generalising the results of case studies has been an issue of ongoing debate. The logic of the generalisation from case studies is replication, and the generalisations are analytic generalisations by nature, not statistical (Yin 1994, 31).

The external validity was established by using the replication approach introduced by Yin (1994, 49). To provide theoretical replication, the cases were selected to produce contrasting results. Two of the cases represent successful ones and two cases represent unsuccessful ones. Important for the replication procedures is the development of a theoretical framework, which should state the conditions under which a particular phenomenon is likely to be found (Yin 1994, 46). In this study the theoretical framework was produced after the literature review and refined after the pilot case study.

Yin (1994, 31) notes that if two or more cases are shown to support the same theory, replication may be claimed. The empirical results may be considered more potent if two or more cases support the same theory but do not support an equally plausible rival theory.

A profound understanding of the problem domain and the cases in the study allow to suggest that it would be useful for SMEs in the selection and planning phases of a business development project to take into consideration the dimensions and factors introduced in the framework for the business development project success.

The enterprises can use the structured questioning as a checklist and evaluate how well their own firm satisfies these issues. In this sense the results are transferable.

This study was conducted in Finland and in Finnish SMEs. The applicability of the results in other countries may be limited because the differences in cultural aspects and values may affect management and leadership style, communication, commitment, motives, etc. Further, the applicability of the results is limited to business development projects that are internal by nature.

8.3.4 Reliability

Reliability demonstrates that the operations of a study can be repeated with the same results (Yin 1994, 33). Gummesson (1993, 15) provides a similar definition.

Reliability requires that the research, when repeated by the researcher or someone else, should lead to the same or similar results. Silverman (1993, 165) argues that reliability can be improved by comparing the analyses of the same data by several researchers. Replication refers to the possibility of obtaining similar results and conclusions when a study is repeated in the same manner (Yin 1994).

In this study the case study protocol and study database were used as relevant tactics. The study database was created by collecting all the relevant material concerning each case in the database. The interpretations were included in the database. High reliability was pursued by tape-recording all the interviews. Also the careful reporting in the study contributes to the reliability of the study. An auditor should be able to repeat the procedures, arriving at the same findings.

Information was collected from multiple data sources, and several informants were interviewed per case company. This made it possible to reduce biases. No inconsistencies between primary and secondary data were found. Maintaining the chain of evidence contributes to the reliability of the information (Yin 1994, 90 – 100).

8.3.5 Limitations and problems of the research

A subjective approach was adopted in this study. The information was largely based on subjective evaluations and interpretations of the reality. This should be taken into consideration when interpreting the results. However, it is possible that using the informants’ subjective evaluations supported by the objective measures leads to a better understanding of the business development success in SMEs (cf. Dess &

Robinson 1984; Ketokivi & Schoeder 2004). The informants were aware of the use of objective data from secondary sources. This might have been an issue to prevent conscious incorrect assessments, e.g. overestimation of the results during the interviews.

The interviews were based on the informants’ open narrative story, a semistructured interview and structured questioning. By using open-ended questions, it was possible to gain information that might otherwise have been missed. However, it should be noted that the list of questions used during the interviews might have guided the informants in their responses at least to some extent.

This study was mostly based on cross-sectional data on projects implemented some years ago. The assessment and measurement of the success and performance changes were conducted afterwards the project implementation. Because the

project success may vary during the project life cycle, the results of this study may not give the right picture for every phase of the project implementation. It is also possible that the current situation and recent incidents have affected the informants’ answers. However, the influence of one respondent was not dominating, because the number of informants was at least three per case company, and because the demand for unanimity was used as a criterion during the case analysis.

The business development projects were implemented as integrated to the daily business and the daily responsibilities of the employees and owner-managers. It might have been difficult for the informants to fully and objectively separate, which of the performance changes were caused by the business development project and which of them were results of changes in some other issues, e.g. changes in the business environment or competition. Attempts to avoid this bias were made by utilising the multi-informant system, and by collecting information from different sources, including industry key figures and information about competitors. The information from the different sources was triangulated and no inconsistencies were found. However, this possible problem should be noted when interpreting the results.

This study focused on searching the general patterns and features of the business development projects implemented in SMEs. This study did not concentrate on special features and special content of different types of projects. This means that the main focus was on the issues connected to context, process and control. The technical content were in a minor role. The results of the present research should be viewed in this light.