Evaluation of the study

When evaluating a study as a whole it is time to look back and estimate what the objectives of the study were, where we are now and what kind of a journey has taken us here. It also includes an estimation of how the interpretations and conclusions appear to the readers. Miles and Huberman (1994) list several viewpoints on how the goodness of the conclusions of a study can be estimated.

The goodness of conclusions mean that they are reliable, valid, possibly or probably true, dependable, reasonable, confirmable, credible, and useful. These factors are discussed below to evaluate the reliability and the validity of the ICT-ESP study.

76

Reliability and validity of the study

Reliability concerns the extent to which the findings would emerge if the study was repeated in the same context (Yin 1994). Once we treat social reality as always changing, it makes no sense to worry about whether the research instruments measure accurately (Silverman 1993). The reliability means to what extent the researcher can trust the findings of the study.

Validity is a concept designating an ideal state – to be pursued, but not to be attained. Validity is like integrity, character, or quality, to be assessed relative to the purposes and their circumstances (Brinberg and McGrath 1985). McNiff and Whitehead (2010) separate personal and social validation. Personal validation or self-validation is about considering how the chosen

methodologies and the knowledge gained via them represent the values the researcher has. In case the evidence gained is in balance with the evidence, then a personal validation is achieved. Social validation is when others test what you are saying in the light of the evidence you are offering.

Validity has been divided into: a) descriptive validity (amount and accuracy of data), b) interpretative validity (whether the events identified in the data are interpreted correctively in relation to the aims of the interviewees), c) and theoretical validity (whether the accounts functions as correct explanations of the phenomena) (Maxell 1992).

The current study utilized method triangulation (Denzin 2009) by using data archives, theme-bases interviews, and both direct and participative observation. Furthermore, also researcher triangulation was utilized during the study by analyzing part of the data from different viewpoints and publishing scientific articles based on the data together with colleagues, and by having a huge amount of official and unofficial team coach meetings.

Generalizability of the findings

Generalizability refers to the extent to which the results can be extended to other organizations, persons, times or settings (Maxell 1992). Generalizability is divided into internal and external generalizability. Internal generalizability refers to generalizing the findings within the community, group or institution studied to persons, events and settings that were not directly observed or interviewed.

Lincoln and Gupta (1985) discussed the generalizability of action research. According to them, if there is to be transferability, the burden of proof lies less with the original investigator than with the

77

person seeking to make an application elsewhere. The original inquirer cannot know the sites to which transferability might be sought, but the appliers can and do. The best advice is to give to anyone seeking to make a transfer to accumulate empirical evidence about contextual similarity; the responsibility of the original investigator ends in proving sufficient descriptive data to make such similarity judgments possible. Flick (2009) also sees the burden of proof in qualitative methods, especially with the Grounded theory method, in a similar way. He suggests that the discussion of validity should concern the transparency of the whole research process rather than concentrate on assessing individuals steps of the research process. Flick goes on by arguing that the application of qualitative methods should be judged for their soundness with regard to embedding them in the process of research and to the issue of the study and less for its own sake.

The concept generalizability in the context of qualitative study is somehow confusing. A qualitative study is usually embedded in a rich context where a group of individuals interact. The objective is in most cases to describe and interpret the participants’ expressions on the topic. The situation similar with the ICT-ESP study as well. The objective of the ICT-ESP study was not to generate generalizable results - instead it was to find out how the current deployment of the ICT-ESP would fit into IT Bachelor education from the viewpoints of the team entrepreneurs studying in it and from the team coaches involved in the team coaching process of it. However, the results may be

analytically (not statistically) generalized, meaning that the findings gained from the ICT-ESP study may reflect the findings of other studies.

The over four-year period of the ICT-ESP study offered a huge amount of qualitative data of the research target and the phenomena around it. Having the research target near to the researcher’s daily work made it possible to utilize theoretical sampling whenever needed to validate the findings and to gain more understanding about the studied phenomena. In addition to the amount of available data, the researcher worked closely with other team coaches – both the team coaches of IT

cooperatives and later with the team coaches of the marketing cooperatives - and was able to discuss and sometimes also test and challenge the assumptions about the team coaching process (searching the negative cases) on a daily basis. The research context made it possible to cross-validate the findings by method and researcher triangulation. The rhythm of the academic work within the University of Applied Sciences with long summer holidays also offered the author the possibility to detach himself from the study for a while and then again access it with a fresh brain.

78

The research process started with theme-based interviews and direct observation, turning later into action research and participative observation, where the author acted as an active agent for change.

The use of the constant comparative method, theoretical sampling, and official and unofficial discussions with the team coaches within the action research framework strengthened the reliability and validity of the ICT-ESP study. The continuous reflection about “what was going on with the ICT-ESP” that took place at several levels with both the team entrepreneurs and team coaches has proved to be a useful tool to guarantee the confirmability of the ICT-ESP study.

The overall research process and the results of the ICT-ESP study were described in a solid and transparent manner (Chapter 3), and in addition to the original papers, quotations from the interviews, illustrations and tables were represented to offer the reader a possibility to make his / her own judgments of the interpretations made. The results were represented as they were with no overstating and they were considered to be true. The author represented his own epistemological and ontological standpoints and possible limitations caused by bias (Chapter 3.2.6) to provide the reader the possibility to evaluate the dependability and credibility of the ICT-ESP study.

The author was able to answer the research question and its subquestions presented in the beginning of the study. The roles and responsibilities of both the student studying at the ICT-ESP and the team coach were clarified. In addition, the challenges faced were clearly described and the coaching processes used was represented and suggestions to develop were represented. The results gained were reasonable and a part of them were already validated in the coaching of the team

entrepreneurs. Suggestions were made for those who were interested in considering the ICT- ESP as an approach for the IT Bachelor education and thus the results were also relevant and useful. From the viewpoint of the objectives set, the ICT-ESP study can be considered as a success.