Effect of different forms of data collection on reliability of

consultative auditing process, Asteri was developed through several phases while auditing different types of educational institutions. New elements were included in the process one at a time due to experience obtained while carrying out the audits. It was recognized by the author of this study that a comprehensive SSM system as well as its content and sections were not known by all auditees. The interest in safety and security matters as well as coverage of the SSM varied among the different educational organizations. Moreover, the way in which the SSM was organized varied among the educational institutions. The safety and security culture was different among the educational institutions, too. There were variations in personal responsibilities, tasks, job descriptions, knowledge and personalities. In addition, the motivation levels to be audited and to be consulted in the SSM were different.

The survey had a role in the development of the Asteri consultative auditing process, too. The key terms used in the survey were defined in the survey before the statements were given to ensure that the respondents understood the statements equally. Fowler Jr.

(1998, 344) states that there are five basic characteristics for questions and answers as far as validity is concerned. The questions should be consistently understood by the respondents, and they should be clearly communicated to the respondents. The respondent should know what an adequate answer is. Equally, unless knowledge is being measured, the respondent should have access to the information needed to be able to answer the question. Finally, the question should be such that the respondent is willing to answer.

There were several statements in the survey concerning the same phenomenal area to increase reliability of the measurement. Statistical analysis was used for analyzing the results of the survey. Factor analysis was performed to form the sum variables of the

statements of the survey (Appendix 5). Cronbach’s alpha was used to identify the internal reliability and consistency of the statements of the survey and, thus, to find out whether the statements measured the same matter.

As mentioned, observation was used for studying auditees’ attitudes on auditing at the starting point of the audit (initial attitude) and at the end of the audit (closing attitude) by using a scale of reserved, neutral and positive. Observation was based on monitoring auditees’ speech and body language. In some ESs, the auditors got the feeling that the organization was not participating voluntarily in the audit, and it was reflected in the reserved attitude. In a few other cases, the attitude of the auditees of the educational institutions changed from neutral to reserved most likely because of guilt or disappointment when the prior expectations of high scores were not met. Investigator triangulation was used as two auditors observed the attitudes together. The audited organization received one general score on initial attitude and one general score on closing attitude. Often there were many participants in the audit, and the overall attitude of the group was recorded. Both auditors made observations by themselves, and only twice were the scores of the researchers unequal. This happened in the early stages of the development of the consultative auditing process. The auditors found a common understanding through discussion and thus adopted joint scores for initial and closing attitudes. There is reason to believe that the results concerning the attitudes toward the auditing are reliable and valid.

The TUTOR model was chosen because of its wide-ranging way of viewing the SSM.

Thus, the content validity was improved. The reason for choosing the Central Uusimaa region was that the TUTOR model was created by the Keski-Uusimaa Department for Rescue Services, and the model will be used in the future for auditing educational institutions of this region.

Statistical analysis was used for analyzing the results of the survey, observation and auditing. A graphical examination of the figures was also conducted. According to Nummenmaa (2011, 14–15, 148, 158–159) and Tähtinen, Laakkonen and Broberg

(2011, 12, 15) in an empirical study, common features and regularities are sought in the investigated sample. In statistical reasoning, the way the phenomenon occurs in a larger population is predicted. The selected sample should have as united characteristics as possible among the general population. It is not evident that the phenomenon occurring in the sample will also occur in the population. Therefore, statistical analysis is used to clarify how likely the phenomenon in the sample is to occur in the population, too. The suitability of statistical methods affects the reliability of the study. Additionally, reliable statistical reasoning requires sampling criteria which, in turn, have an effect on how the results can be generalized. If non-participation bias exists, it reduces the sample size and affects the representativeness of the sample. According to Nummenmaa (2011, 154), the form of the research data distribution affects the suitability of the statistical analysis method. Before making a statistical analysis of the material, the normal distribution of variables was examined. In this study, non-parametric tests were used because the variables deviated from the normal distribution.

Maxwell (1998, 93) argues that triangulation reduces the risk of systematic distortion.

Hirsjärvi, Remes and Sajavaara (2009, 233) mention that different kinds of triangulation exist, such as investigator triangulation, methodological triangulation, data triangulation, theory triangulation and, moreover, mixing methods. In this study, triangulation was used to enhance the confidence in the research findings. Theoretical triangulation was applied using multiple perspectives in the SSM while preparing for the audits, creating the survey, auditing and consulting. Methodological triangulation was used when studying the effects of the consultative auditing process by means of the survey and observation. Feedback about the Asteri consultative auditing process was collected via the survey. The feedback was gathered from all participants of the audited educational institutions through their contact person, and it was received from 61 respondents representing 40 different educational institutions out of 76 (52.6%). There is a reason to believe that the results can be generalized to all Finnish EAs and UASs.

The study could have been performed in another way. The presentation of the TUTOR model, the link to the survey and the audit results could have been sent directly to all

participants of the audit instead of sending them to the contact person of the organization. In addition, on the survey the respondents were asked to choose the six most important development targets of the workplace for the next three years in order of importance after the audit. It would have been easier for the respondents to choose the three most important development targets without putting them in priority. Finally, in the survey, the statements concerning the effects on concrete actions could have been asked later – for example, six months after the audit. This would have provided time for audited organizations to take corrective actions.