Reliability, validity, and ethics

Some authors claim that qualitative research generally is lacking rigour and is biased, as it is to a certain extent unavoidable that the researcher forms his or her own opinion about the analysed data (Chapter Eight How to Analyse Survey Data: A Comprehensive Guide, 2020). Claire Anderson (2010) and others, however, insist that qualitative research can be unbiased and generate in depth, valid, reliable, credible and rigorous results when it is carried out properly. In this context it is crucial to examine the data for reliability and to assess the

objectivity and credibility of the research (Anderson, 2010). This chapter elaborates issues connected to the reliability, validity and ethics of the study, especially regarding the data gathered and analysed within the scope of the member survey and the co-design workshop²⁸.

Reliability relates to the reproducibility and stability of data and asks if a study would have the same results if it was replicated (Morse and Richards, 2002).

One of the key issues that can compromise the reliability of research is subjectivity (Wilson, 2010). In this study, the data gathering and analysis were to a large extent done by a single researcher, and the level of her subjectivity is unknown (Babbie, 2012). However, the researcher tried to take measures to enhance the reliability of the results, for example by choosing the appropriate methodology for the study, deciding on an appropriate timescale as well as a suitable sample for the study (Chilisa and Kawulich, 2012: 51). Furthermore, she indicated individual extrapolations from the data to make sure to distinguish objective facts from subjective opinions (Anderson, 2010). For Murphy, Dingwall, Greatbatch, Parker and Watson (1999) it is one of the strengths of qualitative research that data is always recognized and understood in relation to the context of their production. Hence, the author elaborated the chosen methods and analytical approach in detail and justified them in light of the research question and strategic purpose of the thesis project.

Validity describes the extent to which a measure accurately represents the concept it claims to measure or as McLain and Kim (2018: 113) state “the extent to which a concept, conclusion or measurement is well-founded and likely corresponds accurately to the real world.” Validity can be a challenge in qualitative research, as it is likely that the researcher has some kind of influence on participants and therewith on the research findings. Furthermore, as qualitative studies mostly look at smaller samples, reproduction and generalization naturally can be problematic (Bengtsson, 2016). As this study is a small-scale research conducted for a specific organization in a specific context, any generalizations should be considered carefully. While issues of

28 Although the co-design workshop is not a classical research method, it generated plenty of qualitative data and therefore is also considered here.

validity can never be completely eliminated in a qualitative study, the researcher made all attempts to minimize them throughout the thesis project.

Ethics refer to the claim that all research has to be conducted in an ethical manner. Lichtman (2012: 51) defines ethical behaviour as “a set of moral principles, rules, or standards governing a person or profession.” Within the scope of a qualitative study, a multitude of ethical questions can arise, for example in the course of data collection, data analysis and reporting (McLain and Kim, 2018: 113). In this context, ethical behaviour means ensuring the anonymity and well-being of all participants who willingly contribute to the research by providing their insights. This includes informing participants on the purpose of the research and on whether and how their activities will be documented. Also, participants should be informed beforehand how their data will be stored, treated and reported. All acquired research data must be treated carefully and respectfully. (Guidelines on anonymity and confidentiality in research, 2017; McLain and Kim, 2018)

Member survey

The member survey was created in a mixed method approach, intending to generate mainly qualitative, but also quantitative data on the issue. One main purpose of the survey was to let participants reflect and report about their experiences with digital learning offers of the Alumniportal Deutschland as well as of other providers. Generally, online surveys can cover a large population.

However, it is not clear how much effort respondents take in answering the survey. In effect, the acquired data can be superficial. (Ojasalo et al. 2014: 121) It can be assumed, that those who decided to volunteer in the survey, had an interest in the research topic and developing the learning offers on the Alumniportal Deutschland.

The sampling method used for the survey is self-selection sampling, which means that individuals were able to decide to take part in the research of their own will and accord (Saunders, Lewis, and Thornhill, 2012: 241). All individuals that received the link to the survey were eligible to participate, irrespective of age, gender, ethnicity, or other factors. Participation in the survey was voluntary, and individuals were free to not participate. The author tried to enable

as many individuals of the population as possible to take part in the survey, by distributing the link to the online survey through different channels and by having it available online for three weeks. Furthermore, the questionnaire was made available in English and German language. According to Saunders et al.

(2012), the likelihood of a self-selection sample to be representative for the whole population is rather low, but that it can be useful in studies like the one at hand, where exploratory research is needed. Nonresponse bias of the survey data was checked by comparing the socio-demographic characteristics of the respondents with those of the entire population. The check suggests that non-response bias is unlikely to be present regarding these aspects. However, as the survey was carried out as an online survey, there might be a non-response bias in this regard, as people with limited access to the internet or a poor internet connection might not have been able or willing to participate. The same is true for the co-design workshop. However, as for a virtual alumni portal and digital learning opportunities, the main target group generally consists of people who have sufficient access to the internet, the author assumes that the online survey and digital workshop were able to reach key users and that the generated data hence can to a certain extent describe the beliefs of the alumni population on the platform.

Participants were informed about the purpose and nature of the survey in the introductory text. Also, the introductory text addressed issues of data privacy and protection: all individuals participating in the research study were guaranteed privacy and anonymity. Furthermore, they were assured that all information provided to the researcher was stored at secure levels and treated in a confidential manner. Participants were given the possibility to withdraw their data from the survey and were informed a period of a time during which they could do that.

Co-design workshop

The co-design workshop aimed at exploring user’s experiences and needs more deeply and at gaining a more in-depth understanding of the different viewpoints and expectations regarding digital learning generally as well as on

the Alumniportal Deutschland specifically. Another objective was to collect ideas on how offers for alumni learning should be designed to later include these insights in the redesign of the learning services. As only a small group of the population could be included in the co-design workshop, it was important to select a preferably representative sample of participants. Selecting a sample with reference to relevant variables can help to ensure validity in research (Roberts, Priest and Traynor, 2006: 43). Also, the researcher had no personal connection to the participants that were included in the co-design workshop. For the co-design workshop a two-tier procedure, consisting of a self-selection process as well as purposeful selection by the researcher, was used. In a purposive sampling method, the target audience is selected with the specific purpose of the workshop in mind. That purpose reflects the particular qualities of the participants chosen and their relevance to the topic of investigation.

(Roberts et al., 2006). The requirements for participants comprised sufficient command of English language, technical premises (notebook, laptop or PC with camera and microphone) as well as a stable internet connection. As suggested by literature, the researcher defined diverse characteristics as sample selection criteria before the sampling and tried to select participants that represent different users and cases with regard to gender, age, nationality, fields of study or work, as well as educational level (Palinkas, Horwitz, Green, Wisdom, Duan and Hoagwood, 2015; Patton, 2002; Roberts et al., 2006). The required exploratory work to determine the criteria and nature of the variation was done within the scope of the preceding member survey. While the sample cannot be regarded to be fully representative of the entire alumni population of the Alumniportal Deutschland, the author is sure that the collected data can still

“describe and explain the key themes that can be observed” (Saunders et al., 2012: 239). Even in small samples completely different cases can be present, and patterns can emerge (Patton, 2002; Saunders et al., 2012). Anderson (2010) supports this view: While data and findings from qualitative research, which are usually collected from few individuals, cannot be generalized, its findings can still be transferable to other settings. It can be assumed that the non-response bias was higher for the co-design workshop than for the member survey, as the barriers for participation, due to the selection process as well as the concrete timing and duration of the workshop were higher. Also, the

workshop was conducted in English language only, which automatically excluded individuals without sufficient proficiency of English.

Validity can be a challenge especially in the data collection phase of a workshop, as the researcher likely has some kind of influence on the participants and therewith on the research findings (McLain and Kim 2018:

113). This is also true for the co-design workshop, in which a lot of interaction between the researcher and the participants took place, as the researcher participated in the co-design workshop as facilitator and observer. The fact that the researcher’s facilitator role might have influenced workshop participants to a certain degree has to be taken into consideration when evaluating the objectivity and validity of the workshop findings (Anderson, 2010). Also, it is possible that participants felt compelled to discuss notions of digital learning on the portal in an affirmative way, as the researcher is also an employee of the portal. The researcher tried to minimize the effects of her presence on participants by trying to be as objective as possible and take a neutral stance throughout the workshop. Furthermore, workshop participants were asked to depict their actual experiences and opinions and include individual examples and details of their own context where possible. Still, it would have been advisable to bring in an external facilitator for the workshop to minimize the researchers influence on workshop participants, but resources did not allow it.

(The Role of a Facilitator: Guiding an Event Through to a Successful Conclusion, n.d.)

The co-design workshop was carried out in a video conference, which offers certain advantages like a synchronous form of communication over long distances. Video conferencing enables participants to directly react and respond to what others say or do, and it enables the facilitator to read social cues, which are often absent in written and asynchronous communication (Opdenakker, 2006). However, videoconferencing can also bring along disadvantages, as participants may feel embarrassed or nervous to be on camera and this can have an impact on how openly and in-depth they share their thoughts throughout the workshop (Anderson, 2010; Hay-Gibson, 2009: 43). One can assume that due to the diversity of the workshop participants aspects like linguistic and cultural asymmetry also influenced the dynamic and outcomes of

the workshop (Au, 2019: 58). As suggested by Au (2019), to dismantle power imbalances and social distance, the author worked on naturalizing the workshop situation into a collaborative and conversational form. The author tried to create a safe space, in which participants felt able to engage freely in open communication and give full and frank responses despite knowing that they were being observed and their activities were being documented. The positive feedback from participants suggests that these goals for the workshop were achieved. One person also explicitly emphasized the friendly workshop atmosphere in the feedback form.

Before the workshop, participants were approached with a personal email explaining the purpose of the co-design workshop, the data collection and analysis methods and were given clear examples of how their data would be processed. Furthermore, participants were informed about the utilized tools and their privacy protection notices. The author explained to participants that data obtained during the workshop was confidential and was only analysed for the purpose of the development project, stored at secure levels and not passed on to other individuals and organizations. In this context, participants were also assured that their identity would be kept confidential in the report of the workshop results.

Besides the data acquisition phase, validity can also be an issue in the data analysis and interpretation phase. As the researcher also is an employee and has a role within the organization, this can lead to an evaluation bias and might therewith affect the research (Wilholt, 2009). Therefore, the author paid special attention to issues of validity while analysing the workshop data in a thematic analysis. As thematic analysis it is subjective analysis method and reliant on the researcher’s judgement (Caulfield, 2020; Morse and Richards, 2002), the author decided to include two team members in the workshop. The team members also assessed the data analysis to make sure that the inferences the author made from the text, were not individual but could be repeated by others as well (Anderson, 2010). Last but not least, the author assessed the workshop findings against the findings of the member survey and other methods to ensure that they are reasonable and logical (Morse and Richards, 2002).

4 INSIGHTS GAINED FROM RESEARCH AND SERVICE DESIGN