Evaluation of the study

While the focus of this study was student and learner-centric, the validity and reliability of this study rely solely on myself as the researcher. The selection of the topic for this study was based on my work with ESP in a Finnish university context since 2002, my experiences with students of Business and Economics and my professional

interests in non-formally and informally developed ESP proficiency and the RPL process for ESP at my institution. Having been involved with the teaching and testing of ESP for various study programmes for over a decade and with the ESP exemption examinations for Business and Economics at the UEF Language Centre from their beginning in 2011, I was aware that the examinations, while functional and typically appreciated by those students who attended them, were not perfect yet an effective way to develop them would be from the perspective of the participants, the students.

Because of my position at the UEF Language Centre as an ESP lecturer and RPL assessor involved with the ESP exemption examinations, this research was inherently practitioner research and insider research as I was investigating my own university and students (Mercer 2007; Trowler 2012), which also created significant potential for researcher bias. While this study allowed me the advantages of practitioner and insider research with the convenience of data collection, access to the research situation and relevance of the research topic (Mercer 2007, 6; Punch 2009, 43–44), I also had to be conscious and constantly aware of the potential subjectivity and ethical complications in relation to issues such as study subjects, power relations, professional practice and institutional policies (cf. Drake & Heath 2011, 53–56).

To counter the subjectivity and researcher bias, throughout the research process I employed strategies to ensure the credibility of the research such as including internal and external supervision to encourage reflexivity and constant monitoring of my practices during the process and data collection (Johnson & Christensen 2012, 265; Smyth & Holian 2008, 42–43). At all stages of the data collection, while involved as both the researcher and the ESP lecturer/RPL assessor, I emphasised the anonymity and the objectivity of the research data to encourage the RPL participants and the non-participants, some of whom I was previously familiar with, to provide honest and unrestricted views about their learning of ESP, opinions about RPL and their experiences with the ESP exemption examinations.

Therefore despite my own subjective premise for this study, throughout the research process and in its end result, this written contribution, I attempted to the best of my ability to document and justify my choices to allow the reader to follow and evaluate the quality of the results and mixed methods related inferences as well as the consistency of my interpretations. With the use of the mixed methods research design I attempted to minimise any bias in the data so that the use of several methods for the two groups of Business and Economics students at UEF, the RPL participants and the non-participants and although purposefully sampled, would lessen the validity issues affecting the methods and particularly the qualitative interviews. Conversely, the potential threat to the measurement validity in this study, the low response rate of 14.3 % in the electronic survey for the non-participants, was also countered with the qualitative components in the survey items. However, ideally the qualitative data in this study would have been more extensive to allow for more sophisticated statistical analysis to be conducted so that while respondent numbers could not be influenced beyond the measures already taken, adopting more Likert-scale items to both the questionnaire for the RPL participants (appendix 4) and the electronic survey for the non-participants (appendix 7) would have produced more numerical data for statistical analysis.

119 Overall it could be argued that the research process included conscious monitoring of the construct validity of the study, i.e. the permeation of validity in all phases of the process (Leech, Dellinger, Brannagan & Tanaka 2010, 20). This approach included the carefully considered use of rigorous, consistent and transparent methods, instruments and procedures for the data collection and analysis in all phases of the study, as well as the selection of participants consistent with the overall purpose of the study. The mixed methods process ultimately yielded meta-inferences grounded in both the quantitative and qualitative data and analysis, as presented in the chapter

“Results and Discussion”, with attention paid throughout to interpretative and theoretical consistency, interpretive distinctiveness and integrative efficacy (Teddlie

& Tashakkori 2009, 302–308).

To reduce potential threats to the validity of this study, I also approached and conducted this study with consideration of the levels of legitimisation employed frequently in the assessment of mixed methods research (Cohen, Manion & Morrison 2011, 22; Johnson & Christensen 2012, 273–275; Onwuegbuzie & Johnson 2006, 57), and this study can be argued to represent five levels of legitimisation: sample integration, inside-outside, weakness minimisation, paradigmatic mixing, and multiple validities, explicated in more detail in table 7.1.

Table 7.1. Five legitimisation checks for the study (adapted from Onwuegbuzie & Johnson 2006, 57)

Level of legitimisation Description and connection to this study

Sample integration The relationship between the quantitative and the qualitative sampling designs in this study can be argued to yield quality meta-inferences.

Inside-outside (emic-etic) The researcher has attempted to accurately present and appropriately utilise the insider’s view (as a practitioner and insider researcher) and the observers’ views (per-ceptions of the participants) for purposes such as description and explanation, also with the use of accurate reporting of description information.

Weakness minimisation The weakness from one approach (e.g. only QUAN) has been compensated twice with the selection of the initial mixed methods design QUAN + QUAL and consequently supplemented with the sequential phase → quan since issues requiring more data were raised during the initial data collection process.

Paradigmatic mixing The researcher’s paradigmatic approach through pragmatism and phenomenogra-phy have been the foundation for the selection of the quantitative and qualitative approaches and the overall research design.

Multiple validities The overall research validity has been the result of quantitative, qualitative and mixed validity types as documented in the research process and also narrated in the evalua-tion secevalua-tion of the study.

As this study also employed phenomenography as the qualitative approach so as to highlight the significance of student perceptions in relation to non-formal and informal learning and its recognition, the validity of this study from a phenomenographic standpoint should also be evaluated. Generalisability, for instance, is rarely the target of phenomenographic research which typically focuses on a small number of individuals chosen from a particular population (Marton & Booth 1997, 125), such as in this study. Therefore critics of phenomenography often voice concern about the potential lack of validity, lack of prediction and researcher bias (Bowden 2000, 1),

with Uljens (1996, 125) maintaining that data and results from phenomenographic methods are ultimately a result of a phenomenographic reduction connected to the researcher’s own understanding of the content, i.e. an explication of the researcher’s experience of the data only assumed to represent the conceptions of the participants.

On the other hand Ashworth and Lucas (1998, 417) have argued that the conceptions of students investigated through phenomenography are those that would be expressed even if unaided yet the research process requires the medium and the inevitable interpretation of the researcher.

To ensure the views and conceptions of the students, and not my own, were represented as the results and inferences in this study, throughout the research process I consciously applied bracketing, which in phenomenography refers to setting aside prior assumptions about the nature of the phenomenon under study (Ashworth

& Lucas 1998, 418; 2000, 297; Bowden 2005, 15; Marton 1994, 4428). The purpose was thus to give voice to the perceptions of the students through e.g. descriptive validity of using direct quotations of the participants’ interviews and interpretative validity of accurately portraying the meanings of the participants’ viewpoints and experiences (Johnson & Christensen 2012, 265–266).

Traditionally phenomenographic research has also encouraged reliability through the eyes of many, i.e. interjudge reliability (Säljö 1988, 45) so that ideally a team of researchers would be involved in the analysis of the interview transcripts and the generation of the categories of description. However, more recently scholars have questioned the reliability of involving several individuals and their several interpretations, with Sandberg (1996, 134) viewing interjudge reliability as questionable as it can hinder the achievement of faithful categories of description and instead potentially create theoretical and methodological inconsistency. Also Prosser (2000, 44) has claimed the use of two or more researchers in phenomenography may be beneficial in the actual interview stage but is inherently more problematic in the constitution of the categories of description, with Åkerlind (2005, 70) also emphasising that substantial contributions to phenomenographic research can be accomplished by an individual researcher solely in charge of the interview data collection and analysis, particularly in doctoral dissertation projects such as this study.

Nevertheless, at least one shortcoming in this study with regard to purely phenomenographic research must be admitted. In the questionnaire (appendix 4) RPL participants were asked about their experiences of developing academic and field-specific English outside the classroom and some typical non-formal and informal learning environments were listed in the questionnaire to entice an influx of experiences, situations and responses. In the subsequent interviews (appendix 6) with the same RPL participants the more expansive and detailed descriptions of their non-formal and innon-formal learning environments provided by the same RPL participants can therefore be argued to have been at least in some respects guided and influenced by the earlier questionnaire process. Therefore the purpose of the interviews in this mixed methods study was not to be the sole source of typology concerning non-formal and informal learning and RPL as in a purely single-method phenomenographic study, but to function as a qualitative method with a phenomenographic approach to reach a deeper understanding of the students’ experiences, perceptions and situations

121 they conveyed as inherently associated with non-formal and informal learning of ESP.

Also, because of the implicit and flexible nature of informal learning, it is unlikely that this study’s data collection methods were able to capture all the varieties of tacit, incidental or unintentional informal learning acquired by the participants, and some may argue, as e.g. Dismore et al. (2011, 317) have, that informal or tacit learning does not even require explication or description as it has value in itself. Yet there was purportedly also value in discovering a selection of non-formal and informal learning environments for ESP learning to better understand how university students today learn field-specific language and communication skills.