The online Q Sort process

To perform the assessment the participant was required to navigate to the questionnaire web page that was accessible on VTT’s public servers until April 2016. After navigating to the web address, participants are greeted with a welcome message (see the screenshot in Image 4 below) informing them they are at the right location.

Image 4: Welcome page

After clicking on the Continue button as prompted, the participant was asked to give their consent to participate in the research by using a login code (see Image 5). This page requested that the participant read the Research Information Sheet (RIS) that contains a brief introduction to the research, a description of what data would be gathered and stored, and the information on consent.

By entering a universal user-code provided in the consent information, the participant was

effectively consenting to participate in the research according to the information given in the RIS.

Image 5 Login page

Before the first step of the sorting process, an information page is displayed which gives an

overview of the research and what is being asked of the participant. Much of this information was also provided in the RIS (see Image 6).

Image 6: Introduction page

At the start of every task, an information window is displayed explaining how to perform the necessary tasks for each step (see Image 7 below). The first step consists of ‘pre-sorting’, that is, viewing the forty statements, one at a time, and sorting them into three categories, or ‘piles’ labelled

‘Most Unimportant’, ‘Neutral’ and ‘Most Important’ by dragging and dropping them, or by using the 1,2, or 3 buttons on the keyboard (see Image 7 below). The card appears in the centre of the screen (white card), and then the respondent places it in one of the three boxes using the method described above. The cards change colour according to which box they are assigned: pink for

‘Most Unimportant’, grey for ‘Neutral’, and green for ‘Most Important’.

Image 7: Step 1 of the sorting process- condition of instruction (left), and an example of how the task might look during the process (right).

The bottom of the screen contains a ‘progress bar’ which tells what percentage of the entire process is complete, and to the right of this is a ‘Help Me!’ button which can be pressed to see the

instructions once more.

Step 2 is the main sorting stage where the participants are asked to read through the statements again, beginning with the ‘Most Important’ pile, and place the four statements which they found to be most important in the ‘+4’ column, then to do the same for the ‘Most Unimportant’ pile, placing the statements in the ‘-4’ column (see Image 8 below). This process of moving back and forth from the opposite ends of the sorting table is repeated for the ‘+3’ and ‘-3’ columns, and then again until all the statements from the three piles have been placed into the table. The purpose of completing the polar opposites first is that participants are expected to be able to identify these statements with greatest ease, as these are the ones they feel most strongly about, as they move towards the centre they thus have fewer statements remaining in their piles and are able to compare these more easily.

Image 8: Step 2-the Q Sorting process

Once the sorting is complete, the ‘Continue’ button appears and allows the participants to move forward. After clicking on the ‘Continue’ button, the instructions for Step 3 (see Image 9 below) are shown. In this step, the participants are given the opportunity to double-check that they are happy with the way they have sorted statements, and are able to rearrange statements if necessary. Once they are happy with their sort, they click continue.

Image 9: Step 3- An opportunity to double-check the sort before moving on

Step 4 (see Image 10) asks the participants to explain why they selected the particular statements they placed under the ‘1’ and ‘9’ columns. Participants are asked to give open answers justifying their selections. Once complete, the participants again click on the ‘Continue’ button.

Image 10: Step 4- Participants are asked to give further information as to why they chose to place particular statements under the -4 and +4 columns.

Step 5 contains questions relating to the background of the participant and their experience with the questionnaire. Question 1 asks the participants to indicate whether they are male or female in order to help to assess the final distribution of participants. In Question 2 they are asked to rank on a one-to-five (disagree-to-agree) scale five statements. These are: In my view the questionnaire was interesting; I felt I have the experience to rank the statements; It was difficult to decide in which order I should rank the statements; I thought there were some important aspects which were not covered; I thought some statements overlapped each other. These statements are not expected to contribute to the overall results, but rather give feedback to the researcher about how the

participants viewed the exercise. Question 2.a allows participants to comment further on any of these issues in the provided text box should they wish. Question 3 asks about the capacity in which they have answered the questionnaire: either as a professional, or a member of the public. The distinction is made in order to clarify to which stakeholder group the participant belongs, although admittedly it does require self-reporting and self-assessment of one’s own “qualifications” in terms of experience. Members of the public are not asked to detail their profession, but Question 4 asks

‘Professionals with experience in the field’ to identify what ‘field’ they work in (their profession)

using a drop-down list. Question 5 asks how they would describe their employer using the same method. If the individual feels their profession and/employer are not covered by these categories, they can select ‘Other’ and give more information in a text box in Question 6. In this final question, participants are also asked to give more information such as the country in which they operate, clarifying comments on their profession (for example, someone who chose ‘Researcher’ may clarify their research focus), the name of their organisation, and any other comments they wish to leave.

Only Questions 1, 2, and 3 are marked with an asterisk denoting that they are mandatory for all participants, while ‘Professionals’ are strongly encouraged in the instructions to give responses to 4 and 5. Question 6 is noted as being optional, as some professionals may not wish to disclose their country of operation or organisation’s name. In reality, the HTMLQ application contains a number of bugs in regards to Step five, and thus the only questions that must be filled before the participant can move on to the Submit Data page are Questions 1 and 3.

After completing the mandatory sections in Step 5, the participant again clicks continue and is directed to the ‘Submit Data’ page where they click on the ‘Submit’ button to send their completed questionnaire to the researcher. Should the submission fail from some reason, a failure message is displayed with other options for submission: by email, or by printing.