Discussion: evaluation and limitations

The strategic purpose of the thesis project was to analyse the digital learning offers on the Alumniportal Deutschland in a Service Design process with the goal of designing a new streamlined learning service which increases user satisfaction and engagement. This purpose was divided into two primary aims for the study: 1) to analyse the existing digital learning offers for alumni on the portal; 2) to develop ideas and solutions on how to improve those learning offers. The outcomes of this process have been ten conceptual and practical recommendations on how to advance the digital learning offers in the future.

To reach the strategic objectives of the thesis project, the author conducted a literature review as well as a member survey and applied a number of SD

methods. This chapter examines to what extent the objectives of the thesis project have been achieved. To analyse the digital learning offers with the purpose of gaining a comprehensive understanding of the service, its users as well as the service environment, research and SD methods have been applied The guiding research question was how offers for digital learning on a national alumni portal should look like to add value to the core users and at the same time to be effective from the providers’ perspective. The author believes that the applied methods generated in-depth insights, especially regarding the question of how to add value to users that can serve as a fruitful base for further development of the learning offers. As the findings have been combined with an extensive review of literature and previous research, it can be assumed that the thesis offers a thoughtful view of digital alumni learning. Regarding the second objective, concrete ideas and solutions on how to improve the learning offers have been developed based on the findings of the literature review, research and SD methods. Based on these three pillars and the patterns that emerged within the scope of the research and SD methods, the author formulated ten conceptual and practical recommendations on how to revise and develop the learning offers in the future. The author believes that these recommendations can provide strategic direction for the staff of the Alumniportal Deutschland and could be useful to other providers in the market as well.

The main innovation of the thesis is that alumni as core users were included not only in the member survey, but also in the trend mapping and the co-design workshop. By including alumni in the development process, the author aimed at ensuring that the service innovations meet the specific demands of the target audience. One of the recommendations presented in chapter 5 is to keep including alumni in future activities regarding the design and delivery of offers for digital LLL on the Alumniportal Deutschland, e.g. as experts and lecturers in a peer-to-peer approach.

As with all research and development work, there are also limitations to the study at hand. The biggest limitation is that at the time being it cannot be assessed if the proposed changes to the digital learning offers on the portal will actually add value to users and will be effective from the provider’s point of view. Further development work and evaluation will be needed after

implementation to asses if users find added value in the new services and service elements and if user satisfaction and engagement could be increased.

The Alumniportal team should continue to constantly evaluate the offers and collect users’ feedback. Besides survey and feedback forms, further qualitative research could be helpful in this regard, e.g. in the form of focus groups or interviews. As to the effects on the provider, Barnard and Rensleigh (2008) are convinced that offering valuable learning opportunities to alumni can enable organizations to establish active and engaged alumni communities with sustainable relationships with institutions. While authors like Brant and Regan (2002) and Iskhakova et al. (2017) argue that it is not easy to assess and quantify the points of contact of organizations with their alumni to reliably evaluate the effects of alumni strategies and programs, the thesis author is positive that return on engagement (ROE) or return on relationship (ROR) can serve as significant metrics to measure the effects of the revised learning offers.

ROE and ROR are qualitative metrics that measure the overall brand strength gained from a particular action, strategy or content (Different Types of Marketing ROI: Return on Engagement, n.d.) as well as the value that is generated due to nurturing a relationship (Rubin, 2011). The indicators to measure ROE or ROR can vary from organization to organization and should be defined individually. Relevant indicators to assess the Alumniportal’s learning offers could for example be the number of users who participate in learning offers, site visits and retention time, the share of participants who give feedback on the learning offers, participants rating of the offers, the number of new users, the repeat engagement of users, and the number of comments or discussions started. (Haring, 2020)

Further limitations of the study are connected to the research part. With the member survey, the authors aimed at gaining a comprehensive overview and covering a number of issues related to the topic of digital LLL offers on the portal and in general. At the same time, she had to limit the number of questions and especially open questions, to ensure that participants would complete the survey. The ideal procedure would have been to approach this issue with a combination of in-depth, qualitative interviews with a small number of users, supplemented by a quantitative study in the form of a more specific survey. The survey data could then have been analysed to test if and to what

extend the qualitative data generated through the interviews are true for a larger alumni population. Furthermore, segmentation of users based on the survey data was not as successful as expected. By applying other demographics in the future, significant alumni segments could be identified. As user segmentation provides a basis for effective information and communication measures towards alumni (Barnard and Rensleigh, 2008), it could be worth looking at the survey data again to identify user segments based on alternative criteria and to focus more strongly on user segmentation in the course of future research and development activities. However, as stated by Cannon (2015b), segmenting the target audience can also be a challenge for online alumni portals.

Limitations also exist regarding the Service Design methods. Usually, SD projects have a medium to long duration of eight months or longer. As these projects directly involve users and are based on their contributions, they are highly dependent on users’ time and resources. (Innovate with Design Thinking:

pros and cons, 2019) Within the scope of the thesis project, only a partial version of the SD process with limited purposes could be realized: the study only covers the first phases of the SD process. The first SD phases usually take the longest time and should be done carefully as they are the backbone of the whole development process (Ojasalo et al., 2015). The thesis does not cover the later stages of prototyping, testing, implementing and evaluating the new services or service aspects.

The author is convinced that significant patterns and meaningful ideas evolved during the conducted co-design workshop. However, like in qualitative research methods as focus groups, the outcome of a workshop is highly dependent on participants’ comments and reactions and the results therefore might not be reproducible. Furthermore, the small sample of the co-design workshop is another limitation of this study: there is no guarantee that the sample is representative of the overall alumni population. At the same time, a co-design workshop is not precise science and its aim was not to achieve statistical generalizations. Rather, the intent of the workshop was describing alumni’s experiences, understanding their needs and expectations more deeply, and collaboratively generating a larger number of ideas. The author tried to back the generated ideas and findings of the workshop with findings from the literature

review, research data and other SD methods. The same was done for the assumption based SD methods, like the journey maps, personas, and the trend mapping. One major drawback of the conducted trend mapping was the lack of moderation by the researcher and consequently a lack of interactivity among participants. Possibly, the method could have generated more relevant results if the researcher had conducted a moderated, interactive trend mapping session with selected participants (Guide to Trend Mapping, 2016).

Looking at the context and the scale of the thesis project, it becomes clear that the study was a small-scale research and development project conducted for a specific organization in a specific context. Accordingly, the research findings are tailored for implementation by the commissioner Alumniportal Deutschland.

Therefore, any generalizations regarding alumni learning for other alumni organizations, portals, or in general should be considered carefully (Henriksen et al., 2017; McLain and Kim, 2018). However, the generated insights might have value as a reference for other providers in different contexts as well.

Furthermore, the user-centric approach of the SD process could be guiding principles for other institutions and alumni platforms to generate useful insights into alumni behaviour, expectations, and needs.