Leadership and Management of Education : Evaluation of Education at the University of Helsinki 2007-2008

(1)

Leadership and Management of Education

Evaluation of Education

at the University of Helsinki 2007–2008

University of Helsinki

Administrative Publications 58 Evaluations

ISBN 978-952-10-5255-2 (paperback) ISBN 978-952-10-5256-9 (PDF) ISSN 1795-5408 (print) ISSN 1795-5513 (online)

Seppo Saari & Minna Frimodig (Eds.)

Evaluation of Education at the University of Helsinki 2007–2008

(2)

(3)

Leadership and Management of Education

Evaluation of Education

at the University of Helsinki 2007–2008

University of Helsinki

Administrative Publications 58 Evaluations

(4)

ISBN 978-952-10-5255-2 (paperback) ISBN 978-952-10-5256-9 (PDF) ISSN 1795-5408 (print) ISSN 1795-5513 (online)

Graphic design: Tapio Kovero and Juha Leppänen, Helsinki University Print Helsinki University Print

Helsinki 2009

www.helsinki.ﬁ /julkaisut

(5)

Foreword

As part of its quality assurance and strategic development, the University of Helsinki regularly conducts evaluations of its education and research. The previous evaluation of education was quite broad, covering all teaching and degree programs at the University of Helsinki. It was carried out between 2001 and 2002.

Among many other recommendations, a suggestion for the evaluation of the management of education was introduced.

When planning a new evaluation of education, the Vice Deans for academic affairs as well as the Committee of Academic Affairs made the initiative to focus on the leadership and management of education in the next evaluation. It was seen as a topical issue connected to the reform of University law that will change universities’ legal status and management in Finland. In connection with the reform, the University of Helsinki will reorganise its management and operation systems, including decision-making procedures.

When the evaluation project was planned, it was not known how quickly the upcoming university reform in Finland would be launched. Now, this report is being published in the middle of the university reform. The report offers the University recommendations for strategic planning at an opportune moment and enables the University to beneﬁ t from outside expert views in the upcoming process of change.

The selected focus of the present evaluation is the management of education, investigated from the viewpoint of both leadership and management. This approach to university-level teaching can be deemed to be novel also on the international level.

Self-evaluations in the faculties and departments were implemented in the form of cooperative learning processes. Enhancement-led evaluation has always been the governing principle at the University of Helsinki, and it was implemented also in this evaluation as the University community assessed the strengths and weaknesses of the leadership and management of education. The faculty-speciﬁ c evaluations were produced on the basis of departmental self-evaluations.

The external evaluation was conducted by an international panel consisting of 12 experts. The panel was chaired by Professor, Vice-Rector Eva Åkesson from Lund University, who was responsible for the panel’s performance, the practical division of labour within the panel and the relevance and high quality of the evaluation results. The panel had at its disposal self-evaluation reports by the University and its faculties, including the Language Centre and the Swedish School of Social Sciences. The panel was also provided with strategies and documents steering the

(8)

University’s operations and additional materials requested by the panel, as well as interviews of over 400 members of the academic community.

The University requires a critical viewpoint on its leadership and management of education. The strengths, good practices and development proposals presented in the evaluation report offer both a detailed analysis of the present state of affairs and recommendations for future development. The good practices and recommendations that emerged in the evaluation will be used in the University’s strategic planning. It will be necessary to follow up on the implementation of the panel’s recommendations in the coming years.

The University of Helsinki wishes to express its deepest gratitude to the international evaluation panel for their thorough analysis, felicitous conclusions and recommendations reaching far into the future.

Many thanks are due to the entire University community for participating in the production of the self-evaluation reports, interviews and reviews of the evaluation materials.

The contribution of the Faculty coordinators of the evaluation has been excellent and greatly appreciated, as are the efforts of the Evaluation Steering Group in supervising the evaluation policy and procedures.

Special thanks are due to the Academic Affairs Unit, which has been responsible for all the practical preparations of the evaluation. Ms Minna Frimodig, Advisor in the Academic Affairs Unit, has been responsible for the entire project from the very beginning to the very end. Senior Advisor, Dr Seppo Saari’s expertise has been contributing to the project since August 2008. Many thanks to both experts. Your excellent work is greatly appreciated.

Hannele Niemi Vice-Rector

Chair of the Evaluation Steering Group

(9)

Evaluation panel

Chairperson of the panel

Eva Åkesson, Professor, Vice-Rector of Lund University

Vice-Chair of the panel

Patric Dillon, Emeritus Professor, University of Exeter

Members of the sub-groups Sub-group 1:

Deans/vice-deans in charge of academic affairs, Faculty of Biosciences, Faculty of Medicine, Faculty of Science and Faculty of Pharmacy

Erik De Corte

Emeritus Professor of Educational Psychology

•

Faculty of Psychology and Educational

•

Sciences

University of Leuven, Belgium

•

Areas of expertise:

Learning and instruction, design of learning environments, assessment of learning, problem solving, quality assessment in higher education

C. Alan Lyles

Professor of Public, Private and Nonproﬁ t

•

Partnerships

Division of Government and Public

•

Administration

University of Baltimore, United States

•

Managing pharmaceutical supply, docent of Pharmaceutical Policy and Pharmacoeconomics, member of several evaluation boards/panels

Pasi Sahlberg, chair of the sub-group 1 Adjunct Professor of Behavioural Sciences

•

(University of Helsinki) Lead Education Specialist

•

European Training Foundation, Italy

•

Pedagogical leadership, teaching methods, educational change, learning

Eva Åkesson

Professor of Chemical Physics

•

Vice-Rector of Lund University

•

Faculty of Science

•

Lund University, Sweden

•

Chemistry, physical chemistry, educational management and development, quality assurance and enhancement, accreditation of master degree, Bologna expert

(10)

Sub-group 2:

Support of pedagogic development and staff development, Faculty of Behavioural Sciences, Faculty of Law, Faculty of Social Sciences, Swedish School of Social Sciences and Faculty of Veterinary Medicine

Suvi Eriksson

Student of International Business

•

Communication Faculty of Humanities

•

University of Oulu, Finland

•

Secretary for Academic Affairs 2004-2007 (Student Union, University of Oulu), student involvement in QA

Aalt Willem Heringa

Professor of Comparative Constitutional and

•

Administrative Law Dean of the Law Faculty

•

Faculty of Law

•

Maastricht University, Netherlands

•

(Comparative) constitutional law, human rights, (higher) education management (dean; board of trustees of large school board;

management team Maastricht University)

Kirsten Hofgaard Lycke Professor of Education

•

Institute for Educational Research University

•

of Oslo, Norway

Educational change, quality assurance and development, problem-based learning,

enhancing learning environments, medical education

John Taylor, chair of the sub-group 2

Professor of Higher Education Management

•

and Policy

Director of the Centre for Higher Education

•

Management and Policy

University of Southampton, United Kingdom

•

Management of higher

education; e.g. strategic planning, quality and evaluation, human resource management, resource allocation models

(11)

Sub-group 3:

Management of the academic affairs, Faculty of Agriculture and Forestry, Faculty of Arts, Language Centre and Faculty of Theology

Patrick Dillon

Emeritus Professor of Educational Sciences

•

School of Education and Lifelong Learning

•

University of Exeter, United Kingdom

•

Education, e-learning and multimedia in education, design education, cultural ecology, heritage education and environmental education, docent and visiting professor (University of Joensuu, Finland)

Elisabeth Dumoulin, chair of the sub-group 3 Professor of Food Sciences

•

AgroParisTech

•

Paris Institute for Life, Food and

•

Environmental Sciences, France

Food Science and technology, exchanges of students, international relations, teaching methods

Theo P.W.M. van der Krogt

Emeritus Associate-Professor of Public

•

Management

EAPAA Secretary-General

•

European Association for Public

•

Administration Accreditation University of Twente, Netherlands

•

Public administration, public management, higher education organization, higher education evaluation

Krista Varantola

Professor of Translation and Interpretation

•

of English

Rector of University of Tampere

•

Faculty of Humanities

•

University of Tampere, Finland

•

Chairperson of the Finnish Council of University Rectors

(12)

(13)

1 Introduction

(14)

(15)

Introduction

1.1 Evaluation of education 2007–2008

The University of Helsinki is Finland’s oldest, largest and most diverse institution of higher education conducting research and providing education based on research.

The University of Helsinki consists of 11 Faculties, representing all academic disciplines with the exception of technology and business, and it operates on four campuses. The University community comprises 38,800 degree students (of whom 22,500 are FTEs) and 7,700 staff. The annual intake of new students is over 4,000; only one-ﬁ fth of the applicants pass the demanding entrance exams. The University offers instruction in some 300 subjects or degree programmes. The University of Helsinki is a member of the League of European Research Universities (LERU). LERU was founded in 2002 as an association of twelve research-intensive universities sharing the values of high-quality teaching within an environment of internationally competitive research. Currently, LERU includes 20 European universities.

The strategic aim of the University of Helsinki is to reinforce its position among leading European universities both in research and teaching. To achieve this aim, the University regularly carries out international evaluations of its research and education. The evaluations are a part of the University’s quality assurance system.

The previous evaluation of education was conducted in 2001−2002; this evaluation focused on all the ﬁ elds of education represented at the University, language and communication studies, and subject teacher education.

1.2 Aims and focus of evaluation

The present evaluation of education 2007−2008 focused on the management and leadership of education on various levels, including the University as a whole, Faculties, departments and the Language Centre.

The management of education at the University refers to those academic leadership and administrative management practices which faculties and departments apply in the planning and implementation of education leading to the Bachelor’s and Master’s degrees. The all-embracing goal is to ensure that teaching is carried out in accordance with constructive alignment and that students have the opportunity to complete an academic degree of a high quality. The purpose of the management and the leadership of education is to support the attainment of objectives in the development of teaching and teaching methods, and in the improvement of the quality of learning. The management of education enhances the student’s profound mastery of and expertise in his or her ﬁ eld.

(16)

The evaluation did not include doctoral studies, which will be reviewed in connection with the international evaluation of research scheduled for 2011. The aim of the evaluation was to enhance the management and leadership of education by evaluating its present state from a critical perspective, recognising strengths and areas in need of development, and by receiving international feedback on the quality of operations.¹

1.3 Organisation

The University Senate has made a decision on evaluations to be conducted at regular intervals as part of the implementation of the University’s strategic plan.

For the 2007–2008 evaluation project, on 23 January 2007 the Rector of the University of Helsinki appointed a steering group consisting of representatives of the various campuses and expert groups. The members of the steering group were as follows:

Professor Hannele Niemi, Vice-Rector in charge of academic affairs (chair)

•

Ms Johanna Ahola, student representative (Faculty of Arts)

•

Dr Nina Katajavuori, Senior Lecturer in University Pedagogy (Faculty of

•

Pharmacy, until April 2008)

Professor Sari Lindblom-Ylänne, Head of the Centre for Research and

•

Development of Higher Education (Faculty of Behavioural Sciences) Professor Arto Mustajoki, Head of the Department of Slavonic and Baltic

•

Languages and Literatures (Faculty of Arts)

Professor Jukka Paakki, Dean of the Faculty of Science

•

Ms Päivi Pakkanen, Director of Academic Affairs, Academic Affairs Unit

•

Dr Mirja Ruohoniemi, Senior Lecturer in University Pedagogy (Faculty of

•

Veterinary Medicine, since May 2008)

Ms Minna Frimodig, Coordinator, Academic Affairs Unit (secretary)

•

Dr Seppo Saari, Senior Advisor in Evaluation, since September 2008,

•

Academic Affairs Unit

The evaluation process was operationally coordinated by the Academic Affairs Unit. The Unit coordinated the evaluation by collecting and producing materials, providing instructions and other support, organising the panel visit, collecting the evaluation results for the ﬁ nal reports, and being responsible for communications.

1 Constructive alignment, or consistency in teaching, is deﬁ ned in the Programme for the Development of Teaching and Studies 2007–2009 as follows: “In order to be consistent, all the elements of teaching should promote learning and competence to help students achieve high-quality, profound understanding. From the point of view of consistency, teaching is based on four important stages:

determination of learning objectives, determination of the subject and content of teaching, determination of assessment methods, and determination of teaching methods. In curriculum design, these four stages must be mutually consistent. When the different stages support each other, teaching has a uniﬁ ed and consistent effect on the learner” (p.22).

(17)

1.4 Evaluation method − enhancement-led evaluation

The evaluation of the management of education was conducted in accordance with the principle of enhancement-led evaluation. The primary purpose of the evaluation was to obtain information that the University and its Faculties and departments can use in the development of their operations. At its best, the evaluation was a shared learning process, which enhanced a common understanding of the target of evaluation. In accordance with the principles of learning organisations, the evaluation was drawn on previously acquired evaluation data. Enhancement-led evaluation is generally not based on ready-made standards or predetermined criteria.

The emphasis of the evaluation project was on providing opportunities for the academic community to participate in and affect the planning of the evaluation, its methods and aims as well as its impacts. The general aims and the target of the evaluation were drafted by the University’s Academic Affairs Committee, the members of which include teachers, students and experts from various ﬁ elds represented at the University. The implementation and the theme of the evaluation were debated and commented on by the University’s various cooperation networks, such as the meetings between vice-deans and the meetings between heads of academic affairs. Furthermore, information and discussion meetings open to all members of the academic community were organised. The University community participated in the evaluation through self-evaluations and interviews by the external panel of experts.

1.5 Consequences of the evaluation

The areas in need of development that emerged from this evaluation will receive funding reserved for the development of teaching. The University will possibly award some performance-based funding to the Faculties based on the strengths and good practices identiﬁ ed by the evaluation panel. Moreover, the results of the evaluation and the feedback obtained will be exploited in the drafting of new strategic documents, such as the University of Helsinki Strategic Plan and the Programme for the Development of Teaching and Studies for the period 2010−2012. Finally, use will be made of the evaluation results in the planning of support services for the management and leadership of education, such as in- house training.

(18)

(19)

2 Evaluation process

(20)

1 Background material for the panel 1. Collection of the self-evaluation reports 2. Guidelines for the evaluation panel

3. Education and degree system of the University of Helsinki 4. Strategic Plan of the University of Helsinki 2007-2009

5. Programme for the development of teaching and studies 2007-2009 6. Teaching Evaluation Matrix

7. Final report of the evaluation of the quality of education and the degree programmes in the University of Helsinki 2001-2002

8. Facts 2007. Statistics about the Faculty funding (budget, external, own assets), staff (teaching staff, administration), students (student/teacher ratio, and degrees (BA, MA, PhD).

(Completed version 3.11.08)

9. International mobility 2007 (Completed version 30.10.08) 10. Composition of curricula (Completed version 3.11.08)

11. Statistics about pedagogical training. Centre for Research and Development of Higher Education (YTY).

12. Postareff, L., Lindblom-Ylänne,S., Nevgi, A. 2007. The effect of pedagogical training on teaching in higher education. Teaching and Teacher Education 23 (2007) 557−571. www.elsevier.com/locate/tate.

13. Some examples of the self-evaluation reports at the department level: Department of Chemistry, the Christina Institute, Department of Economics and Management, Department of Economics, Department of Forest Ecology, Department of German, the Haartman Institute, Department of History, Department of Mathematics and Statistics, Department of Practical Theology, Department of Private Law, Department of Psychology, Department of Sociology and the Faculty target programmes and Action plans for the development of teaching and studies.

14. Faculty target programmes and Action plans for the development and teaching and studies in Finnish. (For the Finnish Panel members)

(21)

Evaluation process

2.1 Self-evaluation

The starting point of the enhancement-led evaluation was the self-evaluation conducted by University units. In the self-evaluation, the Faculties and departments described and discussed the strengths and challenges of the management and leadership of their education. The purpose of the identiﬁ cation of strengths and areas in need of development was to aid the units and their leadership to steer the development of teaching and monitor the effects of the measures that were taken.

The self-evaluation was based on unit-speciﬁ c descriptions of the management and leadership of education and on a questionnaire on self-evaluation that the Faculties and departments completed in accordance with instructions (see Appendix 1, page 471). The self-evaluation reports by the Faculties were written on the basis of departmental self-evaluations and reﬂ ections within each Faculty. Based on these preparations, each Faculty formed a perspective on its evaluation and determined the strengths and areas in need of development in its management and leadership of education.

2.2 External evaluation

The external evaluation complemented the perspective formed on the basis of the self-evaluations of the management of education. During the site visits to the University (3−7 November 2008), the evaluation panel had the opportunity to form their opinions based on the self-evaluation reports, other background material¹ and interviews. The site visits were planned in cooperation between the Academic Affairs Unit and the evaluation panel in their preparatory meeting on 7 October 2008. The site visit included a number of interviews with staff involved in management and leadership, teachers and students. The panel was divided into three sub-groups having their sub-chairs.

The programme was organised in such a manner that at the end of each day, the panellists were able to discuss and write down their observations. Each sub- group had a secretary to transcribe all the discussions. The services of the Academic Affairs Unit (e.g., information, facilities, extra materials or revisions of the site visit programme) were available to the panel members throughout the visits to assist the panel in carrying out their expert assignment. During and after the site visit the writing and editing of the report were done on WIKI sites accessible for the Panel.

(22)

2.2.1 The role of the evaluation panel

In the appointment of the external evaluation panel, consideration was given to an equal representation of various disciplines and versatile expertise in the management and development of education. The panel also included two Finnish academics and a student representative.

The panel was expected to:

Familiarise themselves with the assignment and the evaluation task with

•

the help of the background material provided by the Evaluation Ofﬁ ce in the Academic Affairs Unit.

Familiarise themselves with the University of Helsinki with the help of the

•

background material provided by the Evaluation Ofﬁ ce (Academic Affairs Unit).

Study the University’s self-evaluation materials (University-level, Faculty-

•

speciﬁ c and the Language Centre and Swedish School of Social Sciences self-evaluation reports).

With the help of the self-evaluation materials, deﬁ ne the issues that will

•

be considered during the site visits.

Organise the internal division of labour of the panel during the visit and

•

make speciﬁ cations to the programme of the visit.

Make a site visit to the University of Helsinki.

•

Form an opinion on the basis of the self-evaluation materials and the site

•

visit of the quality of the management and leadership of education at the University as a whole and in its Faculties, departments and the Language Centre.

Give recommendations for the improvement of the quality of the

•

management and leadership of education and provide written feedback on the evaluation. The recommendations and the feedback are to be submitted separately to each unit under evaluation.

Participate in the closing seminar of the evaluation visit, where the

•

Faculties and departments will have an opportunity to obtain instant feedback from the panellists and hear the most salient results of the evaluation.

Each panel member was expected to participate as an active and equal member in the panel’s work. The panel was requested to be objective and fair towards all units under evaluation. The special duty of the chair of the evaluation panel was to act as the chair during the panel’s site visit, in the panel meetings and in the writing process as well after the site visit. The chair was expected to promote a good collegial spirit and be responsible for the evaluation assignment and for the equal treatment of the units under evaluation. Three members from among the members were elected to the sub-chairs for the sub-groups.

(23)

2.2.2 Preparations for the site visits

The panel’s work began with studying the self-evaluation reports and the background materials. During this fi rst stage, each panellist made his or her own preliminary observations on the basis of these materials and noted down both general and unit-specifi c questions. In the preparatory meeting of the panel, the implementation of the site visit and the division of labour were discussed. The panellist’s preliminary questions were pulled together, and they served as a basis for the issues to be dealt with during the visit and the relevant interviews. The questions were also grouped in accordance with the panellists’ specialty areas. In the preparatory meetings it was discussed that the questions to be presented in the interviews should be in line with the self-evaluation questions, and specifi cations to the questions can be made during the visit. During the site visits, the Chair of the panel, the sub-chairs and other panellists made sure that essential and previously selected viewpoints were considered.

The various aspects of the management and leadership of education that were discussed in the self-evaluation reports were considered from the following points of view:

The problems and development needs of the management and

•

leadership of education

Sections that identiﬁ ed a speciﬁ c area to be an area of strength, but no

•

detailed reasons were given for this

Sections in the self-evaluation reports that left the responsibilities and

•

processes of the management and leadership of education unclear

2.2.3 Evaluation feedback

The evaluation panel was expected to form an opinion of the management and leadership of education in the Faculties and make evaluative conclusions, as well as to pinpoint strengths and weaknesses in the management of education. The evaluative conclusions were made against the strategic objectives of the University of Helsinki.

In its Teaching Evaluation Matrix, the University of Helsinki presents in a concrete manner the objectives of its Strategic Plan and its Programme for the Development of Teaching and Studies (see Appendix 4, page 515). The matrix is used in the University’s internal quality assurance processes, for example, to support the development of teaching in the Faculties and departments, in continuous self- evaluations and in the selection of the University’s Centres of Excellence in teaching.

(24)

The introduction of the Teaching Evaluation Matrix crystallises the strategic objectives related to the quality of teaching at the University of Helsinki, the University’s enduring values. These enduring strategic values include the following:

Research-based teaching

•

Focus on learning (the principle of student-centredness)

•

Goal-oriented long-term development of teaching

•

Esteem and support for the teaching profession

•

The members of the evaluation panel were requested to consider these strategic objectives throughout the evaluation: how do these strategic aims permeate the management and leadership of education in the light of each of the self-evaluation questions.

The panel’s evaluation was based on the consensus between the panel members. The panel was asked to point out the strengths and good practices of the units under evaluation and to give recommendations for improvement in accordance with the template provided. The template was provided to the panellists in electronic form.

In addition to their written feedback, the panellists were provided instant feedback about their observations and the most salient results of the evaluation at the closing seminar of their visit. The panel discussed the issues that were raised in the seminar.

The panellists shared the responsibility for the provision of feedback. The division of labour was organised in three sub-groups, and the preliminary results were presented by the sub-group members. During the panel visit, time was reserved at the end of each day for a panel meeting and for noting down feedback.

The feedback was based on the self-evaluation reports and on information gathered and observations made during the evaluation visit about the management and leadership of education. The following principles² were observed in the production of the written feedback:

Providing evidence and documentation 1.

Maintaining a connection between the evaluation and the evaluation 2.

materials

Writing in the active voice 3.

Writing on a concrete level 4.

2 1. Providing evidence and documentation. The text should mention the source of a description of a practice or evaluation: a self-evaluation report, discussions during the site visit/interviews, the panel’s own recommendations.

2. Maintaining a connection between the evaluation and the evaluation materials. The feedback should make references to the self-evaluation process and indicate areas in need of development or present development ideas. The evaluation panel is also expected to draw its own conclusions.

3. Writing in the active voice. The feedback should be written in the active voice, e.g., meaning that the panel should express exactly who should improve their operations.

4. Writing on a concrete level. The panel should give concrete examples and express its ideas in speciﬁ c terms.

(25)

2.3 Communications, discussions and analysis

The panel interviewed over 400 representatives of the University. The time schedule with each group was restricted, most often for 40 minutes. It was clear that such a short time affects the character of the discussions and puts pressure on the interviewers as well as the interviewees. It is good to be aware that the larger picture of the process and entity under discussion can be seen only by the panellists. This does not, however, mean that the object was unclear or reached at random by the panel. On the contrary, a combination of the self-evaluation report, the questions prepared beforehand and other informative material was the basis of the discussion steered by the panel, and these contributed to building a broad and deep understanding about the management and leadership on different levels at the University.

2.3.1 Some observations by the Faculties and by the steering group

The feedback from the Faculties was mainly positive, covering the process and evaluation results as well. Self evaluation was seen as communal learning process and very useful as itself. Faculties had many crucial discussions about the topic itself and about its signiﬁ cance for the development of studies and teaching.

The deans in charge of academic affairs shared their experiences about the usefulness of the evaluation after checking the draft of the report. The comments by the deans were as follows:

One actual question is now how to change the strategy in the light of the

•

evaluation results.

The recommendations will help to rebuild the structures. The evaluation

•

was felicitous, although some of the recommendations came as a surprise.

The evaluation conﬁ rmed the direction in which to go. The description in

•

itself was very accurate.

The tone and style of the writing was empathic and constructive.

•

The recommendations will be taken seriously, and they came at the right

• time.

No additional remarks to add to the report.

•

A fruitful way to execute the evaluation; the self-evaluation was

•

experienced in itself as very useful.

The report is critical enough, and the timing is punctual.

•

The recommendations were considered important and very useful.

•

The evaluation itself was felt to be positive.

•

The ﬁ rst reaction was disappointment, but the next day the observations

•

and recommendations were understood to be an excellent tool for development.

(26)

The Faculty coordinators of the evaluation had an essential role in the cooperation with the Academic Affairs Unit and the Faculties. The feedback by the Faculty coordinators on the evaluation was received both in the ﬁ nal meeting and are verbatim as follows:

The guidelines were always clear.

•

The time schedule to react and respond was realistic, giving time to

•

prepare well in advance.

The WIKI was found to be an excellent tool where the entire common

•

database was available at the same website.

The coordinators’ network gave peer support throughout the process.

•

According to the teachers’ feedback, the process itself was experienced as light enough:

The atmosphere in the interviews was positive.

•

The evaluation contributed to good discussion throughout the Faculty.

•

The panel came across as experts.

•

The time schedule was too tight in the interviews.

•

The steering group also made some observations about the evaluation in general and the aims and process of the evaluation. As a whole the evaluation process was seen as a good learning process for the university community, and it produced very relevant and useable information for development work.

The object of the evaluation

The object of the evaluation – the management and leadership of education – in itself was a quite new and challenging theme for the university. The steering group devoted much time to discussing and deﬁ ning the meaning of the topic. This explication and the deﬁ nition of the subject were very useful when establishing the instructions to the Faculties, which substantially helped the self-evaluation process.

Organising the evaluation

Early on it was decided that this evaluation process should be lighter and more limited in scope than the previous evaluation. The focus was limited, and much evaluation material was produced centrally. The support offered by the Academic Affairs Unit worked well. The organisation of all the practical matters by the Academic Affairs Unit moderated the burden on the Faculties and departments.

The time schedule was carefully planned, and it remained attainable. Also, all the instructions were very well designed and feasible. One of the most important decisions was to offer the Faculties consultation as well as ﬁ nancial help for their self-evaluations. The cooperation between the Academic Affairs Unit and the Faculty coordinators was also remarkable. This evaluation process also cultivated the know-how of how to carry out an evaluation process at the University, and it is

(27)

important that these kinds of professional methods for evaluation procedures will also be carried out in the future at the University.

External Evaluation

The steering group was very impressed with the positive and quick answers of the invited evaluation experts. The panel was considered very committed to the project, and the panel’s expertise was seen as high quality. The panel was chaired well and was well organised. The good atmosphere in the discussions was highly commended by the Faculties. It is easy to rely on the panellists’ expertise. The interviewers were also well prepared, and the strategy in the division of labour between the panellists worked well.

The support offered by the Academic Affairs Unit to the panel was also remarkable.

The panel was supported with clear and speciﬁ c guidelines. The introduction to the evaluation task (including e.g. the panel’s “homework” and interview exercises) in both pre-meetings (Frankfurt 17.10.08, Helsinki 2.11.08) was seen as essential to the success of the panel’s work. Much planning and preliminary work was also done in several meetings during the spring and autumn of 2008 with the chair of the panel, Rector Eva Åkesson. The role of Eva Åkesson as a chairperson and her exceptional commitment to her duty were praised as being extremely important to the success of the entire external evaluation process.

Closing Seminar

The concept of the closing seminar was new in the University’s evaluation process.

The idea was to offer the University community the opportunity to obtain instant feedback from the panellists and to hear the preliminary results of the evaluation.

For the panellists it was a very challenging situation; they had to compress their main observations into a very short time and had to comment on issues that were not yet thoroughly digested. In any case the closing seminar was seen as a good ending to the evaluation week, and it raised expectations for and interest in the ﬁ nal report. The closing seminar also made it possible to see how all the panellists worked together and in this way increased trust in the panel’s operation. More time would have been needed for the discussion in the seminar.

Final Report

The evaluation feedback by the panellists covers the topic broadly enough and is detailed and concrete. The value added by it is quite obvious. The report is analytic, keen and gives a broad view of the topic under evaluation. The report introduces relevant recommendations. Special surveys made beforehand may have contributed material necessary or useful to the interviews.

For the future

In their ﬁ nal meeting, the Steering Group thoroughly discussed the evaluation method used. It was known beforehand that by this method it is not possible to get a detailed picture about leadership practices at the grass roots. For that

(28)

the panellists should spend a minimum of one day in each department. In the future the university will have a smaller number of departments, but nevertheless, grasping everyday life practices at the department level may be an attainable goal using this kind of evaluation method.

For planning and conducting further assessments, the Steering Group has expressed some ideas to be considered:

It may be reasonable to change the actual focus of the assessment every 1.

time.

The University of Helsinki and its Faculties and departments might 2.

beneﬁ t more from an assessment if it is conducted by using some kind of benchmarking procedure. Possible candidates for comparison could be other LERU universities.

The assessment could include a survey on teachers’ and students’

3.

opinions, attitudes and wishes. This material might serve as a starting point for the panel to work with or instead of a self-evaluation.

To be more comparable with the research assessment exercise, one should 4.

think about the possibility to use rewards on the basis of the assessment.

(29)

2.3.2 The panellists’ observations of the evaluation process

The Evaluation Panel made a couple of remarks about the evaluation process.

First of all, eliciting such an evaluation was a good and brave endeavour of the University, which deserves due recognition. However, it was also extremely time consuming and very expensive. The instrument should be used sparsely and wisely.

Furthermore, the evaluation process was well designed, and the support for the evaluation panel was ﬁ rst class. The pre-meeting of the Evaluation Panel was very useful for becoming acquainted with each other and for organising the work before the site visit. It was an intensive learning experience for all involved. The Evaluation Panel members were all very grateful for being invited to the University of Helsinki: we all learned a lot during the course of action. All the panellists made a great effort to ensure the success of the evaluation; during the site visit, we had an intensive week, interviewing during the days and writing during the nights.

Secondly, this evaluation process was directed at the management and leadership of the educational process in the University of Helsinki. Although this is a clearly identiﬁ able aspect of university life, the Evaluation Panel lacked information on the context in which this educational process takes place, such as the research (strategies), some (numerical and content) details about majors, minors and independent Master’s programmes and their students, and information on the human resources policy/policies. Another element that was missing was information on the actual teaching performance of the Faculties. Information on the plans for the reduction of the number of departments was also missing. The above information would have been needed because management, and especially leadership, cannot be evaluated apart from the content and its context. To focus on the process and not also on the output and outcome is a drawback.

A third point is related to the development process of the University management and leadership. This evaluation process was not the ﬁ rst, but information on what the results of former evaluations were and what has been done with the results of these was only partially available. The same observation can be made for the Faculty level. The meetings the panel had with the different ‘stakeholders’ at the University and Faculty level were very helpful. The sub-panels of 4 persons were very workable. However, to interview 10 persons in 40 minutes was not an optimal situation; smaller groups would be preferred.

A bit more time for the management level was available in the ‘second’ interview, to raise a couple of issues that resulted from the other meetings. This aspect was partly taken care of in an extra meeting with all deans/vice-deans in charge of education after the Faculty visits were completed. Another aspect, taking into account the procedure of the evaluation as it was set up (short interviews and no separate contacts with all the Departments), it was not evident that separate comments on the Departments were expected in this exercise.

(30)

All the panellists had academic integrity and gave their honest view. The panel offered good advice as critical friends and really hopes it will be useful to the University so that the aim of the evaluation, to enhance the management and leadership of education, can be fulﬁ lled. However, there is one severe drawback.

In the evaluation, the panel has been given the mission to ﬁ nd both strengths and good practices. It was easy to ﬁ nd numerous examples of good practice.

Nevertheless, the panel must emphasise strongly that there are most certainly other strengths and good practices in the Faculties and Departments that the panellists did not discover during the short visit to the University.

Finally, the members of the Evaluation Panel want to express their gratitude to Minna Frimodig and Seppo Saari for their fantastic service and support.

(31)

2.3.3 Realisation of the aims of the evaluation

The aim of the evaluation was to enhance the management and leadership of education by evaluating its present state from a critical perspective, recognising strengths and areas in need of development, and receiving feedback on the quality of operations. The aims of the evaluation were set beforehand by the University itself. The emphasis of the evaluation project was on providing opportunities for the academic community to participate in and affect the planning of the evaluation, its methods and aims as well as its impacts. The self-evaluation phase was designed to be a learning process. According to the feedback received from the Faculties, this aim was well reached. The ﬁ nancial and consultation support provided by the Academic Affairs Unit were seen as helpful in fulﬁ lling this aim.

The external evaluation consisted of, in addition to all the documents provided, a large sample of representatives from all levels of the institution, who introduced understanding of the management and leadership in education. The phenomenon itself became better understood in the University, and the process increased the common understanding about the importance of management and leadership procedures, responsibilities, duties and organs such as committees, leaders, managers and the staff involved in these procedures.

The Panel was expected to identify the problems and development needs of the management and leadership of education, and to identify areas of strength and sections in the self-evaluation reports that left the responsibilities and processes of the management and leadership of education unclear. The Panel was requested to consider the strategic objectives throughout the evaluation and how the strategic aims permeate the management and leadership of education in the light of each of the self-evaluation questions. The Panel provided the evaluation observations based on evidence connected to the evaluation documents and discussions. The active voice and concrete recommendations that specify the actors responsible are presented in the report.

The Panel prepared the questions in order to raise relevant dialogue and discussion, and to avoid the experience of grilling the interviewees. The strict time schedule was implemented not by the Panel but by the University’s decision to schedule one week for the site visit. That had some negative consequences, but the optimum ratio between the time allotted for the Panel and all the interviews and the costs was an optimised combination of many factors. The discussions in the interviews are always a checkpoint, clarifying some aspects, trying to pinpoint insubstantialities and looking for the essential aspects. The panel does not build a discussion which attempts to present an entity to the audience in the interviews, but is piecing together a puzzle and trying to present a complete picture in the report after all the interviews. The better the self-evaluations and other documents are, the better analysis the panel can provide in the report. All in all, the tone and the style of the report are correct. The enhancement-led approach in evaluation was

(32)

implemented in an ideal way. The approach was present in the interviews and can be recognised in the report text as well.

One of the aims of the evaluation was to document the ﬁ ndings on the Faculty and departmental levels in addition to the University level. In practice, this was not possible, as the Chair of the Panel states in this report. The Panel met only some members that could be recognised as representing the departments. For this reason, the departmental level is not covered in the report. The departmental level feedback received by some of the Faculties can be considered added value. It also proves that the Panel acted moderately and was realistic about what is possible to observe and conclude. The evaluative conclusions were expected to be in line with the strategic objectives of the University of Helsinki. Comments on these are presented in many cases in the report. The Panel’s observations are based on an analytical view of the whole data. The ﬁ ndings are also reasoned and credible.

On the university level, the report includes 16 recommendations, and common areas in need of development 27 recommendations. On the Faculty level 67 recommendations are presented. The Panel recognised 20 University level and 104 Faculty level strengths and good practices. The report will best serve the University after the outcomes are carefully analysed and implemented step by step. Each recommendation will be prioritised according to major versus minor and urgent versus long-term recommendations. The evaluation can be implemented by taking the outcomes of the evaluation into account and integrating them into all the documents and procedures to which they are relevant. The Academic Affairs Unit as its developmental role will continue to follow the implementation of the evaluation also in the long term.

(33)

3 Management of education at the

University level

(34)

3.1 Self-evaluation report at the

University level

(35)

A Introduction

The management of education at the University refers to those academic leadership and administrative management practices which faculties and departments apply in the planning and implementation of education leading to the Bachelor’s and Master’s degrees. The all-embracing goal is to ensure that teaching is carried out in accordance with constructive alignment and that students have the opportunity to complete an academic degree of a high quality. In order to be consistent, all the elements of teaching should promote learning and competence to help students achieve high-quality, profound understanding. From the point of view of constructive alignment, teaching is based on four important stages: determination of learning objectives, determination of the subject and content of teaching, determination of assessment methods, and determination of teaching methods.

The purpose of the management of education is to support the attainment of objectives in the development of teaching and teaching methods and in the improvement of the quality of learning. The management of education enhances the student’s profound mastery of and expertise in his or her ﬁ eld.

To a great extent, the management of education at the University means collective responsibility for the development of teaching and the enhancement of learning.

The expertise of the teaching staff, students and various academic administrators in the faculties and departments is not only called upon in collegial decision- making, but also in daily academic activities and their development. The overall responsibility for the development and objectives of fi eld-specifi c education rests with academic leaders, i.e., deans, vice-deans and heads of department. The special characteristics of the University’s general management system are clearly visible in the daily management of education, which includes both the management of operations and the leadership of people. The University’s operations manual defi nes leadership and management as practical actions that enable the work community and its members to be able to fulfi l their core duties. Through his or her activity, the manager promotes the achievement of programme goals and provides the prerequisites for the activities of the work community and its members.

The self-evaluation of the management of education at the University level was conducted as a last, summarising stage after the faculty-level self-evaluations. Once the faculties had submitted their self-evaluation reports, a three-hour workshop was arranged on 18 April 2008 to draw overall conclusions of the strengths, weaknesses and areas in need of development of the management of education at the University level. An invitation to this workshop was sent to the rector and the vice-rectors, members of the University Senate, deans and vice-deans in charge of academic affairs, as well as heads of academic affairs, planning ofﬁ cers and experts from the Administration Ofﬁ ce. The number of invitees was 55, of whom 33 participated in the workshop. The participants discussed four themes from section

(36)

B of the faculty self-evaluation reports under the leadership of a consultant, and on the basis of this discussion, a planning ofﬁ cer from the Department of Strategic Planning and Development and a project employee compiled this University-level self-evaluation report. The descriptions in this report draw from the operations manuals of the University of Helsinki and its faculties, the duties and responsibilities deﬁ ned in the Universities Act and the University’s internal regulations, and from materials and texts available on the University’s website and intranet.

B A description of the management of education at the university

level and its pivotal strengths, weaknesses and areas in need of development

Setting strategic objectives and translating them into concrete measures

The University’s operations management process

At the University of Helsinki, setting strategic objectives and translating them into concrete measures takes place through its operations management process.

Operations management ensures that the University operates in accordance with its strategic plan, and it applies to all University activities and resources. The University’s operations management and quality assurance systems are closely related.

The salient stages of operations management include:

Devising a strategy in order to determine objectives and areas in need of

•

development

Formulating action and target programmes to determine concrete

•

measures, responsibilities and available resources

Implementing the strategy and following up on its success in target

•

negotiations and target and performance seminars

(37)

Operations management can be divided into the following primary processes and sub-processes:

Operations management process chart -primary and subprocesses-

OPERATIONS MANAGEMENT

Strategy and development programmes

Dissemination of the strategy Implementation

of the strategy Follow-up and evaluation of the strategy

- writing/updating of the strategy and development programmes

- agreement on the objectives of the three-year period - principles of the allocation of

funding and the allocation model

- drafting of unit-speciﬁc target programmes

- agreement on annual targets - distribution of funding - projects and unforeseeable needs - service agreements

- unit-speciﬁc annual follow-up report

- project reporting

- follow-up of the strategy and development programmes - closing of the accounts - annual report - expertise databases - annual statistics

The University administers its activities according to a three-year strategic plan.

The strategic plan deﬁ nes, on the basis of the University’s values, duties and future prospects, strategic goals for the University’s core duties (i.e., research, teaching and societal interaction relevant to research and teaching) and for the development of its operational preconditions. It also deﬁ nes key areas of development which need to be addressed to achieve the set strategic goals.

The University implements its strategy through various policy programmes which present the concrete measures to be taken, objectives, responsibilities and resources. From the point of view of the management of education, the most important of these policy programmes is the Programme for the Development of Teaching and Studies. Since 1992, this programme has encompassed the objectives and focus areas of the development of teaching and learning. The programme is devised in the form of a team effort including the entire university community, i.e., students, teachers and academic administration. A signifi cant part of the process is a one-day seminar, which brings together the expertise of a large and versatile group of participants to defi ne focus areas and priorities. After the seminar, the draft programme is circulated widely for comments before it is fi nalised by the Academic Affairs Committee and the University Senate. The Programme for the Development of Teaching and Studies plays a signifi cant role in the allocation of three-year project funding for the development of teaching, which is distributed on the basis of applications from the faculties.

(38)

The faculties, independent institutes and the Administration Ofﬁ ce draw up their own target programmes on the basis of the University’s Strategic Plan and policy programmes.

The organisation of the target programme’s content is identical for all faculties.

The independent institutes and the Administration Offi ce devise their own target programmes and apply the set organisation of the programme as appropriate. The target programmes defi ne concrete objectives which need monitoring and set out a general plan for their implementation. The target programmes also take into account the performance agreement between the University and the Ministry of Education. In addition to listing unit-specifi c aims, the target programmes include commonly agreed objectives concerning the entire University which have been defi ned in meetings between deans, directors of independent institutes and the rector. Unit-specifi c targets are agreed upon in the unit’s performance negotiations.

Tables containing information on the resources of the strategy period’s ﬁ rst year, targets and key ﬁ gures as well as a three-year service agreement are enclosed with the target programmes.

Most faculties draft action plans on the basis of their target and policy programmes, such as three-year action plans for the development of teaching. Among other things, the action plans prioritise the development challenges identiﬁ ed in the target programmes and distribute responsibilities for the implementation and follow-up of measures to be taken. Guidelines for the development of teaching are prepared collegially, drawing from the work of joint development seminars and various committees and networks. Students actively participate in the preparation and decision-making processes. The faculty-level target programmes and action plans are translated into concrete measures through departmental strategic plans and guidelines for the development of teaching.

The implementation of the University’s Strategic Plan and policy programmes, and the target programmes of the faculties, the independent institutes and the Administration Offi ce, are evaluated in connection with performance negotiations with the rector and in target and performance seminars related to operations management. The faculties, independent institutes and the Administration Offi ce report annually on their activities especially with regard to the implementation of their target programmes and provide analyses of the reasons for and consequences of their performance. The rector provides written feedback on these reports and focuses in particular on the implementation of target programmes. In connection with the reports and the rector’s feedback, target and performance seminars are arranged for the deans to discuss, on the basis of reports and key fi gures, the previous year’s activities and the implementation of the relevant policy and target programmes, and to anticipate the success of the current year’s objectives.

(39)

Strengths, weaknesses and areas in need of development

The various methods and instruments used in connection with the University’s operations management process are a great beneﬁ t, for, on the one hand, they facilitate the devising of the University’s Strategic Plan, and on the other hand, they provide useful tools for planning and management after the strategy has been written. An example is the dialogue questionnaire used in the devising of the Strategic Plan 2007–2009, which involved gathering evaluations of the implementation of the previous strategy through an online discussion. The instruments related to the planning and management of operations include the University, faculty and departmental operations manuals and process descriptions, and the University’s Teaching Quality Evaluation Matrix, which facilitate the orientation of new employees and enable also older employees to perceive concretely the University’s operations. The multitude of methods and instruments also has its downside: due to lack of time and energy, they all cannot be used.

The short strategy period is the most obvious weakness of the operations management process. The present three-year period is too short, for the planning of the next strategy period is launched at the same time that the implementation of the current strategy period is beginning. Not much information has accumulated about the implementation of the previous strategy period, so the planning of the following period begins from scratch, when, in fact, the planning should be based on the experiences and systematic analyses of the previous strategy period. Implementing changes and detecting the effects of reforms in educational structures are slow processes: completing the basic degree takes ﬁ ve years, and in order to witness all the effects of a reform we should wait for the graduation of the class of students affected by the changes. If reforms are carried out too quickly, there is not enough time to analyse their effects. A clear development challenge is to devise separate strategies for short and long terms: although the demand for three-year strategy periods comes from outside the University, i.e., from the Ministry of Education, it does not prevent the University from drawing up its own strategic guidelines for ﬁ ve or ten years.

Further strengths include the fact that the present Strategic Plan has fewer key areas of development than the previous ones; in other words, the University has been able to select its objectives skilfully. However, the fact that the faculties and departments are not always able or do not dare to implement the set priorities to the full is a weakness. Differences between the faculties and departments should be taken into account when devising strategies and programmes of various levels, thus providing them with opportunities to apply on an individual basis the key areas of development deﬁ ned at the University level. The University’s large size is also both a strength and a weakness: the great variety in the type of units and levels of administration generates versatile good practices, but at the same time, presents a great challenge for management at the University level. Also, because of its large size, the University has not succeeded in involving all members of the University

(40)

community in its strategic objectives, and not all members of the University are aware of its salient objectives. The University should continue to clearly highlight the core issues and main objectives, which should be familiar to every member of the University community and to which they all should be committed.

The fact that knowledge obtained in practical operations is not adequately exploited in the drafting and implementation of the strategy is a weakness. Daily work generates a vast amount of experience and observation which are not collected in any form. Experience-based knowledge is thus an unused resource.

Yet another weakness is that problems are not properly identiﬁ ed and announced in the negotiations connected to operations management processes. Numerous issues of great importance are noted down as areas in need of development in strategic plans, but no real action is always taken.

The support provided by numeric and qualitative follow-up data for the management of education

Numeric follow-up data

Numeric data is obtained from a variety of sources. In addition to electronic information systems (such as Oodi and Ilmi), data is collected by various ofﬁ cials by using various questionnaires, for example. Numeric follow-up data is used, among other things, in the ETAPPI study progress checkpoint system, and the most important data are gathered together annually on all levels of administration.

Key performance indicators

The faculties and the independent institutes are responsible for monitoring their activities using key performance indicators and other information, some of which are common across the University. The key performance indicators are specifi ed in target programmes. Key fi gures related to education include the number of applicants/admitted students, student/teacher ratio, the number of completed degrees, degree completion times and the employment of graduates in positions corresponding to their academic qualifi cations. The key performance indicators are examined in target and performance negotiations to assess how well the unit has progressed towards attaining the agreed objectives. Electronic information systems are used in the collection of the relevant fi gures. Resources are examined from the point of view of the implementation of agreed objectives. Key performance indicators are used to describe operations and changes in operations over several years. In addition, defi ciencies in activities and needs for improvement can be detected with the help of performance indicators. At the University level, key performance indicators are collected for reports submitted to the Ministry of Education and Statistics Finland.

Leadership and Management of Education : Evaluation of Education at the University of Helsinki 2007-2008

Leadership and Management of Education

Evaluation of Education

at the University of Helsinki 2007–2008

Seppo Saari & Minna Frimodig (Eds.)

Leadership and Management of Education

Evaluation of Education

at the University of Helsinki 2007–2008

Contents

Foreword

Evaluation panel

Chairperson of the panel

Vice-Chair of the panel

Members of the sub-groups Sub-group 1:

Sub-group 2:

Sub-group 3:

1 Introduction

Introduction

1.1 Evaluation of education 2007–2008

1.2 Aims and focus of evaluation

1.3 Organisation

1.4 Evaluation method − enhancement-led evaluation

1.5 Consequences of the evaluation

2 Evaluation process

Evaluation process

2.1 Self-evaluation

2.2 External evaluation

2.2.1 The role of the evaluation panel

2.2.2 Preparations for the site visits

2.2.3 Evaluation feedback

2.3 Communications, discussions and analysis

2.3.1 Some observations by the Faculties and by the steering group

2.3.2 The panellists’ observations of the evaluation process

2.3.3 Realisation of the aims of the evaluation

3 Management of education at the

University level

3.1 Self-evaluation report at the

University level

A Introduction

B A description of the management of education at the university

level and its pivotal strengths, weaknesses and areas in need of development

Setting strategic objectives and translating them into concrete measures

The University’s operations management process

Strengths, weaknesses and areas in need of development

The support provided by numeric and qualitative follow-up data for the management of education

Numeric follow-up data

Key performance indicators