The Panel was generally impressed with the training programs outlined in the RC documents.
It also noted the confidential Summary report on doctoral students’ and principal investigators’ doctoral training experiences, and some issues of concern in this report.
The Panel notes that there are high proportions of students who are not in doctoral programs (53% biology/environment, 65% agriculture/forestry, 60% veterinary medicine). The frequency of supervision varied notably among areas with biology/
environment/vet students getting frequent supervision but in some areas it was typical to receive supervision either monthly or every 2 months, and a substantial fraction rarely interacting with their supervisors. It is noteworthy that “both students (45%) and PIs (39%) highlighted supervision in the research process – including giving practical help and advice concerning the research topic and research methods, as well as planning the research and reporting on it – to be a supervisor’s most important task” whereas this might be expected as the central role of the supervisor. There were differences between areas, with research supervision being least important in veterinary science and coaching of students being highlighted in biology/environment.
The report notes that “constructive supervision and feedback, working conditions, and sense of belonging to the scholarly community” scored lowest for biology/
environment students, whereas for veterinary science there was a strong sense of belonging. The majority of students were satisfied at least “partly” with their supervision but biology/environment/veterinary science/agriculture tended to score at the lower end when compared to other faculties. Students have quite a high success in finding job placements, although it is noteworthy that a substantial fraction of placements (<50%) do not involve research. This makes it noteworthy that “the students also reported that the studies were less likely to provide the skills needed in work outside of university”
and that “the students also reported a need for extra courses, especially in research methodology, management and project skills, and career planning”.
The Panel took account of these comments in preparing its recommendations. The general recommendations of the Panel to the University are as follows:
1. To improve the quality of doctoral training, more support should be provided for promoting teaching by graduate students. Currently the time allocation to teaching is only 5%. Teaching experience provides an important component of the doctoral experience and improves the breadth of training.
2. The structure of a PhD thesis is currently quite prescriptive in many RCs, in terms of specifying a substantial number of accepted and submitted papers. In some graduate
schools and RCs the requirement amounts even to 5 accepted papers. To create more flexibility, we recommend that this requirement be relaxed. When there is a strong focus on a requested minimum number of publications, student projects are likely to be less risky, and there is a tendency to focus too much on the smallest potential publishable units rather than well integrated papers in high profile journals. In our experience, a single paper with a high impact can be preferable to a series of papers.
3. The Panel recommends that ways are explored to reduce the time students take to complete their PhD training. In some RCs, training can take many years, and this may have a negative effect on employability. We suspect that the adoption of (2) above will assist in reducing completion times.
4. More attention should be paid to developing consistent course components for doctoral students that widens their training. It is clear that many graduates will not find employment as researchers within the Finnish system. To equip graduates for positions outside a research environment, we recommend that courses are developed that allow students to acquire a wider set of skills in areas like communication, biostatistics and so on.
5. Steps should be taken to ensure that all students are aware of multidisciplinary approaches in tackling research questions. In many RCs there is awareness of and emphasis on multi-disciplinary training, but the Panel feels that steps should be taken to ensure that all doctoral students have some exposure to multiple disciplines as part of their normal training. This might be achieved by ensuring that students have placements in more than one laboratory, as is often done overseas.
6. The nature of doctoral training seems to depend too much on the type of funding available to support the project. If funding for doctoral training is attached to a specific project, there is a particular danger that the doctoral training experience will be quite narrow.
The Panel recommends that ways of standardizing training within disciplines across the BIO sector be explored. The Panel also notes the large difference in the frequency of interactions between the student and supervisor across disciplines and wonders if the monthly interactions in the Agricultural sector are sufficient to ensure high quality doctoral training.
7. Ways should be explored to give doctoral students greater ownership of their projects by linking funding to performance. In some institutions students develop and present their potential project ideas to a forum, and funding depends on the grades they receive at this time. In institutions in some countries (unlike in Finland) students are able to directly apply for their own funding, and this also promotes independence and project ownership.
8. No information was provided about drop out/completion rates of PhD students and the Panel recommends that that this type of information be made available through the University. This type of information (along with graduate placements) is critical when assessing the success of doctoral training.
3.2 DEVELOPMENT OF THE RC CONCEPT
After assessing and discussing the written material provided by the RCs, the Panel came up with a number of recommendations to assist in the further development of the RC concept. In making these recommendations, the Panel noted the University’s strategy for 2010–2012. In particular, it noted that the University aims to “establish up-to-date and discipline-related performance indicators that measure the quality and quantity of research activities” to recognize strengths, and that the University “seeks to improve the funding base
for its research activities” as well as being interested in research quality assessment that was to be followed up regularly. It was also noted that UH “will define its most important research infrastructure requirements and technology services and will participate in national and international projects on these issues”. The Strategic document also indicated the University’s interest in taking “its national and global community responsibilities into consideration” with a focus on societal development and interest in “societal significance of science and education”. To meet these strategic goals, the Panel felt that several changes to the RC concept should be considered.
1. The Panel recommends that any funds earmarked by the University for the further development of the RCs should not be allocated based on a ranked list of numerical scores, including the scores from this evaluation round. Instead funding allocations should take account of the comments from the Panel, future potential development of the RCs, and the category as well as size of RCs.
2. The number of categories available for the RCs should be reduced. In particular, category 2 is very closely related to category 1 and these could be collapsed as long as an evaluation is made of the research opportunity and seniority of principal investigators involved in an RC. That way RCs involving promising young researchers can be identified and separated from those based around more senior and established researchers. The Panel also felt that high quality specialist research does not specifically need to be separated from category 1 research. In our evaluations, high quality specialist groups often performed as well as category 1 RCs – for example based on bibliometrics.
3. The evaluation exercise needs to place more emphasis on research adoption and path to impact. Given the University’s increasing interest in innovation for promoting industry-based developments and for strengthening interactions with government agencies, the Panel felt that a greater emphasis on impact outside of scientific publications was warranted. We note that the Academy of Finland currently provides the bulk of support for the RCs, whereas a greater focus on path to impact could assist in diversifying funding sources including access to TEKES funding and some EU schemes.
4. Where PIs participate in more than one group, the component of the PIs funding attributable to that group needs to be clearly defined. This helps in assessing the funding gained by a particular RC.
5. Metrics need to be expressed relative to the number of investigators in a group.
Opportunities that PIs have for undertaking research need to be explicitly spelt out in defining group size. Indeed, no information on: 1) output per full time professor / persons with fixed position 2) output per funding unit (e.g. number of publications (+
IF) per 100,000 or 500,000 euro funding) obviously taking into account the variation in working cost per field (animal experimentation, high-throughput sequencing, expensive culture media for stem cell related work)
6. An opportunity should exist for panels to interview representatives from the RCs. Site visits should be part of the normal assessment procedure in the future rounds, although we appreciate that this was impossible if 27 RCs are being evaluated like in this first round.
7. Mechanisms should be put in place to facilitate steps to broaden the funding base of RCs.
Low funding from EC sources seems to partly reflect researcher concerns about onerous administrative requirements associated with these grants. However these EC funds can be valuable because there are substantial benefits associated with collaboration across the EC. The University should explore ways of reducing this burden for researchers.
8. RCs should provide lists of top 10 papers that are essential in the past and indicate which of those are a solid basis for the future development of the RC. In addition, brief information should be provided about the relevance and impact of each paper for the development of the RC.
9. The University needs to improve its definition of an RC. Tighter definition is required around the size of the RC and expectations. We prefer RCs that involve a merger of disparate areas where there is likely to be a major benefit, leading to new research directions and capturing novel and innovative concepts. Our expectation is that an RC is a dynamic initiative that leads to the development of intellectual capital that can promote high quality PhD training and exciting multidisciplinary research (but this was not addressed in many proposals).
10. The RCs should have defined more clearly the stage of their development (although some did). This helps in the evaluation process.
11. Internal funding available to the RCs needs to be clearly defined. Internal funding was never quantified even though it clearly contributes to (for instance) centre of excellence initiatives.
12. Because the categories available to RCs emphasize different components, the final evaluation of an RC should weight sections of the application differently depending on the category selected.
13. It is inappropriate to provide ratings for societal impact of RCs unless the RCs specifically nominate to be evaluated as part of this category. The Panel appreciates that societal impact should be documented by all RCs but numerical scores should not be assigned unless there is a specific focus on this category.
3.3 DISTRIBUTION OF SCORES
Table 2. Numeric evaluation of RCs in the Panel of Biological, Agricultural and Veterinary Science
RCS (27) QUALITY OF
RESEARCH DOCTORAL
TRAINING SOCIETAL
IMPACT CO-
OPERATION CATEGORY
FITNESS SUM OF
SCORES CATEGORY
ARC 3.0 1.0 2.0 2.0 3.0 11.0 1
BIOSYST 3.0 4.0 4.0 4.0 4.0 19.0 3
CellMolBiol 5.0 5.0 4.0 5.0 5.0 24.0 1
CoE MRG 5.0 4.5 4.0 4.0 5.0 22.5 1
CoE_VIRRES 4.5 4.0 4.5 4.5 4.0 21.5 1
CoE-MiFoSaPLUS 5.0 5.0 4.0 5.0 5.0 24.0 1
EGRU 5.0 4.5 4.5 4.0 5.0 23.0 1
ENIGMA 4.0 2.0 5.0 3.0 3.0 17.0 1
EvoDevo 5.0 5.0 4.0 5.0 4.0 23.0 3
FoodNutri 4.0 3.0 4.0 4.0 4.0 19.0 5
FRESH 3.0 2.7 4.0 2.7 4.0 16.4 3
HelDevBio 5.0 5.0 5.0 5.0 5.0 25.0 1
INBIOS 3.0 3.0 4.0 3.5 3.5 17.0 4
LEGMILK 2.5 3.0 2.0 2.5 3.0 13.0 4
MEMBREC 4.0 4.5 4.5 4.0 4.0 21.0 2
MICRO 4.0 5.0 4.0 4.0 4.0 21.0 1
MUSGEN 4.0 4.0 5.0 4.0 4.0 21.0 4
PEATLANDERS 5.0 3.0 4.0 4.0 4.0 20.0 3
PHABIO 3.0 3.5 3.0 4.0 3.0 16.5 3
PHYTOPATH 4.5 4.0 4.0 5.0 4.0 21.5 1
SB&B 4.0 4.0 4.0 4.0 4.0 20.0 1
SSA 3.0 3.5 4.0 3.0 4.0 17.5 2
SUVALUE 3.5 4.0 4.0 3.0 4.0 18.5 5
VetSci 4.0 4.0 4.0 4.0 4.0 20.0 2
ViiGen 4.5 4.0 4.0 4.0 1.0 17.5 4
VITRI 3.0 4.5 5.0 4.0 2.5 19.0 1
VMPS 5.0 5.0 4.0 5.0 5.0 24.0 1
Average 4.02 3.88 4.02 3.93 3.89 19.7
The table is organized in alphabetical order. The mean of the scores in quality of research is relatively high, 4.02 (panels’ average 3.96). For a comparison of panels, see Table 33. The RCs are 27 altogether.
52
Figure 2. Distributions of the numeric evaluation of the Panel of Biological, Agricultural and Veterinary Sciences
The bars are organised according to the order of the first four evaluation questions. Category fitness was added to the results. Category fitness can be considered to represent the type of performance other than the previous four evaluation questions. Only in three cases, category fitness would change the numeric order of the RCs.
0,0 5,0 10,0 15,0 20,0 25,0 30,0
HelDevBio CellMolBiol CoE-MiFoSaPLUS VMPS EvoDevo EGRU CoE MRG CoE_VIRRES PHYTOPATH MEMBREC MICRO MUSGEN VITRI ViiGen PEATLANDERS SB&B VetSci BIOSYST FoodNutri SUVALUE ENIGMA SSA INBIOS PHABIO FRESH LEGMILK ARC
Panel of Biological, Agricultural and Veterinary Sciences (RCs 27)
Quality of research Doctoral training Societal impact Cooperation Category fitness
Figure 2. Distributions of the numeric evaluation of the Panel of Biological, Agricultural and Veterinary Sciences
The bars are organised according to the order of the first four evaluation questions. Category fitness was added to the results. Category fitness can be considered to represent the type of performance other than the previous four evaluation questions. Only in three cases, category fitness would change the numeric order of the RCs.
3.4 PUBLICATION STATISTICS
The next publication statistics are based on the publications exported from the TUHAT RIS.
Figure 3. Biological, Agricultural and Veterinary Sciences: Number of WoS and A1−A4 publications (TUHAT), number of RCs 27
Figure 4. Biological, Agricultural and Veterinary Sciences: Number of citations, number of RCs 27
0 100 200 300 400 500 600 700 800
No. of publications with WoS id No. of A1-A4 publications
1.4 Publication statistics
The next publication statistics are based on the publications exported from the TUHAT RIS.
Figure 2. Biological, Agricultural and Veterinary Sciences: Number of WoS and A1−A4
publications (TUHAT), number of RCs 27
Figure 3. Biological, Agricultural and Veterinary Sciences: Number of citations, number of RCs 27
The RCs’ order in the figures: the first figure (on the left) shows the ratio of WoS publications to A1−A4 (TUHAT). The figure on the right can be predicted by the first figure. If the ratio of WoS to A1-A4 is high, the total citations (TCS) of RC can be expected to be high as well. The correlation between the indicators (WoS publications/(A1−A4) and TCS) in the figures is 0.63.
Publications in WoS (Web of Science) and TCS (total citations) are based on the CWTS/Leiden indicators.
A-publications are categorised in the TUHAT RIS as follows:
0 500 1000 1500 2000 2500 3000 publications to A1−A4 (TUHAT). The figure on the right can be predicted by the first figure.
If the ratio of WoS to A1-A4 is high, the total citations (TCS) of RC can be expected to be high as well. The correlation between the indicators (WoS publications/(A1−A4) and TCS) in the figures is 0.63. Publications in WoS (Web of Science) and TCS (total citations) are based on the CWTS/Leiden indicators.
A-publications are categorised in the TUHAT RIS as follows:
A1 Refereed journal article A2 Review in scientific journal
A3 Contribution to book/other compilations (refereed) A4 Article in conference publication (refereed)
3.5 BIBLIOMETRIC INDICATORS
The bibliometric indicators are based on the CWTS/Leiden analyses.
Table 3. Biological, Agricultural and Veterinary Sciences: publications, citations and field-normalized figures in CWTS analysis (2005–2010)
RCS (27) ALL AC PWOS TCS MCS PNC MNCS MNJS THCP10 INT_COV
ARC 328 102 80 578 7.29 32.50 1.38 1.47 1.67 0.70
BIOSYST 632 393 174 567 3.26 45.40 0.59 0.71 0.50 0.47
CellMolBiol 183 176 164 2445 14.97 13.41 1.63 1.49 1.85 0.94
CoE MRG 341 293 239 2174 9.21 21.34 1.66 1.46 1.95 0.75
CoE_VIRRES 148 145 123 704 5.76 18.70 0.83 1.45 0.45 0.88
CoE-MiFoSaPLUS 566 366 318 2070 6.57 24.53 1.27 1.13 1.10 0.83
EGRU 258 195 172 1286 7.64 22.09 1.64 1.28 1.77 0.78
ENIGMA 228 171 117 785 6.80 29.91 1.57 1.24 1.85 0.61
EvoDevo 147 137 100 927 9.49 29.00 1.45 1.28 1.82 0.75
FoodNutri 880 497 354 2025 5.72 22.03 1.33 1.28 1.13 0.82
FRESH 374 218 103 424 4.12 33.98 1.19 1.18 0.82 0.68
HelDevBio 184 176 141 1364 9.73 22.70 1.36 1.27 1.42 0.93
INBIOS 524 260 179 596 3.35 44.13 0.74 1.01 0.67 0.68
LEGMILK 184 138 91 405 4.45 28.57 1.11 1.00 1.19 0.81
MEMBREC 146 130 121 1530 12.64 18.18 1.36 1.34 1.11 0.93
MICRO 405 337 263 1307 4.97 24.71 0.89 1.05 0.70 0.83
MUSGEN 144 118 100 780 8.04 18.00 1.36 1.15 1.48 0.90
PEATLANDERS 304 200 131 793 6.05 19.85 1.77 1.38 2.01 0.73
PHABIO 125 115 72 390 5.42 22.22 1.41 1.09 1.35 0.84
PHYTOPATH 174 129 98 538 5.49 24.49 1.23 1.17 1.26 0.80
SB&B 341 337 292 2227 7.66 20.89 1.91 1.26 0.79 0.87
SSA 1130 698 343 1224 3.57 36.73 0.86 1.03 0.76 0.71
SUVALUE1 434 283 101 231 2.29 44.55 1.21 1.06 1.17 0.50
VetSci 604 439 279 734 2.67 38.35 0.95 1.21 0.84 0.76
ViiGen 251 234 195 2851 14.73 19.49 3.09 1.56 1.78 0.85
VITRI2 88 59 35 79 2.26 37.14 1.06 0.99 0.62 0.58
VMPS 199 147 111 1749 15.76 16.22 2.34 1.71 2.34 0.89
Total 9322 6493 4496
12
1 CWTS analysis covered under 40 percent of scientific publications of the RC, thus the HU Library analyses were also applied.
2 The number of publications of RC was under a critical point i.e. 50 publications although the internal coverage was over 40 percent.
CWTS analysis: Number of publications (PWoS), Total number of citations (TCS), Number of citations per publication (MCS), Percentage of uncited publications (pnc), Field-normalized number of citations (MNCS), normalized average journal impact (MNJS), Field-normalized proportion highly cited publications (top 10%) (THCP10), Internal coverage, i.e. inside WoS publications (int_cov).
The number of all publications (publication types from A to I) is based on the TUHAT RIS data where A and C publications belong to the scientific publications:
A1 Refereed journal article A2 Review in scientific journal
A3 Contribution to book/other compilations (refereed) A4 Article in conference publication (refereed) C1 Published scientific monograph
C2 Edited book, compilation, conference proceeding or special issue of journal
MNJS in relation to MNCS of publications of RCs
EVALUATION OF RESEARCH AND DOCTORAL TRAINING 2005–2010 10
proportion highly cited publications (top 10%) (THCP10), Internal coverage, i.e. inside WoS publications (int_cov).
The number of all publications (publication types from A to I) is based on the TUHAT RIS data where A and C publications belong to the scientific publications:
A1 Refereed journal article A2 Review in scientific journal
A3 Contribution to book/other compilations (refereed) A4 Article in conference publication (refereed) C1 Published scientific monograph
C2 Edited book, compilation, conference proceeding or special issue of journal
MNJS in relation to MNCS of publications of RCs
Figure 4. MNJS in relation to MNCS In Figure 5, the number of RCs is 263
Figure 5
. The RCs’ publications value for MNJS is placed on the horizontal axis, and the value for MNCS on the vertical axis. The axes indicate the world average (1.0). The MNJS indicator refers to the field-normalised average journal impact and describes the impact of the journals in which the RC published their papers. This describes the researchers’ level of ambition when choosing the journal in which to publish their research results. indicates that at least 20 of the 26 RCs belong to square 1, i.e., the RCs publish in high impact publications and their impact is high as well. Five of the rest of the RCs belong to square 2, publishing in high impact journals with MNCS very close to the world average.
Combination of indicators is applied by the Evaluation Office.
3The figure includes RCs with WoS publications ≥50, thus VITRI is excluded.
0,0
MNJS/MNCS relation (Biological, Agricultural and Veterinary Sciences)
4 1
2 3
Figure 5. MNJS in relation to MNCS
In Figure 5, the number of RCs is 263. The RCs’ publications value for MNJS is placed on the horizontal axis, and the value for MNCS on the vertical axis. The axes indicate the world average (1.0). The MNJS indicator refers to the field-normalised average journal impact and describes the impact of the journals in which the RC published their papers. This describes the researchers’ level of ambition when choosing the journal in which to publish their research results. Figure 5 indicates that at least 20 of the 26 RCs belong to square 1, i.e., the RCs publish in high impact publications and their impact is high as well. Five of the rest of the RCs belong to square 2, publishing in high impact journals with MNCS very close to the world average. Combination of indicators is applied by the Evaluation Office.
3 The figure includes RCs with WoS publications ≥ 50, thus VITRI is excluded.
Interpretation of square areas in the figure
Square 1: RCs publish their papers in high-impact journals and the number of citations they receive exceeds the world average in their field.
Square 2: RCs publish their papers in high-impact journals and the number of citations they receive falls below the world average in their field.
Square 3: RCs publish their papers in low-impact journals and the number of citations they receive falls below the world average in their field.
Square 4: RCs publish their papers in low-impact journals and the number of citations they receive exceeds the world average in their field.