Evaluation use

Evaluation utilisation has been a constant topic of lively debate and of research on evaluation since the 1970s.⁴⁶⁸ It is arguably the most studied evaluation area.⁴⁶⁹ Despite the substantial and ever-increasing American literature on the theme, oddly, evaluation use has been left remarkably untouched in the European context, let alone in Africa, the context of this research.⁴⁷⁰ Indeed, studies conducted on evaluation use in the African context, except for Marra and Podems, which I know, are rare.

465Carman2010, 256; Snibbe 2006

466 Fowler 2000, 138

467 Carman & Fredericks 2010, 84–104

468 e.g., Alkin & Taut 2003; Amo & Cousins 2007; Balthasar 2009; Baptiste 2010; Bhola 1990; Carlsson 2000; Chelimsky 1995; 1998; Christie 2007; Cousins & Shulha 2006; Dahler-Larsen 2009; Davidson 2005; Feinstein 2002; Fetterman 2001; Fleischer & Christie 2009; Forss, Cracknell & Samset 1994;

Forss, Rebien & Carlsson 2002; Harnar & Preskill 2007; Henry & Mark 2003; Johnson 1998; Johnson, Greenseid, Toal, King, Lawrenz & Volkov 2009; King 2007; Kirkhart 2000; 2005; 2011; Ledermann 2012; Lehtonen 2005; Leviton 2003; Lindqvist 1999; Mark 2011; Mark & Henry 2004; Marra 2000;

Patton 1997; 2007; Podems 2007; Preskill 2004; Preskill, Zukerman & Matthews 2003; Raivola 2000;

Rajavaara 1999; Rönnholm 2005; Saunders 2012; Schaumburg-Müller 2005; Stufflebeam & Shinkfield 2007; Taut 2007c; Valovirta 2002; Vedung 1997; Weiss 1988; 1998; 1999; Widmer & Neuenschwander 2004

469 see e.g., Christie 2007, 8; Fleischer & Christie 2009, 171; Mark & Henry 2004, 35

470 Christie 2007, 8; Ferry & Olejniczak 2008; Fleischer & Christie 2009, 171; Mark & Henry 2004, 35;

Marra 2000; Podems 2007; Saunders 2012, 422 144

When Marra studied utilisation of evaluations in the development field and concentrated on the use of evaluation findings of the World Bank Institute to contribute to anti-corruption activities in East Africa, Podems conducted a case narrative on process use of an HIV/AIDS programme evaluation in Southern Africa.⁴⁷¹

When speaking generally about use, such features as dissemination, diffusion, application, and exploitation are essential, as is the word utilisation. Again, usefulness, usability, use or utility of evaluation, whatever you want to call it, has become a significant indicator to measure evaluation quality. Utility, one among other standards of evaluation quality — feasibility, propriety, accuracy, and evaluation accountability — set for evaluation by the Joint Committee on Standards for Educational Evaluation, basically ensures that intended users of evaluation could gain the needed information from evaluation.⁴⁷² For instance, the OECD-DAC went further by differentiating in evaluation use such concepts as evaluation feedback and dissemination. To them feedback ensures that evaluation lessons become part of a learning cycle within organisations rather than just dissemination, distribution of evaluation findings.⁴⁷³ Feedback, then, is essential not only when referring to new development activities, but also is worth linking directly and strongly to existing, on-going development activities. Manifestly, these kinds of capacities need to be developed, for on-going practices are essential for conducting and managing evaluations, and crucial in promoting both ownership and the use of evaluation processes.⁴⁷⁴

Evaluation use is a contributory factor in evaluation impact. The changes originating from evaluation utilisation are called various things. These evaluation impacts, results, consequences, outcomes, and influences can cover, for instance, improvement and development in practices of an intervention, or decision making based on evaluation findings about this intervention, or the process of learning through collaborative evaluation practices.⁴⁷⁵ Mark and Henry, as well as Kirkhart (in her 3-dimensional integrated theory of evaluation influence) prefer to call these outcomes evaluation influences.⁴⁷⁶ To Kirkhart this influence was a person or thing’s

471 Marra 2000; Podems 2007

472 Kuula 1999, 30; Rajavaara 1999, 34–35; Rönnholm 2005, 62; Stufflebeam & Shinkfield 2007, 87, 90–92; Yarbrough, Shulha, Hopson & Caruthers 2011

473 OECD-DAC 1990 in OECD-DAC 2001, 60

474 see e.g., Feinstein & Beck 2006, 541

475 Raivola 2000, 66; Weiss 1998, 21, 23

476 Kirkhart 2000, 7; 2005; 2011, 74–75; Mark & Henry 2004, 46

power or capacity to contribute effects on the others by using the evaluation in different ways,⁴⁷⁷ while to Saunders as well as Pickford and Brown, they were simply evaluation impacts.⁴⁷⁸

Traditionally, evaluation use is narrowly linked with utilisation of evaluation findings. These results, with conclusions or recommendations used, are typically written on evaluation reports published.⁴⁷⁹ This is exemplified, for instance, in the studies made by Valovirta and Ledermann. Valovirta dealt with evaluation utilisation as argumentation in his empirical study made on the use of evaluations in Finnish government agencies in 2002. He accomplished this by studying the important roles of all accessible evaluation reports from argumentative perspectives: discussion, dialogue and argumentation.⁴⁸⁰ Ledermann identified conditions which are necessary but insufficient for evaluation use when studying 11 Swiss development program and project evaluations in 2011.⁴⁸¹ Evaluation use, in these cases, has been inquired from the viewpoints of which procedures and implications have been taken, after the evaluation, on the grounds of recommendations through feedback processes.⁴⁸²

Evaluation use has various definitions. If Christie regards evaluation use as “the effect the evaluation has on the evaluand—the ‘thing’ being evaluated—and those connected to the evaluand;”⁴⁸³ then Saunders’s definition is consonant with Weiss’, where evaluation use simply stands for producing positive change.⁴⁸⁴ Johnson, Greenseid, Toal, King, Lawrenz, and Volkov used the evaluation use or utilisation concepts interchangeably by defining evaluation use “as the application of evaluation processes, products, or findings to produce an effect.”⁴⁸⁵

However, not only can the findings of evaluation be used, but the process of evaluation can be used also, as Mark, Patton, Vedung, as well as Alkin and Taut emphasised.⁴⁸⁶ Saunders also named the process use in his typologies of evaluation use apart from instrumental, conceptual, enlightenment, and persuasive or symbolic (justificatory) uses.⁴⁸⁷ To Kirkhart the source for the change in evaluation can be

477 Kirkhart 2000, 7; 2005; 2011, 74–75

478 Pickford & Brown 2006, 4; Saunders 2012, 427–431, 434

479 Patton 1998, 225

480 Valovirta 2002, 78

481 Ledermann 2012, 159–178

482 Forss, Rebien & Carlsson 2002, 29; Laukkanen 1998, 154

483 Christie 2007, 8

484 Saunders 2012, 433; Weiss 1998, 31

485 Johnson, Greenseid, Toal, King, Lawrenz & Volkov 2009, 378

486 Mark 2011; Patton 2007, 99; Vedung 1997

487 Saunders 2012, 425

146

either the evaluation process, then called process-based, or evaluation findings, known as results-based, or their combination.⁴⁸⁸ Besides these two elements, Christie and Alkin as well as Cousins and Shulha identified a third evaluation dimension, the commissioning of the evaluation, which could be utilised as well.⁴⁸⁹

Both evaluation findings and processes can be used instrumentally according to Alkin and Taut, either conceptually or legitimately. This means that instrumental or conceptual findings can be produced either formatively or in a summative way, while legitimate use refers only to summative evaluations and legitimisation of decisions based on evaluation findings.⁴⁹⁰ To Greene, instrumental use of evaluation is action-oriented, conceptual use is learning and education-oriented; while symbolic use is either persuasive or politically-oriented.⁴⁹¹

The direct, linear, instrumental use of evaluation has typically taken place after the evaluation is at least partly complete. Then, evaluation recommendations and findings have been used in the decision-making process and problem solving to modify and improve, in some way, the object of evaluation, the evaluand.⁴⁹² To Carlsson, an evaluation is instrumental if “the operations are tried and tested and where the results from such a testing are fed back into the planning and implementation of the operations.”⁴⁹³ In addition, evaluation processes can be used instrumentally, to generate actions. When referring to evaluation purposes, judgement-oriented and improvement-oriented evaluations include the instrumental use of evaluation based on which actions are taken.⁴⁹⁴

The conceptual evaluation is used for educative purposes.⁴⁹⁵ This type of evaluation use is also called enlightenment. It refers to changes occurring in thoughts, ideas, insights, concepts, generalisations, and feelings while evaluating.⁴⁹⁶ Through understanding, conceptualising (conceptual schema) and learning, thinking can be stimulated regarding issues, and knowledge or understanding can be

488 Kirkhart 2000, 7; 2005; 2011, 74–75

489 Christie & Alkin 1999, 3–6; Cousins & Shulha 2006; Rich 1977 in Baptiste 2010, 46; Shulha &

Cousins 1997

490 Alkin & Taut 2003, 7

491 Greene 1986 in Garaway 1995, 88

492 Harnar & Preskill 2007, 27; Ledermann 2012, 160; Mark 2011, 106; Nevo 2009, 297; Saunders 2012, 425; Shulha & Cousins 1997; Weiss 1998, 23–24

493 Carlsson 2000, 121

494 Mark & Henry 2004, 36; Mark 2009, 61; Patton 1997, 70

495 see also Harnar & Preskill 2007, 27; Levin-Rozalis, Rosenstein & Cousins 2009, 195; Mark 2009, 61–62; 2011, 106; Mark & Henry 2004, 36; Nevo 2009, 297; Patton 1997, 72; Shulha & Cousins 1997

496 Mark 2011, 108; Mark & Henry 2004, 36

improved about a problem or its solution,⁴⁹⁷ in order that old ideas can be questioned by providing new views as a result of evaluation findings or processes. Both evaluation elements could be used by the local program people by increasing knowledge and creating new ways of structuring program operations.⁴⁹⁸ This can take place without influencing decisions or activities; so that anyone’s — not only those participating in the program or its evaluation — knowledge in the field can be generated, or through rearranging the policy agenda, for instance.⁴⁹⁹ Next, we concentrate on the processual use of evaluation in more detail.