Evaluation

Programme evaluation provides valuable information about the operations of the program-me, the extent to which the programme has met its intended aims, and beneficiaries’

percep-tions of the programme. Organisapercep-tions can adjust the contents and scope of their program-mes accordingly to better meet beneficiaries’ needs. The results of such evaluations can also be used as supporting evidence for programmes seeking to secure additional funding or the support of partners to continue their programme.

While baby box organisations acknowledged the value of programme evaluation, many found it difficult to conduct a sufficient impact evaluation. Organisations frequently report-ed difficulty in designing a suitable evaluation strategy, especially surrounding the decisions of what to measure and how to measure it. Additionally, many programmes were in the early stages of implementation, making it difficult to gather statistically significant sample sizes for scientifically valid evaluation. Other practical concerns included limited budgets and unreliable data collection. Programmes collaborating with partners to deliver boxes some-times relied on the same partners to conduct an evaluation for practical purposes. However, this partnership did not always allow the programmes to directly control the quality of the collected data.

Organisations may be able to address these design challenges by developing evaluation strategies based on the programme’s purposes and intended aims. The baby box programme in Jordan was a part of a wider multi-component programme, which aimed to empower women in a refugee camp via increased access to livelihood opportunities. Therefore, the organisation conducted end-of-project cycle evaluations and commissioned an independent researcher to monitor the programme and assess its expected impact on women’s empower-ment in the camp, their ability to meet their financial needs, and their role in making deci-sions within their family and community, as well as the expected impact on the reduction of domestic violence among programme beneficiaries.

Programmes with quantitatively measurable aims, such as increasing rates of ANC or facility deliveries, may find it easier to conduct an impact evaluation. The evaluation of pro-grammes in South Africa and Zambia were designed as randomized controlled trials to as-sess whether the baby box intervention would incentivise mothers to promptly seek ANC or deliver at a health facility. The programme in Canada evaluated their intervention’s effects on maternal psychosocial health by examining mothers’ postnatal depression scale scores throughout the study.

Several programmes also considered economic data while evaluating their programmes.

The study in Zambia included a cost-effectiveness analysis, measuring the estimated cost for each maternal or neonatal death averted by the baby box intervention. The programme in Canada acknowledged the importance of including economic data, such as return on invest-ment data, in future evaluations.

The baby box

64

Other programmes designed their evaluations using a more qualitative approach by col-lecting beneficiary feedback. Beneficiary feedback typically included whether items in the box were useful and which elements of the programme did or did not work well for them.

Feedback was collected in various ways, such as conducting interviews, group discussions, or surveys. Programmes used beneficiary feedback to revise their approach and the contents of their boxes, as well as to foster a more human-centred, locally and culturally relevant pro-gramme. Programmes emphasised the necessity of collecting user opinions as early as possi-ble in programme development to orientate further pilots or the official launch. For example, the programme in Scotland conducted development-stage research with parents. They used the results to guide decisions regarding the box contents, the optimal time for parents to receive the box, which health professional would be best to communicate with parents, and which core messages most accurately relayed the programme’s intentions to the public.

In addition to evaluating the programme’s impact on beneficiaries, monitoring and as-sessing operational processes also played an important role in helping programmes achieve their goals. Programmes commonly obtained operational data by keeping records of deliv-ered boxes and box recipients, as well as recipient participation in related classes, activities, and events. The programme in New Zealand developed a monitoring and evaluation system based on the theory of change, a methodology which can be used to define the inputs and outputs required to achieve long-term social change. Their system included both operational data and user feedback not only to continually refine the operation of the programme, but also to measure the degree to which the long-term goals were reached.

Implementation of a baby box programme requires a significant amount of time and must be continuously refined throughout the duration of the programme. Obstacles may arise in any part of a programme and through every stage of a programme’s lifecycle. Fortu-nately, these challenges are also the springboards for future innovations that may improve the lives of baby box recipients: mothers and babies.

9 CONVERSATIONS

Here we expand on the results of our Thinking Outside the Box study to explore timely top-ics surrounding the baby box. The following questions have generated interest in the media, general public and academic circles. Globally, the vital need to support infant and maternal wellbeing has guided questions about the baby box and its potential role in safe sleeping, social protection, gender issues, humanitarian emergencies including refugee settings, and children’s rights. The international scope of our study encourages an insightful, diverse set of conversations surrounding these and other timely topics, presented in this chapter.