Round Table – On going discussion in Delft – Summary Block 3

INTRODUCTION

 

Sun & Music… good combination for starting the reflection on yesterday’s session.

The round table began the second day with two presentations about defining indicators for institutional change. By chance, both presentations focused on projects in Kenya.

All presentations will be available at the General Information Box

PRESENTATIONS

Presentation 1: Richard Rheinghans – Emory University, Atlanta

Measuring success of school WASH: evaluating impacts or evaluating for impact?

The team at Emory University is working with CARE and other partners in Western Kenya to address the question of whether schools could be a key entry point for WASH even though children of school age are not a key target group (i.e. they do not have the highest rates of mortality.  This is an ongoing project. The hypothesis was that this is plausible because:

•             Schools have an institutional presence in the community

•             In Kenya there is relatively equitable access

•             Children can influence parents. Could schools trigger behaviour change?

Research and Evaluation Objectives

The aim is to assess the strengths of children’s influence in motivating households to adopt health-seeking and hygiene behaviour: to assess the potential for sustaining and scaling school based WASH interventions.

Research interventions were jointly planned with CARE and partners (who carried out the interventions, starting in 2007). Emory was responsible for the evaluation. The aim was to determine:

•             Improved knowledge and uptake of behaviours like handwashing

•             Diffusion of behaviour change into the community

•             Health impact anaemia, helminths and diarrhoea

•             Educational impact (absenteeism and test scores)

•             Cost effectiveness

 

Long-term outcomes (for such things as diarrhoeal mortality) cannot be directly measured and need to be extrapolated. The team is seeking ways to extrapolate for example from improved test scores to (eventually) to increased productivity.

Emory conducted:

School-based data collection in a total of 185 schools in three different districts. In each school: 

·         There was an assessment of facilities

·         25 children were interviewed

·         Water quality was tested

·         Hand rinse sampling was conducted (to measure contamination)

Community assessment through

·         Household interviews

·         Helminths and anaemia testing

Measures of success depend on the questions you want to answer. Questions include: 

·         Do school programmes provide valuable benefits to children and communities?

·         Is this where money should be going?

·         Did this programme provide benefits to this population?

·         How can this programme provide the most benefits?

 

Each of these questions demands different tools to find an answer.

Donors recently asked a series of tough questions about effectiveness.

•             How can we ensure we are doing the most we can?

•             Are interventions likely to be sustained after three years?

•             How can we maximise the sustainability of the interventions.

How to measure this became a touchy question. In general, in projects, nobody wants an evaluation because performance is never quite what you promised. It seems a risk because evaluation can lead to the cutting off of funds.

In early 2008, Emory conducted a survey in 55 of 60 schools where CARE had implemented a minimum package of interventions in 2005.

They looked at some key indicators to show sustainability. Did the schools

·         Provide water in safe storage containers?

·         Test drinking water?

·         Provide soap and water for hand washing?

·         Self report containers and taps that were broken.

The results in fact showed rather low sustainability:

·         36% of schools providing drinking water,

·         20% had treated water but only 9% had chlorine residual (which showed that water had been treated).

·         18% provided water for hand washing but only 2% (one school) had soap.

In some ways this confirmed their worst fears about sustainability. However there were some more encouraging indicators:

·         75% had repurchased water treating products.

·         50% had allocated funds for water treatment and soap.

·         16% had active health clubs

It also identified barriers to lack of water access:

·         Hardware design

·         Financial resources

·         People and institutions

An important learning point was that they sent time with partners to identify strategies to address these barriers. They were then able to make a more positive presentation to donors: Here is what we were doing. Here is where we are missing the mark. Here is what we are planning to do to close the gap.

Rick Rheingans addressed how this approach could pose questions related to Governance evaluation.

·         How can improvements in governance provide measurable benefits to users?

·         Did the project provide measurable impact to beneficiaries?

·         Did the project provide improvement in governance or likely to do so?

The bigger question is whether there an evidence base to connect project demonstrations of improvements in governance or outcomes to long term impacts?

Discussion 1

 

·         Going from outputs to outcomes creates risks for projects and incentives are needed to do that. The risk is that you will be ‘punished’ for failure – instead you need to find incentives to make it more attractive to the NGO or implementer.

·         There was a control group of schools that will receive an intervention later and they were randomly selected.

·         Some of the barriers did relate to governance. One of the improvements they needed was more support from the Ministry of Education.

·         Other key issues were leadership within the school and community support.

·         This was outcome evaluation: what would impact evaluation mean in this context?

·         Impact evaluation would be the educational behaviour and health effects to different interventions such as anaemia, diarrhoea etc. We measure what we can measure and maybe can extrapolate a lot of things we really care about like productivity. We think about impact as the magnitude, and sustainability as the time period. The research would assign an economic value to the cost of diarrhoea. However, the impact on mortality (life years) would have to be modelled because we cannot measure it directly.

·         Governance issues become more important as you move from a project to scale. Then the point of entry is no longer the school, but the government (Ministry of Health or Education or Water). Co-ordinating groups become the intermediaries between those and the NGOs. Responsibility for different interventions, lies with different agencies.

 

Presentation 2:  Alistair Sutherland  - National Resources Institute UK.

IDENTIFYING INDICATORS:  AGRICULTURE R&D….

This presentation focuses on presenting an example from the agriculture sector in Kenya on the considerations and methods used for defining the indicators related to an action research project.

The objective of the project was to define how it could be possible to show the impact of conducting a research. The focus of this project was to look at the good practice that facilitates to put the findings of a specific research in use.

The most relevant aspects covered during the research were:

·         Local service extension delivery: The objective was to look at the impact or importance of using the results of the research from the service delivery perspective. Among others factors such as coverage and how the good performance of the service deliverer is rewarded were analysed.

·         The definition of targets and its application was also mentioned. The targets defined in terms of service delivery are not always realistic.

During the design phase, there were some delays due to that it was not clear at the first place how the results of the study would be used.

The expected outputs of the research were defined in four main groups. Group 4 was related to influence policy. The objective was to cover all the aspects that can influence the use of research results.

In order to conduct this research, some indicators were defined. The following are the highlighted points related to the methodological approach and aspects considered when defining those indicators:

·         The starting point was the project Log-frame

·         They used a mixture of indicators. Nonetheless, the focus was to answer WHY – Qualitative information

·         The involvement of different stakeholders at different levels was considered important

·         Different perspectives were considered when defining the indicators

·         The KAP (Knowledge – Attitudes and Practice) model – model for defining change was used

·         It was pointed out that the use of simple indicators (e.g age) could be meaningful in some cases

·         Other less “SMART” indicators can add value (e.g. Economic indicators and indicators developed by the community)

·         Indicators were defining considering the perceptions and needs of the beneficiaries. Often they are focused on the tangible results, it is what is important for them

·         The definition of indicators should be done considering how it could be analyzed and presented to its users

·         The context around indicators is also important. Some indicators are much more meaningful when information about its context is available and presented

During the presentation some insights on how it is possible to engage Policy makers were presented. One of the most important actions mentioned was to allow beneficiaries to talk about/ present the results of the indicators. In this context the definition of good governance was discussed. There was not a final word, but it was clear that this definition has an impact on the way in which a specific project is designed.

At the end a summary of points for reflection was presented. The highlighted points were:

·         Tension between using IE as a learning tool, while it is not defined and done but the implementers

·          Can corruption be defined as an outcome / impact?

·         Can Governance be defined as a process? Can we change it?

·         Is the intervention theory of projects focused on Governance issues clear?

 

 

 Discussion 2

 

After the presentation, the following points were discussed:

·         How to balance the need of having quick lead indicators and to define indicators related to IE? – The discussion brought this dilemma; the answer is that what is important is to be clear about what can be measured and which moment. Also what is clear is that there is the need of having different kind of indicators in place (Outcome, input and outcomes), it facilitates to say something about the project performance

·         An example of using mix methods was mentioned. It is possible to add value by looking at both; the outcomes (quantitative) and changes/ achievement at the institutional level.

·         Risk management should be considered as important factor during the whole project life cycle. It has been found that after a long period some project are “deleted”, because the risks around it were not considered