Round Table – On going discussion in Delft – Summary Block 2

INTRODUCTION

The afternoon session was focused on methods. The objective was to have the opportunity of discussing a sample of methods which consider different variables and approaches (quantitative/ qualitative)

Just so that you can feel our pain, KPN the national telecom provider in the Netherlands had a major crash leaving us in this wonderful venue without Internet access. The ‘near time’ blog was therefore serviced by Sandra – who wrote the blog in the first place – jumping on her bicycle and finding an alternative system so that she can upload and download. This gives a whole new meaning to “posting” a blog.

We started by taking a quick look at five methods that are available for evaluation. The presenters each had 12 minute to present what could each be a whole da’s discussion – so these are very much summaries. Their compete PPTs will be made available on the General Information Box. At the end of the afternoon there was an opportunity to question each of the presenters. Beneath each presentation is therefore a brief summary of some of the points from that discussion.

Picture by Peter MCIntyre – Denis Zoungrana from 2IE Burkina Faso and Sompit Puyaratabandhu from the Office of Agricultural Economics in Thailand voting on what they see as the main challenges in evaluating WASH governance.

PRESENTATIONS

Presentation 1

Kristof Bostoen from the London School of Hygiene and Tropical Medicine (LSTM) focused on what he calls the dichotomy between global and local monitoring. He says that people often expect to get from local monitoring what they get from global monitoring. This is a mistake that often leads people to claim to measure impact when they don’t.

He gave a brief history of monitoring water and sanitation outcomes in the sector to underline that people have been struggling with this question for a long time. During the water decade (1980’s) WHO was the only organisation monitoring sector progress, based on data from national authorities. They filled in a questionnaire that did not take into account what was collectable or available, and they used different definitions for ‘access’.

In 1990, WHO and Unicef joined forces in the Joint Monitoring Protocol but continued to rely on national data, until 2000 when the LSHTM collated the Global Water Supply and Sanitation Assessment report, almost entirely based on population based data. From then on the reports have used national surveys but support to local sector monitoring disappeared.

Survey questions were open to interpretation – e.g. what is you main source of drinking water? Does time to collect water include waiting time? Analysis is often weakened by the fact that neither questions nor answers can be compared between countries. Sampling is also a problem (who you sample and how).

JMP is very focused on MDG targets and so does not example address sanitation (for which there is no target). There are also technical requirements that lead for example to previous estimate values changing for each publication.

Global and national data needs are different. It should be possible to collect water at a local (rather than national) level that would still be compatible with global data needs. At global level the JMP does not do a bad job (better than some critics say) but the nearer you get to local level the less useful it becomes.

The main lesson to draw is that there is no universal indicator. You need a lot of different data just to decide on one aspect – e.g. access.

Points from the discussion

· There was a way of refining questionnaires to make data compatible between local and national level.

· It would often be much lower cost for country bodies to do monitoring themselves because they did not needd to cmpress their activities into such a short time frame.

· The questionairres prepared for JMP could be improved on. However, if someone else produced a quesitonnaire it would not be accepted, becaus the bodies would nto be considered as authoritative. Once people start editign the quesitons they end up asking something different and the comparability is lost,

· Very often in governance we look at the impact of the infrastructure. People value that pipe or toilet rather than the dialogue and the software approaches are forgotten.

Presentation 2:

Harry Jones from ODI:

Focus of Outcome Mapping

Outcome mapping starts from an understanding that focusing on measuring impact is not realistic in relation to many development projects. The ‘impact’ of a particular intervention, especially policy, will not be felt properly by beneficiaries for a number of years. We need to consider the limitations on the influence of a project, and shape our planning, learning, and accountability functions around outcomes, which are further ‘upstream’ from impacts.

A project has:

· Sphere of control – its operational environment

· Sphere of Influence – Relationships & Interactions

· Sphere of Interest – social, economical, environmental states & trends

·

We cannot control everything we’d like to see change. Sustainable change involves the combination of a number of different factors, and is a product of the interaction of many different actors and stakeholders. The more successful a programme is, in some ways, the sooner its influence disappears (because activities and methods are integrated). We need to need to recognise messy realities.

Outcome Mapping is therefore concerned with the level where a programme has direct influence. The focus for learning and accountability are the boundary partners who are individuals, groups, or organisations with whom the programme:

· Interacts with directly to effect change

· Anticipates opportunities for influence

· Engages in mutual learning

You look for changes in their behaviour that would contribute to the wider change you would like to see. Behaviour is defined as:

- Actions

- Activities

- Interactions

- Relationships

In monitoring there is a need for a reality check. It is a fact of life is that many people have to work within log frames. Many donors insist on log frames and they have practical value and would not be so widely used if they were useless. The important issue is how to integrate principles from outcome mapping into a log frame?

Points from the discussion

· This is perhaps more of a management tool than an evaluation tool.

· We should be modest in our expectations and in what we promise. Change is not easy.

· There is great difficulty in institutionalising monitoring and learning in an institutional culture.

· Outcome mapping tells you what you need to measure but not how you need to measure it.

· This tool would be useful in monitoring governance.

Presentation 3:

Ratna Reddy: Assessing the Impact of Participatory Development Programmes: A Sustainable Livelihoods Approach (SSL)

It is a participatory method which looks to the different perspectives/ capitals that are affecting the Livelihood’s sustainability. The capitals considered are:

· Natural capital: natural resource stock (land, water, CPRs, etc)

· Physical capital: Basic infrastructure (infrastructure, physical assets, etc)

· Financial capital: Financial resources (income, savings, credit, etc)

· Socio-political capital: horizontal and vertical social resources (networks, group membership, migration, etc)

· Human capital: skills, knowledge, health, etc

Based on the analysis of those capitals, it is possible to define impact. The analysis/ comparison of those capitals could be done by using one of the following methods: ‘double difference’ method (before-after and with-without comparisons), Statistical validity (pair‘t’ test and‘t’ tests are used) and Causality tests (regression analysis).

The method allows analyzing different dimension and its relationship; nonetheless there are some limitations. The highlighted points were:

· The quality of the baseline data is not always at the appropriate level

· The attribution problem is the major problem

· The definition of indicators and the balance between capitals is not easy

Discussion

This method is very useful for monitoring interventions for poverty reduction.

Presentation 4

Deirdre Casella: Qualitative Information Systems (QIS)

This method is based on the participatory assessment approach. It results from the challenges faced when trying to define indicators related to sustainability and equity. The objective when defining this method was to be able to come out with a demand driven approach.

The following are the most relevant QIS’ characteristics:

• It is a mixture of both; quantitative and qualitative factors
• It allows to look at several factors together in order to facilitate to describe the sustainability level of a specific intervention
• It works based on the definition of a descriptive order of score. It means that different scenarios are created in order to describe the worst and best situation related to the intervention. In the middle is the intervention’s objective, which is considered the benchmark. Thus, based on those scenarios it is possible to self assess the current situation and explain why a specific score is given.
• It allows putting in number qualitative information (scenarios), in this way it is easier to disseminate the results obtained from the evaluation.

The limitations using QIS are mainly related to the source of information used for supporting the score defined to specific intervention. The method requires a strong support for validating the results. The application of process documentation techniques is required for avoiding this limitation.

Discussion

We concluded that it is useful for getting information about outcomes that are precursors to saying something about impact on sustainability and equity. It is useful for informing policy as well as programmes.

GENERAL PANEL DISCUSSION

After the presentations some questions raised from the table. The most relevant questions were:

1. What do QIS and SL methods measure?
2. What is the cost of applying the methods presented? – Information related to costs will be bring to the table
3. Which of those methods are “really” related to Impact Evaluation?
4. How could the risk of using a specific method be handled?
5. How can we select a method?
6. What can be the role of beneficiaries for defining IE?
7. What methods can be used for defining the intervention theory of a specific programme?

The group exercise will hopefully answer some of the questions. Nonetheless those questions will used as inputs for the production of the Thematic Overview Paper on Impact Evaluation

Before opening the group exercise, there was the need to take some time for analyzing/ defining the Impact Evaluation concept. The result of this discussion was that people used this to mean different things. There probably would not be a general definition that would be universally accepted, but there is a need to be clear about the different levels and approaches that can be used. When people used this term they should be clear about what they were measuring. There did seem to be agreement that monitoring was needed both of long term outcomes and of intermediate impacts or outcomes, but there was no need to have a clear definition of where the boundaries are.

Deirdre Casella from IRC said that the aim of the round table was to expose what is known about impact evaluation and where we do not have enough shared knowledge. There is a need to match this to the work we are doing so we do not spend vast resources on measuring the wrong sort of things.