Round Table – On going discussion in Delft – Summary Block 1

THE SUN HAS GOT ITS HAT ON IN DELFT

The expert round table opened in Delft in brilliant sunshine and with brilliant minds (we hope). However, the Internet link crashed just as we began so this brilliance has been delayed. We hope it was worth waiting.

WELCOME

The round table was opened by Ben Lamoree, IRC Director. He cited a World Bank definition of Impact Evaluation (IE):

“Impact evaluation assesses the changes in the well-being of individuals that can be attributed to a particular intervention, such as a project, programme or policy.”

He highlighted however a number of points that made it somewhat less clear-cut:

· The way of perceiving and understanding impact evaluation changes according to your point of view.

· The what and how questions are always open and reflect the tension behind IE.

· For whom IE is done/ useful

EXPECTATIONS

The opening exercise was focused on mapping the expectations of the Round Table participants. The following are among the expectations outlined:

· Talk about reality: Theory is good, but it is important to be able to see the “real” side of it during the discussion here

· Use specific examples and specific applications: it is important for participants to be able to have an idea about how the discussions can be applied in their work

· To understand how IE can be done in the context of complex interventions

· How to use IE for learning? How to use this learning?

· To balance – Why and How of IE: The why should lead the how

· IE should be used for showing whether or not for example the local Government is ‘on Track’

· IE is a wider concept: during those two days it is important to recognize the different components of this concept

· To think outside the box, not only to think about results but also about processes

· How it is possible to get value from the money/ resources relates to IE

· To analyse how the tension between management expectations (project point of view) and Government expectations can be balanced, and especially with what professionals can deliver

· IE vs corruption: there is a need to understand how IE can lead to a lower corruptions levels

In block 4, there will be the opportunity for reconsidering those expectations and define which points should be discussed deeper.

INTERVIEWS

The objective was to bring to the table specific examples of interventions focused on Institutional Change by asking to some participants to shares their experiences related to IE in a real case. Four participants were quizzed by Joep Verhagen.

Picture by Peter McIntyre – Steve Mogere, Monitoring and Evaluation Advisor for the Japan International Cooperation Agency (JICA) in Kenya, being interviewed by Joep Verhagen of IRC

Interview 1: Stephen Mogere: JICA – Kenya

JICA is Japanese government donor agency which does not do direct project implementation but has three main components in Kenya:

1. Infrastructure

2. Capacity Building

3. Networking

When trying to apply IE, it was a challenge to handle the tension between Governmental objectives and project objectives was a big challenge; it was difficult to link the project’s results from the Government perspective and the real implication at the ground/ beneficiaries level.

Government often sees evaluation done by donors as an audit. This makes it difficult to link IE with reality. It can come to be seen as a routine process rather than something they can learn from.

Interview 2: Sompit Puyaratabandhu – Office of agricultural economics Thailand

Her example came from her 20 years experience in IE as a Governmental officer. She feels that the work she had done was often more about outcome evaluation rather than impact evaluation. Her department had to respond to the immediate needs for this kind of evaluation and she would like the opportunity to do outcome evaluation after a long time frame to discover for example whether farmers who had benefited from an irrigation project still had access to water and an improved income. Often the problems were not immediately apparent, especially if maintenance was not properly done.

Interview 3: Donal O’Leary – WIN

WIN work has three main components:

1. Awareness raising

2. Networking

3. Resources on the ground

IE has been done mainly as a requirement of the donors. The idea is to link the three programme elements in order to be able to “prove” impact in the ground. However it is was also needed by those engaged in the work. If these issues are not addressed it damages the sustainability of water supply and sanitation and ultimately the health or people. But it was expensie and time consuming. For example, if a water facility was properly reporting its performance, good audotrs talk to people on the ground (including the users) and track losses and corruption.

Interview 4: Patrick Moriarty – EMPOWERS

The objective of Empowers was to support the improvement of Integrated Water Resources Management (IWRM) by strengthening local governance. At the beginning it was not clear how to achieve this objective. Process documentation was used to track changes in the thinking and interactions of actors at many levels.

However he said he was skeptical about impact evaluation when trying to make changes in governance. Impact would perhaps be felt over a ten year period and the sector is seething with activity. This made it very difficult to separate out the impact of a particular intervention.

Regarding to IE, the following were the most important points:

· How to consider all the factors related to the programme

· How to use IE for supporting learning

· How to balance the external and internal factors which are affecting the intervention

“The objective is create self reflective learning processes, so that learning is intrinsic to the process. It has a very strong learning element. I would make a difference between internal learning and external log-frame driven evaluation.”

MAIN CHALLENGES

In four groups participants discussed some of the main challenges they needed to address

We have tried to summarise them here.

Group 1

1. The intervention needs to have a clear and coherent strategy if you are going to do an impact evaluation – your need to know the objectives and the planned a means of getting there.
2. Do you need control groups? This is difficult in situations where you are working with institutions and they are interconnected. Should you be doing it and if so how?
3. How do you know what is te the impact of your invention and that of other inputs.

Group 2

1. If you are seeking institutional change then the evidence will be found on the ground in good services and responsive organisations. It takes a long time to show that, but you do not have to do it all at once. If you have good processes indicators you can go ahead with intermediary outcomes.
2. It is important that stakeholders are engaged but then they are not likely to bring out negative aspects in an evaluation. The challenge is to change the artuide. This is hardest to achieve with government stakeholders so that they do not refard IE as fault finding but a means of leaning, so that they cone ot their offices and engage in the process.
3. It is very had to get the evidence based documentation.

Group 3

1. Understanding why the intervientions are taking place is difficult. Log frames are often done after the fact, and are no use for learning.
2. Should focus on the impact we want to have not on what we did base don an assumption that we already assumed it would have an impact.
3. What are different reasons for having evaluation: upward (donors) and downward evaluation (to beneficiaries) but also learning as a key purpose (how do we do thins better?). Depending on the audience impact evaluation may be different .
4. Best practice how do you do what you do best?
5. What is The framework ? The project does not control all of the field and it has a temporal limitation.

Group 4

1. Impact analyses are often perceived as audits that you have to do for someone else and not integrated into the pojrect. How can you change mindsets so they do it for their own benefit?
2. How can you attribute effects to the programme. Impact is difficult to measure. We could maybe measure the possibility that what we have done night lead to the outcome, especially if the programmes have a short time frame.
3. Donors fund the process and they do not want to hear bad news.
4. Unplanned outcomes (positive and negative) are not measured.