FEATURE STORY

Evaluating the Impact of Development Programs: Turning Promises into Evidence

May 14, 2012


Image


Romania, like many EU Member States, is exploring opportunities to advance evidence-based policymaking through more strategic use of monitoring and evaluation (M&E), including impact evaluation. To maximize the impact of its strategic policies in various areas and build upon policies that work and rectify those that do not, more evidence-based policymaking is needed.  In short, policy makers need evidence-based reporting to determine whether programs are achieving the required results.

International experience suggests that advancing the impact evaluation agenda benefits from a core group of people who can provide advice to senior policymakers on the detailed aspects of impact evaluation. The recent functional analysis of the education sector undertaken by a World Bank team in Romania during 2010-11 also confirmed that M&E is an important challenge.

To respond to such challenges, the World Bank Office Romania hosted an Impact Evaluation Workshop – “Evaluating the Impact of Development Programs: Turning Promises into Evidence” delivered jointly by an ECA Human Development (ECAHD) and Human Development Network (HDNET) team.  The objective of the course was to give an overview of impact evaluation and provide hands-on, directly relevant learning opportunities for participants from the higher education and social sectors. The program included presenting cutting edge techniques of impact evaluation, an understanding of its importance for policymaking and program design, a clear understanding of how to construct a results chain, and practical skills in the application of impact evaluation techniques to their own projects. Paul Gertler, former HD Chief Economist, currently professor at University of California, Berkeley, Adam Ross, Economist, HDNCE, Joost de Laat, Sr. Economist, ECSH4, were part of the team at the event, which was coordinated by Roberta Bassett, TTL, ECSH2, and Mariana Moarcas, Sr. Operations Officer, ECSH2.

The event was organized as part of a regional technical assistance program on higher education with the support of the Executive Unit for Financing Higher Education, Research, Development and Innovation (UEFISCDI). It was attended by 35 representatives of various higher education institutions, as well as counterparts from the social protection sector.

The workshop was highly interactive, with a mix of small and large group sessions including a “hands on” application of what is learned.  During the first two days, the participants received training on impact evaluation methods as well as “hands-on” experience designing a high quality impact evaluation for their projects.

 


" This workshop was extremely useful given our current efforts to increase our capacity to undertake both ex-ante and ex-post evaluations of our programs and policies  "

Adrian Curaj

UEFISCDI Director

The immediate relevance of this exercise came through clearly: “This workshop was extremely useful given our current efforts to increase our capacity to undertake both ex-ante and ex-post evaluations of our programs and policies”, said Professor Adrian Curaj, UEFISCDI Director.

The teams presented their proposals on the third day of the workshop in order to receive feedback from each other.  Participants left having acquired an overview of cutting edge techniques, their policy uses and practical skills in the application of these to their own project work. “This was a great opportunity for us to learn about scientific methods for measuring impact”, stressed Professor Tudor Prisecaru, Vice-Rector of the University Politehnica of Bucharest.

This Impact Evaluation Workshop is expected not only to lead to increased capacity and identify possible impact evaluation opportunities in the short run, but to be part of a more sustained engagement between the Bank and the Romanian Government on this important agenda.

Presentations and handouts

Evaluating Impact: From Promise to Evidence
Paul J. Gertler, Professor, University of California - Berkeley
• Why evaluate?
• The policy impact of rigorous evaluation
• Types of evaluations
• Counterfactuals
• Introduction to quantitative methods for evaluating development programs
 
Measuring Impact: Impact Evaluation Methods for Policy Makers
Joost de Laat, World Bank
• Causality
• Experimental Methods
 
Monitoring and Evaluation: The Foundation for Results
Adam Ross, World Bank
• Global Focus on Results
• Monitoring vs. Evaluation
• Using a RESULTS Chain
• Introduction to Clinics
 
Clinic Session 1: Establishing a Results Chain

Adam Ross, World Bank

Evaluating Impact, Turning Promises to Evidence

Paul J. Gertler, University of California - Berkeley
 
Case Study: Telephone Campaigns to Encourage Voting

Adam Ross, World Bank

The theoretical concepts discussed throughout the day in the policy and technical track were reinforced through group work on a case study.  The case study entitled “Get out the Vote” focused on the various identification strategies and methodologies for constructing a counterfactual.

Api
Api

Welcome