Synopsis
The World Bank has systematically been using rigorous impact evaluations in Bank-funded education projects around the world to help generate knowledge about how to improve results in education since 2006. Drawing on evidence from 22 rigorous impact evaluations across 11 developing countries1, culminating this year in the book, Making Schools Work, the Bank has been able to show that student and school “report cards” in Pakistan led to improvements in learning achievements, that parental oversight of teachers in Kenya was strong when combined with other reforms, leading to improved test scores, and that the use of contract teachers in India led to improvements in learning outcomes.
Approach
Impact evaluation techniques are important development tools that compare the impact on beneficiaries of a policy action or project with another similar group that has not been exposed to these activities. The results can provide evidence on whether current policies are working and help inform policymakers about where to allocate scarce resources.
The Bank’s work on education has focused on promoting rigorous evaluations of three of the most common types of accountability to promote reforms in basic education: school-based management, information provision to empower school stakeholders, and teacher contracting and incentive reforms. The Bank has also significantly contributed to the knowledge base on demand-side interventions that aim to increase school attainment among the poor.
Results
Information for accountability is a potentially useful tool for improving learning outcomes, but its success at leveraging change relies on many factors. For information to be effective, service providers must have the ability to change in response to the accountability pressures it can induce. If providers are circumscribed in their ability to take action, information can have little or no impact. Information is therefore critically linked to local decision-making authority. The impact of information is also likely to be higher if it is linked to performance rewards or sanctions.
- In Liberia, two strategies for improving early grade reading outcomes were evaluated and compared with a control group beginning in 2008. The first evaluated approach consisted of sharing results with parents and communities and showing teachers how to prepare report cards. The second approach added an intensive teacher-training component focused on methods for reading instruction. By 2010, the information-only intervention increased only the measure of letter fluency by a 0.24 standard deviations. On the other hand, the “full” intervention—which included the information dissemination as well as the teacher training components—had statistically significant impacts on all the indicators measured. The effect sizes were estimated to be large, from a 0.38 standard deviation in listening comprehension to 1.40 standard deviations for unfamiliar word decoding. The study also showed significant impact on assessment of math skills, even though math instruction was not part of the training.
School-based management (SBM) can improve learning outcomes, though the results are mixed. It is likely that for school autonomy to be effective, it must entail a real transfer of authority (that is, provide school committees and parents with real tasks) and the building of capacity to carry out those tasks. Nevertheless, recent evidence shows modest but significant gains in learning in some contexts.
- In Mexico, two randomized trials study the effect of weak-to-intermediate autonomy reforms. Both SBM interventions are in early implementation. They show small effects on outcomes, mostly in early primary grades; but they also show changes in school climate, parental (low-income, rural, indigenous) perceptions and participation, and engagement.
- In the state of Colima, a study of the federal Programa Escuelas de Calidad (Quality Schools Program, or PEC) between 2006 and 2009 found that test scores increased overall for both treatment and control schools, although the differences were small. When disaggregated by grade, however, the cohort that was exposed to treatment the longest experienced the largest gains with scores increased by a 0.5 standard deviation between third and sixth grade if they were in the treated schools, while those in control schools increased by a 0.34 standard deviation.
- Preliminary analysis of the Apoyo a la Gestión Escolar (School Management Support, or AGE) program in four states shows a marked improvement in test scores for double-dose AGE schools, between 2007 and 2010, especially for indigenous schools, and in repetition and failure for some grades. Dropout rates fell by more than 1.5 percent and students moved ahead by about a year in reading and math. At the school level, 3rd grade test scores in Spanish and math increased significantly. At the individual student level there are positive effects, especially among third, fifth and sixth grades.
The results of the studies led to program changes in both cases. In the case of Colima, a more focused school policy was adopted, while nationally the school management program was enhanced and better targeted.
Evidence on the use of contract teachers suggests large efficiency gains in student learning outcomes, though caution at the sustainability of such an approach is recommended. On pay-for-performance reforms the most recent and robust developing-country evidence suggests that bonus-pay incentives can improve learning outcomes. Nevertheless, much more effort to link reforms is needed.
- In Kenya, The Extra Teacher Program (ETP) provided funding to a randomly selected group of schools to allow their school committees to hire local contract teachers from 2005 to 2007. Researchers found that contract teachers were significantly more likely to be in class and teaching 74 percent of the time versus 59 percent of the time for civil service teachers. Students randomly assigned to contract teachers under the ETP program scored a 0.21 standard deviation higher on reading and math tests than their schoolmates assigned to civil service teachers, suggesting higher—or at least more effective—teaching effort by the contract teachers despite their lower salaries. However, the fact that contract teachers were assigned to a single class of students and stayed with those students for two successive years (unlike the typical pattern of a different teacher each year) may have played a role, too.
Bank Contribution
The World Bank is drawing evidence from rigorous evaluations funded by country budgets, research support budget and especially from our development partners. Trust funds financed by the governments of the Netherlands, Spain, the United Kingdom and the Education Program Development Fund have provided funding of more than US$10 million over the past six years.
Partners
The Bank is working with many partners and donors to finance impact evaluations. For example, since 2007, the Spanish-World Bank Trust Fund for Impact Evaluation and Results-Based Management in Human Development Sectors has invested more than US$13 million in impact evaluations around the world. The Bank has complemented its resources and staff technical expertise with contributions from other donors. Together, the World Bank and donor investments in education impact evaluations represent 86 percent of the total cost of these impact evaluations, with the rest financed by governments.
Bank staff have worked with many partners on these impact evaluations, including governments of countries where evaluations are being carried out; donor-country governments, most notably through the multi-donor Global Partnership for Education (formerly known as the Education for All – Fast Track Initiative), the Bank-Netherlands Partnership Program, the Spanish Impact Evaluation Fund, and with additional support from Finland, Norway and the United Kingdom. The Bank has drawn on evaluation expertise from many universities and research institutions in both developed and developing countries, and has built impact evaluation capacity through regional workshops. By increasing demand and providing experience for local researchers, the Bank has helped build domestic demand for high-quality impact evaluations as well as capacity to carry out such evaluations.
Moving Forward
The Bank continues to promote evidence-based policymaking, deepen the knowledge base through impact evaluations, and synthesize results for use beyond evaluated projects. The development of rigorous impact evaluations marks a permanent shift in how the development community does business in the education sector. Improving the lives of children around the world and achieving learning for all is not possible through financing alone; it requires the multiplier effects of knowledge at the cornerstone of all interventions.
1Brazil, India, Indonesia, Kenya, Liberia, Mali, Mexico, Nepal, Pakistan, Peru and Uganda.
