|
IEG's
evaluation tools used in promoting accountability and learning
are Project Reviews, Country Assistance Evaluations, Sector
and Thematic Reviews and Process Reviews. A brief description
of each of these tools and the total number of products
per year is provided in this section.
Project
Reviews
Projects
are selected for performance assessments based on a variety
of criteria. First, operational staff prepare a self-evaluation
(known as an Implementation Completion Report, or ICR) for
every completed project, and also rate project performance. IEG staff review every ICR, validate the self-rating, and
identify projects that offer good potential for further
learning (because of particularly good or bad performance)
as candidates for a project performance assessment (PPAs).
Second, projects related to sectors, thematic areas, or
countries that are soon to be evaluated are attractive candidates
for project performance assessments because they can be
inputs for those evaluation tools. Third, projects selected
for project performance assessments are clustered in order
to reduce their cost and increase their impact.
One
in four completed projects (or about 70 a year) is subject
to a Project Performance Assessment Report, which takes
about six staff weeks to produce and normally includes a
field mission. Project Performance Assessment Reports (PPAs),
rate projects in terms of their outcome (taking into account
relevance, efficacy, and efficiency), sustainability of
results, and institutional development impact. PPAs carried
out after Bank funds have been fully disbursed to a project,
are similar to the completion evaluations carried out by
many development agencies, and are the main project-level
evaluations conducted by IEG . They are products in themselves
but are also intermediate inputs-building blocks-for the
other three kinds of evaluations.
Country
Assistance Evaluations (CAEs)
Country
Assistance Evaluations (approximately 10 each year), examine
Bank performance in a particular country, usually over the
past four to five years, and report on its conformity with
the relevant Bank Country Assistance Strategy (CAS) and
on the overall effectiveness of the specific CAS.
Sector
and Thematic Reviews
Sector
and Thematic Reviews (about 6 each year) examine Bank performance
and experience in a lending sector (such as agriculture,
transport, and the like) or thematic area (poverty, gender,
and so forth) over five to ten years and report on their
conformity to Bank policy and good practice, as well as
on the development effectiveness of the Bank's activities.
To maximize
the impact of the sector and thematic as well as the country
evaluations, their delivery is scheduled upstream of the
revision of either a Sector Strategy Paper (SSP) or a CAS.
The flow of revised SSPs is scheduled up to three years
in advance, and CASs, up to two years in advance. IEG sector
and thematic reviews are produced from six months to a year
before the SSP is finalized, so that the revised SSP can
incorporate lessons or recommendations from the IEG review,
which in turn is discussed by the Board's Committee on Development
Effectiveness (CODE) when the review is complete.
Process
Reviews
Process
Reviews (2 or 3 per year) examine ongoing activities such
as aid coordination or development grant-making and report
on their overall efficiency, consistency with stated objectives,
and effectiveness. Process evaluations are produced in response
to a Board request or as an answer to other demands. IEG 's
review of Aid Coordination grew out of discussions at the
Development Assistance Committee (DAC) about harmonization
of donor procedures. The review of Poverty Assessments was
a follow-on to an earlier IEG evaluation of Poverty Assessments,
and Board members requested that the follow-up study pay
special attention to the views of policymakers in developing
countries.
IEG
also prepares the Annual Review of Operations Evaluation
(AROE), which reviews the state of evaluation processes
at the Bank. It also produces the Annual Review of Development
Effectiveness (ARDE), a meta-evaluation that provides
a comprehensive assessment of the Bank's development effectiveness.
The ARDE draws on the evaluation work done during the year
to ascertain trends in the Bank's operational performance
and to review experience gained in special topical areas.
IEG's
evaluation processes include interaction with Bank staff
not only in planning evaluations, so as to ensure that their
delivery will be timely in the context of the decision-making
process, but also during the evaluations. Such interactions
with staff are important-they provide Bank staff with the
opportunity to discuss IEG interpretations of documents
they prepared. Although IEG is independent, it is not isolated.
It does engage in an active dialogue with Bank management,
Bank staff and beneficiaries of Bank interventions. Impact Evaluations
Impact evaluation is the systematic identification of the effects positive or negative,
intended or not on individual households, institutions, and the environment caused by
a given development activity such as a program or project. It is a type of evaluation
which has received increasing attention in recent years.
Development interventions such as a project or program can be conceptualized as having
a results chain ― from the intervention’s inputs, leading to its immediate outputs, and
then to outcomes and final impacts. Evaluations can focus on all or part of this results
chain. Thus, evaluations which focus on efficiency, for example, are concerned with the
relationship between inputs and outputs. Evaluations which include an analysis of final
impacts are termed impact evaluations. These focus on the effects ― positive or
negative, intended or not ― of an intervention on individuals, institutions, and the
environment. Impact can be conceptualized as the difference between what happened
with the project or program and the situation if the intervention had not been made, i.e.,
the counterfactual situation.
There are several methods or models of impact evaluation:
• Rapid assessment or review, conducted ex post. This method can encompass a
range of approaches to endeavor to assess impact, such as participatory methods,
interviews, focus groups, case studies, an analysis of beneficiaries affected by the
project, and available secondary data;
• Ex-post comparison of project beneficiaries with a control group. With this method,
multivariate analysis may be used to control statistically for differences in attributes
between the two groups ― this is one way of estimating the counterfactual
situation;
• Quasi-experimental design, involving the use of matched control and project (beneficiary) groups. This method involves the use of a “non-equivalent” control group to match as closely as possible the characteristics of the project population – either through propensity score matching or using a multivariate regression approach. This method often involves the use of large scale sample surveys, and sophisticated statistical analysis; and
• Randomized design. This involves the random assignment of individuals or households either as project beneficiaries, or as a control group which does not receive the service or good being provided by the project. This is also known as the experimental method, and is used in health research, for example, in areas such as
evaluating the effectiveness of new drugs and medical procedures.
|
