1. By end FY03, IEG had issued Country Assistance Evaluations (CAEs) assessing the outcomes, sustainability and institutional development impact of Bank assistance to 58 countries. These evaluations cover roughly 41 percent of borrowers and 65 percent of Bank lending and non-lending activities.

2. This methodological note describes the key elements of IEG 's country assistance evaluation (CAE) methodology. (1)

CAEs rate the outcomes of Bank assistance programs, not Clients' overall development progress

3. An assistance program needs to be assessed on how well it met its particular objectives, which are typically a sub-set of the Client's development objectives. If an assistance program is large in relation to the Client's total development effort, the program outcome will be similar to the Client's overall development progress. However, most Bank assistance programs provide only a fraction of the total resources devoted to a Client's development by donors, stakeholders, and the government itself. In CAEs, IEG rates only the outcome of the Bank's program, not the Client's overall development outcome, although the latter is clearly relevant for judging the program's outcome.

4. The experience gained in CAEs confirms that program outcomes sometimes diverge significantly from the Client's overall development progress. CAEs have identified assistance programs which had:

• satisfactory outcomes matched by good Client development;
  
  
• unsatisfactory outcomes in Clients which achieved good overall development results, notwithstanding the weak Bank program; and,
  
  
• satisfactory outcomes in Clients which did not achieve satisfactory overall results during the period of program implementation.

Assessments of assistance program outcome and Bank performance are not the same

5. By the same token, an unsatisfactory assistance program outcome does not always mean that Bank performance was also unsatisfactory, and vice-versa. This becomes clearer once we consider that the Bank's contribution to the outcome of its assistance program is only part of the story. The assistance program's outcome is determined by the joint impact of four agents: (a) the Client; (b) the Bank; (c) partners and other stakeholders; and (d) exogenous forces (e.g., events of nature, international economic shocks, etc.). Under the right circumstances, a negative contribution from any one agent might overwhelm the positive contributions from the other three, and lead to an unsatisfactory outcome.

6. IEG measures Bank performance primarily on the basis of contributory actions the Bank directly controlled. Judgments regarding Bank performance typically consider the relevance and implementation of the strategy, the design and supervision of the Bank's lending interventions, the scope, quality and follow-up of diagnostic work and other AAA activities, the consistency of Bank's lending with its non-lending work and with its safeguard policies, and the Bank's partnership activities.

Evaluation in Three Dimensions

7. As a check upon the inherent subjectivity of ratings, IEG examines a number of elements that contribute to assistance program outcomes. The consistency of ratings is further tested by examining the country assistance program across three dimensions:

(a) a Products and Services Dimension, involving a "bottom-up" analysis of major program inputs -- loans, AAA, and aid coordination;

(b) a Development Impact Dimension, involving a "top-down" analysis of the principal program objectives for relevance, efficacy, outcome, sustainability, and institutional impact; and,

(c) an Attribution Dimension, in which the evaluator assigns responsibility for the program outcome to the four categories of actors (see paragraph 5. above).

Rating Assistance Program Outcome

8. In rating the outcome (expected development impact) of an assistance program, IEG gauges the extent to which major strategic objectives were relevant and achieved, without any shortcomings. Programs typically express their goals in terms of higher-order objectives, such as poverty reduction. The country assistance strategy (CAS) may also establish intermediate goals, such as improved targeting of social services or promotion of integrated rural development, and specify how they are expected to contribute toward achieving the higher-order objective. IEG 's task is then to validate whether the intermediate objectives produced satisfactory net benefits, and whether the results chain specified in the CAS was valid. Where causal linkages were not fully specified in the CAS, it is the evaluator's task to reconstruct this causal chain from the available evidence, and assess relevance, efficacy, and outcome with reference to the intermediate and higher-order objectives.

9. Evaluators also assess the degree of Client ownership of international development priorities, such as the Millennium Development Goals, and Bank corporate advocacy priorities, such as safeguards. Ideally, any differences on dealing with these issues would be identified and resolved by the CAS, enabling the evaluator to focus on whether the trade-offs adopted were appropriate. However, in other instances, the strategy may be found to have glossed over certain conflicts, or avoided addressing key Client development constraints. In either case, the consequences could include a diminution of program relevance, a loss of Client ownership, and/or unwelcome side-effects, such as safeguard violations, all of which must be taken into account in judging program outcome.

Ratings Scale

10. IEG utilizes six rating categories for outcome, ranging from highly satisfactory to highly unsatisfactory:

Highly Satisfactory:	The assistance program achieved at least acceptable progress toward all major relevant objectives, and had best practice development impact on one or more of them. No major shortcomings were identified.
Satisfactory:	The assistance program achieved acceptable progress toward all major relevant objectives. No best practice achievements or major shortcomings were identified.
Moderately Satisfactory:	The assistance program achieved acceptable progress toward most of its major relevant objectives. No major shortcomings were identified.
Moderately Unsatisfactory:	The assistance program did not make acceptable progress toward most of its major relevant objectives, or made acceptable progress on all of them, but either (a) did not take into adequate account a key development constraint or (b) produced a major shortcoming, such as a safeguard violation.
Unsatisfactory:	The assistance program did not make acceptable progress toward most of its major relevant objectives, and either (a) did not take into adequate account a key development constraint or (b) produced a major shortcoming, such as a safeguard violation.
Highly Unsatisfactory:	The assistance program did not make acceptable progress toward any of its major relevant objectives and did not take into adequate account a key development constraint, while also producing at least one major shortcoming, such as a safeguard violation.

11. The institutional development impact (IDI) can be rated as: high, substantial, modest, or negligible. IDI measures the extent to which the program bolstered the Client's ability to make more efficient, equitable and sustainable use of its human, financial, and natural resources. Examples of areas included in judging the institutional development impact of the program are:

• the soundness of economic management;
  
  
• the structure of the public sector, and, in particular, the civil service;
  
  
• the institutional soundness of the financial sector;
  
  
• the soundness of legal, regulatory, and judicial systems;
  
  
• the extent of monitoring and evaluation systems;
  
  
• the effectiveness of aid coordination;
  
  
• the degree of financial accountability;
  
  
• the extent of building NGO capacity; and,
  
  
• the level of social and environmental capital.

12. Sustainability can be rated as highly likely, likely, unlikely, highly unlikely, or, if available information is insufficient, non-evaluable. Sustainability measures the resilience to risk of the development benefits of the country assistance program over time, taking into account eight factors:

• technical resilience;
  
  
• financial resilience (including policies on cost recovery);
  
  
• economic resilience;
  
  
• social support (including conditions subject to safeguard policies);
  
  
• environmental resilience;
  
  
• ownership by governments and other key stakeholders;
  
  
• institutional support (including a supportive legal/regulatory framework, and organizational and management effectiveness); and,
  
  
• resilience to exogenous effects, such as international economic shocks or changes in the political and security environments.