This methodological note describes the key elements of IEG’s country assistance evaluation (CAE) methodology. (1)

CAEs rate the outcomes of Bank assistance programs, not the Clients’ overall development progress

A Bank assistance program needs to be assessed on how well it met its particular objectives, which are typically a sub-set of the Client’s development objectives. If a Bank assistance program is large in relation to the Client’s total development effort, the program outcome will be similar to the Client’s overall development progress. However, most Bank assistance programs provide only a fraction of the total resources devoted to a Client’s development by donors, stakeholders, and the government itself. In CAEs, IEG rates only the outcome of the Bank’s program, not the Client’s overall development outcome, although the latter is clearly relevant for judging the program’s outcome.

The experience gained in CAEs confirms that Bank program outcomes sometimes diverge significantly from the Client’s overall development progress. CAEs have identified Bank assistance programs which had:

• satisfactory outcomes matched by good Client development;
• unsatisfactory outcomes in Clients which achieved good overall development results, notwithstanding the weak Bank program; and,
• satisfactory outcomes in Clients which did not achieve satisfactory overall results during the period of program implementation.

Assessments of assistance program outcome and Bank performance are not the same

By the same token, an unsatisfactory Bank assistance program outcome does not always mean that Bank performance was also unsatisfactory, and vice-versa. This becomes clearer once we consider that the Bank's contribution to the outcome of its assistance program is only part of the story. The assistance program’s outcome is determined by the joint impact of four agents: (a) the Client; (b) the Bank; (c) partners and other stakeholders; and (d) exogenous forces (e.g., events of nature, international economic shocks, etc.). Under the right circumstances, a negative contribution from any one agent might overwhelm the positive contributions from the other three, and lead to an unsatisfactory outcome.

IEG measures Bank performance primarily on the basis of contributory actions the Bank directly controlled. Judgments regarding Bank performance typically consider the relevance and implementation of the strategy, the design and supervision of the Bank’s lending interventions, the scope, quality and follow-up of diagnostic work and other AAA activities, the consistency of the Bank’s lending with its non-lending work and with its safeguard policies, and the Bank’s partnership activities.

Rating Assistance Program Outcome

In rating the outcome (expected development impact) of an assistance program, IEG gauges the extent to which major strategic objectives were relevant and achieved, without any shortcomings. In other words, did the Bank do the right thing, and did it do it right. Programs typically express their goals in terms of higher-order objectives, such as poverty reduction. The country assistance strategy (CAS) may also establish intermediate goals, such as improved targeting of social services or promotion of integrated rural development, and specify how they are expected to contribute toward achieving the higher-order objective. IEG’s task is then to validate whether the intermediate objectives were the right ones and whether they produced satisfactory net benefits, and whether the results chain specified in the CAS was valid. Where causal linkages were not fully specified in the CAS, it is the evaluator’s task to reconstruct this causal chain from the available evidence, and assess relevance, efficacy, and outcome with reference to the intermediate and higher-order objectives.

For each of the main objectives, the CAE evaluates the relevance of the objective, the relevance of the Bank’s strategy towards meeting the objective, including the balance between lending and non-lending instruments, the efficacy with which the strategy was implemented and the results achieved. This is done in two steps. The first is a top-down review of whether the Bank’s program achieved a particular Bank objective or planned outcome and had a substantive impact on the country’s development. The second step is a bottom-up review of the Bank’s products and services (lending, analytical and advisory services, and aid coordination) used to achieve the objective. Together these two steps test the consistency of findings from the products and services and the development impact dimensions. Subsequently, an assessment is made of the relative contribution to the results achieved by the Bank, other donors, the Government and exogenous factors.

Evaluators also assess the degree of Client ownership of international development priorities, such as the Millennium Development Goals, and Bank corporate advocacy priorities, such as safeguards. Ideally, any differences on dealing with these issues would be identified and resolved by the CAS, enabling the evaluator to focus on whether the trade-offs adopted were appropriate. However, in other instances, the strategy may be found to have glossed over certain conflicts, or avoided addressing key Client development constraints. In either case, the consequences could include a diminution of program relevance, a loss of Client ownership, and/or unwelcome side-effects, such as safeguard violations, all of which must be taken into account in judging program outcome.

Ratings Scale

IEG utilizes six rating categories for outcome, ranging from highly satisfactory to highly unsatisfactory:

The institutional development impact (IDI) can be rated as: high, substantial, modest, or negligible. IDI measures the extent to which the program bolstered the Client’s ability to make more efficient, equitable and sustainable use of its human, financial, and natural resources. Examples of areas included in judging the institutional development impact of the program are:

• the soundness of economic management;
• the structure of the public sector, and, in particular, the civil service;
• the institutional soundness of the financial sector;
• the soundness of legal, regulatory, and judicial systems;
• the extent of monitoring and evaluation systems;
• the effectiveness of aid coordination;
• the degree of financial accountability;
• the extent of building NGO capacity; and,
• the level of social and environmental capital.

Sustainability can be rated as highly likely, likely, unlikely, highly unlikely, or, if available information is insufficient, non-evaluable. Sustainability measures the resilience to risk of the development benefits of the country assistance program over time, taking into account eight factors:

• technical resilience;
• financial resilience (including policies on cost recovery);
• economic resilience;
• social support (including conditions subject to safeguard policies);
• environmental resilience;
• ownership by governments and other key stakeholders;
• institutional support (including a supportive legal/regulatory framework, and organizational and management effectiveness); and,
• resilience to exogenous effects, such as international economic shocks or changes in the political and security environments.

Footnote:

1 In this note, assistance program refers to products and services generated in support of the economic development of a Client country over a specified period of time, and client refers to the country that receives the benefits of that program.