This report is a self-evaluation of the Independent Evaluation Group's (IEG) Country Assistance Evaluations (CAEs). CAEs examine World Bank performance in a particular country, usually over the past four to five years, and report on its conformity with the relevant Bank Country Assistance Strategy (CAS) and on the overall effectiveness of the specific CAS. This Retrospective addresses the question "What Have We Learned?" by compiling lessons relevant for developing country assistance strategies from the most recent batch of CAEs. Second, it assesses revisions to the CAE process, methodology, and presentation to answer the question "How Can the CAE Instrument Be Improved?"

What Have We Learned?

The first part of this Retrospective summarizes some major findings and lessons from the 25 CAEs completed during FY01-03, complemented by additional insights gained from CAEs completed in FY04. Overall the lessons suggest that the Bank needs to understand better the country context, deepen its country knowledge, and better calibrate its assistance programs to observable progress on reform implementation.

This Retrospective finds that an evaluation at the country level yields a more complete picture of the outcome of the Bank's assistance programs than do evaluations of individual programs or projects. In one-third of the country programs evaluated, aggregate outcomes from projects were satisfactory, but the country assistance program was unsuccessful overall. This indicates the importance of assessing the country program in aggregate through a CAE. Otherwise, evaluation may not capture critical omissions in the country assistance programs, such as the fact that project interventions may not be addressing key development constraints.

The Retrospective also notes that the Bank's success varied by sector. The outcomes in health and education have been relatively successful. But, in private sector development, public sector management, and rural development, the Bank's efforts have been less successful overall. Factors responsible for the unsuccessful outcomes in these sectors are: (a) reforms in these sectors seem to face stronger opposition from vested interests and potential losers; (b) institutional capacity constraints take time to resolve and many projects in these sectors attempt to improve upon the legal, institutional, and regulatory framework, but implementing change requires overcoming inertia and adverse incentives in the bureaucracies of many countries; and (c) these sectors also suffer larger adverse effects from exogenous events and macroeconomic shocks. Improving outcomes in all sectors would imply focusing more on measuring and supporting results-based indicators.

The Retrospective has developed a number of lessons for improving the formulation and implementation of the Bank's country assistance strategies. First, a key component of successful country programs is that they are tailored to the country context and an understanding of the political economy of reform is essential. Domestic politics and vested interests largely determine the pace and content of reforms in countries. The Retrospective found a number of cases where an insufficient understanding of the political economy of reforms led the Bank to push reforms that stood little chance of success. More active dialogue with parliaments, local government, and stakeholders enhances the Bank's understanding of political economy considerations.

Second, country knowledge is strongly associated with success. It is well understood that project interventions are more successful when they are based on in-depth analytical work. The same finding conveys to the success or failure of the country programs. The Retrospective finds that in more than two-thirds of successful country programs, the Bank's analytical work was timely. Analytical and advisory activities can also be an effective vehicle for engaging governments in policy dialogue and informing civil society, but adequate attention needs to be paid to dissemination. In many cases, the attention paid to dissemination has been inadequate. This failing can also be a feature of participatory analytical work, as findings still may not be widely shared outside the government ministries who collaborated in the analytical work with the Bank.

Third, a number of CAEs provide clear evidence of the role technical assistance and investment loans can play in promoting institutional development and capacity-building. But the sustainability of benefits requires that these operations are part of a broader macro stabilization and reform program. Linking technical assistance and investment loans with policy reforms supported by adjustment loans also improves the probability of success.

Fourth, adjustment lending can be successful, especially when combined with a strong government commitment to macro stabilization and structural reform; however, adjustment lending in the absence of sustained progress on stabilization and reform saddles the country with debt and weakens the incentive to reform. The Bank needs to resist pressures to persist with adjustment lending in the absence of government commitment to, and a satisfactory track record in implementing reforms. The evidence from IEG 's evaluations of country assistance programs, however, provides only limited proof that the Bank resisted pressures. In many countries, accommodating pressure to lend resulted in a number of unsatisfactory outcomes and more importantly, weakened the incentive to reform.

Finally, the Retrospective finds several strategies that would serve to improve the outcomes of Bank assistance programs:

Undertaking more robust risk analysis to carefully assess borrower commitment to reform and implementation capacity. This should be informed by the analysis undertaken by others and by the feedback obtained through wider dialogue with stakeholders.

Reducing the level of planned assistance when faced with clear evidence of policy slippage. The Bank needs to set clear and meaningful triggers for its assistance programs and when confronted with slippage, cut back its level of assistance.

Lending more prudently in turn-around situations. This is especially the case in the presence of longstanding issues of implementation failures. Levels of assistance should be initially prudent and calibrated to measurable outcomes and meeting concrete benchmarks. The outcomes of Bank lending are better when lending goes up the ladder with the reform program, not ahead of it.

Generalizing these points, optimistic projections or expectations with inadequate risk analysis often weaken the performance of country strategies. Programs should not be based on the best possible of forecasted outcomes. And, finally, country strategies need to be flexible, not rigid and narrow with only one path.

How Can the CAE Instrument Be Improved?

The primary purpose of an IEG CAE is to draw lessons from the Bank's past activities to guide future strategies. The first part of the Retrospective synthesized a number of useful lessons for Bank country assistance strategies. These lessons, which accord well with the findings of other IEG evaluations, illustrate the usefulness of the CAE instrument. Nevertheless, a country assistance evaluation is a complex task, and while many multilateral and bilateral agencies now undertake country program evaluations, there is no 'standard' methodological approach. This second part of the CAE Retrospective analyzes IEG 's current methodology and perceptions of the strengths and weaknesses of CAEs with a view to making further improvements in the methodology, evaluation process, and structure of CAEs.

IEG formalized its methodology for CAEs in FY00 and the CAE approach became more standardized and consistent. The existing methodology starts with an evaluation of the relevance of the Bank's assistance strategy given the country's development needs and challenges, examining, among others, whether any key development constraints were omitted. It then evaluates the assistance program across three dimensions. First, a bottom-up review of the Bank's products and services (lending, analytical and advisory activities, and partnerships), which are used to achieve Bank objectives. Second, a top-down review of whether the Bank's program achieved its objectives or planned outcomes and had a substantive impact on the country's development. The first and second steps test the consistency of findings from the product and services and the development impact dimensions. Finally, the CAE attributes responsibility for the country strategy outcomes to the Bank, Borrower, other donors, and exogenous events.

For the Retrospective, IEG undertook a number of usage surveys. All objective measures - a tracer study of CAE recommendations, discussion of CAE findings and recommendations in subsequent CASs, and Board discussions of subsequent CASs - indicated that CAEs are frequently referred to and their recommendations figure to a large extent in the Bank's country assistance strategies. Members of CODE, who were interviewed for this Retrospective, felt that CAEs were useful, but were skeptical that Bank staff were using the CAE findings to the maximum extent. This was reflected in Bank staff views, which were on the whole much less positive, but were more positive when they perceived that a CAE had had an impact on the Bank's strategy. The conclusion reached in the Retrospective is that the CAEs are playing a useful role, but revisions to the methodology, rating system, and interactions with involved parties would lead to a high payoff in terms of acceptance and utilization.

IEG 's own assessment of the CAE methodology and the findings of an independent external evaluator used for this study concluded that the methodological framework, while fundamentally sound, had several shortcomings. First, the structure tends to place too much emphasis on the Bank's instruments rather than on achievement of the program's objectives. Second, different sections of the CAE can appear to lead to different conclusions. For example, the outcome rating of the overall assistance program may deviate from the aggregate of project ratings. While this apparent discrepancy reflects the more comprehensive nature of a CAE, it can appear as an inconsistency to CAE users. However, this deviation, combined with the current structure of the CAEs, led to questions about the consistency or clarity of the methodology. Finally, the organization of the CAE tends to shift focus from the country to the Bank back to the country, which makes it hard to follow the story line. IEG 's assessment is also consistent with the perceptions of CODE members who felt that CAEs focused too much on formal compliance issues and the impact of Bank instruments to the neglect of the Bank's development impact and the results achieved.

This Retrospective concludes that the way forward is the adoption of a results-based approach, which clearly links the Bank's objectives, instruments, and the outcomes achieved. The approach would be to outline the key objectives and results that the Bank is trying to achieve in each country, based on the CASs. The strategy discussion would be based on a simple logic model that links inputs with expected results based on the information presented in the CAS itself and related documents. For each objective (and intended result), the CAE would evaluate the efficacy (and relevance) of the instruments used, as well as the relative contribution of the Bank (as opposed to other contributions) to that objective. This approach would address the Board members' main recommendation that the CAEs should focus on the actual impact at the country level.

CAEs currently contain ratings of the outcome of Bank assistance, its sustainability, and institutional development impact. The CAE ratings have been the subject of considerable debate and criticism, which to some extent reflects the longstanding debate within the evaluation community on learning versus accountability. CODE members in general favored the ratings, arguing that they held the greatest potential for ensuring management attention to the CAE results. The majority of Bank staff were critical of the ratings, at times citing apparent inconsistencies between the ratings and the narrative, or, in other instances, citing a failure to rate different periods differently. But the key issue that arose is that many readers incorrectly perceive the outcome rating as either a rating of the country's development performance or as a rating of the Bank's performance. These distinctions, while clear in the methodology, still continue to confuse readers and more needs to be done to make clear the nature and basis of the CAE ratings.

The timing of the CAE is crucial to its usefulness. Bank staff prefer a CAE early in the CAS process and CODE members prefer a CODE discussion close to the discussion of the relevant CAS. This Retrospective found that the large majority of CAEs had been timed to appropriately inform CAS preparation and discussions. IEG 's review of the timing of the CAEs completed during FY01–03 indicates that more than 50 percent of the CAEs for which there have been subsequent CAS Board discussions had CODE discussions within six months of the CAS and two-thirds within a year. Moreover, despite the perception of a lack of timeliness by Bank staff, all 22 of the CAEs discussed by CODE before Board discussion of the CAS were mentioned in those CASs. IEG will continue its policy of delivering CAEs to the Board and Bank staff around three to nine months ahead of planned CAS discussions, but notes that the unpredictable scheduling of CAS discussions is a major challenge in meeting this performance target.

Finally, there was a widespread perception that more interactions with country teams during CAE preparation and with in-country beneficiaries would enhance both the quality of the CAEs, as well as the acceptance of its findings. This Retrospective finds that, while there have been many more interactions with in-country beneficiaries than perceptions indicate, IEG will make an effort to ensure more uniformly that in-country beneficiaries are consulted and their views incorporated. Similarly, the Retrospective also finds that a more systematic interaction with country teams during the process of preparing the evaluation may help clarify and improve the acceptance and incorporation of CAE findings in country programs. Therefore, IEG proposes to involve the Country Team in mid-term reviews of future CAEs, to receive early feedback on emerging findings and recommendations.