Evaluation Framework for Phase 2 Report and 26 Case Studies

The Phase 2 report and the case studies follow a common outline and address 20 evaluation questions (table C.1) that derive from IEG 's standard evaluation criteria (table C.2), the 14 eligibility and approval criteria for global programs (table C.3), and the 8 eligibility criteria for DGF grant support (table C.4).

The sheer number of these criteria, some of which overlap, can be daunting even to an evaluator. The IEG evaluation team thus reorganized these criteria into four evaluation issues, which correspond to the four major sections of each report (table C.1):

• The overarching global relevance of the program
• Outcomes and impacts of the program and their sustainability
• Governance, management, and financing of the program
• The World Bank's performance as a partner in the program.

While program sponsorship by major international organizations may enhance the program's legitimacy in the Bank's client countries, it ensures neither developing-country ownership nor development effectiveness. "Relevance" and ownership by the Bank's client countries are more assured if those countries demand the program. On the other hand, some supply-led programs may acquire ownership over time by demonstrating substantial impacts, as in the case of the Internet. Assessing relevance is the most challenging task in global programs, since global and country resources, comparative advantages, benefit, costs, and priorities do not always coincide. Indeed, the divergence of benefits and costs between the global and country levels is often the fundamental reason why global public goods must be provided in the first place. Evaluating the relevance of global action to the Bank's client countries is nonetheless important, because the global development agenda is becoming crowded while finances have stagnated; selectivity has become more important.

For the global programs that have been operating for some time, efficacy can be assessed not only in terms of program outcomes but also in terms of impacts in developing countries. Outcomes and impacts, in turn, depend on the clarity and evaluability of each program's objectives, the quality of the monitoring and evaluation of results, and, where appropriate, the effectiveness of global program activities' links to the country level.

Since global programs are partnerships, efficiency must include some assessment of whether the benefits from the partnership, net of its costs, are superior to what the partners could achieve by acting alone. The institutional development impact and the sustainability of the program itself (as opposed to that of the outcomes and impacts of the program's activities) are also addressed in this section of each report.

Finally, these evaluations focus on whether the Bank uses its comparative advantage in its partnerships. The Bank is variously convener, trustee, and donor to global programs, and lender to developing countries. The Bank's financial support to global programs-including oversight and liaison activities and links to the Bank's Regional operations-comes from a combination of the its net income (for DGF grants), its administrative budget, and Bank-administered trust funds. In the case of the Global Environmental Facility (GEF), the Bank is a trustee; in the case of the Global Fund to Fight AIDS, Tuberculosis, and Malaria (GFATM), it is a "limited" trustee. The Bank is also an implementing agency for GEF and MLF. Thus, assessing Bank performance includes the use of the Bank's convening power, the Bank's trusteeship, Bank financing and implementation of global programs, and, where appropriate and necessary, links to country operations. Bank oversight of this entire set of activities is an important aspect of the Bank's strategic and programmatic management of its portfolio of global programs.

The first column in table C.1 indicates how the 4 sections and 20 evaluation questions addressed in the Phase 2 report and case studies relate to the 8 evaluation issues that the Bank's Executive Board raised in the various Board discussions of global programs during the design of IEG 's global evaluation:¹

• Selectivity
• Monitoring and evaluation
• Governance and management
• Partnerships and participation
• Financing
• Risks and risk management
• Links to country operations.

The third column in table C.1 indicates how the 4 sections and 20 evaluation questions relate to IEG 's standard evaluation criteria for investment projects (table C.2), the 14 criteria endorsed by the Development Committee and established by Bank management for approving the Bank's involvement in global programs (table C.3), and the 8 criteria for grant support from the Development Grant Facility (table C.4).

The 14 eligibility and approval criteria for the Bank's involvement in global programs have evolved since April 2000, when Bank management first proposed a strategy to the Board for such involvement. They include the four overarching criteria endorsed by the Development Committee, as well as the four eligibility criteria and the six approval criteria presented by Bank management to the Board. Each global program must meet at least one of the eligibility criteria and all six of the approval criteria. The first two eligibility criteria relate directly to the Bank's global public-goods and corporate-advocacy priorities (table C.3). Although the six approval criteria resemble the topics covered in a project concept or appraisal document for Bank lending operations, global programs need approval only at the concept stage (unlike lending operations, which also need approval at appraisal). New global programs need approval only from the managing director responsible for the network proposing the new program, not from the Board.

Program approval is logically separate from, and prior to, financing (whether from the DGF, trust funds, or other sources). The eight eligibility criteria for grant support from the DGF (table C.4) were actually established in 1998, although the processes of program approval, trust fund mobilization, and their relationship to the DGF have evolved considerably since then. Because each approval process and each set of criteria were developed independently, they are not always consistent with each other. Twenty of the 26 case studies and about two-thirds of the Bank's total portfolio of global programs have received DGF grants.

¹ IEG 2001a, p. 21. "Partnerships and participation" were originally listed as two separate evaluation issues in the evaluation strategy document. "Monitoring and evaluation" is now interpreted more broadly to include not only an assessment each program's monitoring and evaluation procedures but also the findings of previous evaluations about each program's outcomes and impacts, and their sustainability.

Suggested Appraisal Template for Global Programs