|
 Countries
such as Brazil have stressed a whole-of-government
approach to the setting of program objectives
and the creation of a system of performance
indicators. Others such as Colombia are
combining this with an agenda of rigorous
impact evaluations. Yet others, such as
Australia, the United States and the United
Kingdom, have stressed a broader suite of
M&E tools and methods: including performance
indicators, rapid reviews, impact evaluations
and performance audits. Some countries have
succeeded in building a whole-of-government
M&E system, while others such as Uganda
comprise an as yet uncoordinated and disparate
collection of about 16 separate sectoral
monitoring systems. And the poorest countries
-- those which are required by multilateral
donors to prepare Poverty Reduction Strategies
-- stress the regular collection of performance
indicators to measure the millennium development
goals (MDGs).
This tells us that not only are the starting
points faced by each country different,
but so are the destinations to which they
aspire. There is no single, "best
approach" to a national or sectoral
M&E system. Instead, it all depends
on the actual or intended uses of the information
which such a system will produce: whether
to assist resource-allocation decisions
in the budget process; to help in preparation
of national and sectoral planning; to aid
ongoing management and delivery of government
services; or to underpin accountability
relationships.
It is clearly important to tailor efforts
to build or strengthen government M$E
systems to the needs and priorities of each
country. This underscores the desirability
of conducting a baseline diagnosis of M$E
activities to guide the identification of
opportunities for institutionalizing M&E;
this also provides a yardstick for measuring
a country's progress over time, recognizing
that it is a long-haul effort to build and
sustain both demand and supply for M&E.
A national diagnosis of M&E would be
expected to map out a number of key issues.
These include the nature of existing demand
for M&E and the nature of current M&E
activities and functions within government,
the actual and potential supply of M&E
capacities in the country, the extent of
M&E utilization, and both the actual
and potential synergies between M&E
and public sector reforms in areas such
as a poverty reduction strategy, the budget
system, strengthening policy analysis skills,
creation of a performance culture in the
civil service, improvements in service delivery,
government decentralization, and greater participation
by civil society. A list of the key dimensions
of a government M&E system is presented
in the following table.
Key Issues for a Diagnosis of a Government's M&E Systems
1. Genesis of the existing M&E system: role of M&E advocates or champions; key events which created the priority for M&E information (e.g., election of reform-oriented government, fiscal crisis).
2. The ministry or agency responsible for managing the M&E system, and for planning evaluations. Roles and responsibilities of the main parties to the M&E system — e.g., finance ministry, planning ministry, president's office, sector ministries, parliament or congress. Incentives for the stakeholders to take M&E seriously — strength of demand for M&E information. Possible existence of several, uncoordinated M&E systems, at the national and sectoral levels. Importance of federal/state/local issues to the M&E system.
3.The public sector environment and whether it makes it easy or difficult for managers to perform to high standards, and to be held accountable for their performance. Are public sector reforms underway which might benefit from a stronger emphasis on the measurement of government performance, such as a poverty reduction strategy, performance budgeting, strengthening policy analysis skills, creation of a performance culture in the civil service, improvements in service delivery such as customer service standards, government decentralization, greater participation by civil society, or an anti-corruption strategy?
4. The main aspects of public sector management which the M&E system supports strongly, such as: (i) budget decision-making; (ii) national or sectoral planning; (iii) program management; (iv) accountability relationships (to the finance ministry, to the President's office, to parliament, to sector ministries, to civil society).
5. Actual role of M&E information at the various stages of the budget process — such as policy advising and planning; budget decision-making; performance review and reporting. Possible disconnect between the M&E work of sector ministries and the use of such information in the budget process. Existence of any disconnect between the budget process and national planning. Opportunities to strengthen the role of M&E in the budget.
6. Extent to which the M&E information commissioned by key stakeholders (e.g. the finance ministry) is used by others, such as sector ministries. If not, what are the barriers to utilization? Any solid evidence concerning the extent of utilization by different stakeholders (e.g., a diagnostic review or a survey). Examples of major evaluations which have been highly influential with the government.
7. Types of M&E tool which are emphasized in the M&E system: regular performance indicators; rapid reviews or evaluations; performance audits; rigorous, in-depth impact evaluations; other. Scale and cost of each of these types of M&E. Manner in which evaluation priorities are set — are they focused on 'problem programs', pilot programs, high-expenditure or high-visibility programs, or are they based on a systematic research agenda to answer questions about program effectiveness?
8. Who is responsible for collecting performance information, and for conducting evaluations (e.g., ministries themselves, or academia or consulting firms)? Any problems with data quality or reliability, or with the quality of evaluations which have been conducted. Strengths and weaknesses of local supply of M&E. Key capacity constraints and the government's capacity-building priorities.
9. Extent of donor support for M&E in recent years. Donor projects which support M&E at whole-of-government, sectoral or agency levels — provision of technical assistance, other capacity-building and funding for the conduct of major evaluations, such as rigorous impact evaluations.
10. Conclusions: overall strengths and weaknesses of the M&E system. Its sustainability, in terms for example of vulnerability to a change in government. How dependent is it on donor funding or other support? Current plans for future strengthening of the M&E system. |
A Diagnostic Guide and Action Framework.
This guide provides detailed checklists
of issues to consider in conducting a diagnosis.
It also presents a simple taxonomy for classifying
countries, according to whether their demand
for and supply of M&E is strong/weak.
It is also available in French and in Spanish.
A Diagnosis of Colombia's National M&E System, SINERGIA exemplifies this diagnostic framework. It identifies the strengths and weaknesses of the M&E system, and identifies a number of actions to further institutionalize it and to ensure its future sustainability. (Spanish Version | French Version)
See also Good Practice Government Systems for M&E: the Cases of Chile and Colombia >>Video presentation>>
Strengthening Capacity for Monitoring and
Evaluation in Uganda: A Results Based Management
Perspective provides an example of the application
of this diagnostic framework. This paper
is also available in Spanish. This paper contains a taxonomy for comparing the M&E systems for different countries, focusing on dimensions such as: main objectives and intended uses; nature of their legal framework; roles and responsibilities; extent of coordination, and reporting arrangements:
Experience with Institutionalizing Monitoring and Evaluation Systems in Five Latin American Countries: Argentina, Chile, Colombia, Costa Rica and Uruguay
A list of key M&E issues to consider
concerning M&E systems in poor countries
which prepared poverty reduction strategy
papers (PRSPs) is provided in:
2002
Annual Report on Evaluation Capacity Development,
Annex F
Another diagnostic guide relates to universities
and other organizations which provide M&E
training or trainer-training -- such training
is a common element of action plans to help
institutionalize M&E:
Guide
to Conducting Reviews of Organizations Supplying
M&E Training
Assessment
of Ghanaian Organizations Which Provide
M&E Training provides one application
of this diagnostic guide.
See Developing Action Plans and
IEG’s Evaluation Training websites
There is merit in undertaking an M&E
diagnosis -- which in itself is a form of
evaluation -- not just at the start of efforts
to institutionalize M&E but as the process
of institutionalization is underway, in
order to make needed mid-course corrections.
Most countries have not developed their
M&E systems in a linear, predictable
manner, but rather have developed them opportunistically
depending on emerging opportunities and
as they develop their understanding concerning
which initiatives are or are not working
well. See also Lessons from Experience
A final question which is often asked is
how long should it take to conduct an M&E
diagnosis? -- how much time and effort should
be spent in preparing one? There is no simple
answer to this question; it all depends
on the purposes for which a diagnosis is
intended, the range of issues which are
judged sufficiently important to investigate,
and the available time and budget. In some
cases a week-long mission to a country has
provided a sufficient starting point for
a broad understanding of the key issues
facing a government which is interested
in strengthening its M&E functions.
At the other end of the spectrum is a more
formal, detailed and in-depth evaluation
of a government evaluation system, such
as the one which the Chilean government
commissioned the World Bank to undertake.
Such in-depth evaluations can cost several
hundred thousand dollars, depending on their
breadth and depth. See, for example:
Evaluation of Chile's Evaluation System
and Annexes (Spanish)
|
