Global HIV/AIDS Monitoring and Evaluation Team in the MAP Projects (GAMET)

Introduction: Lead Monitoring and Evaluation Specialist Susan Stout, and Senior M&E Specialist Joseph Valadez, both members of the HDNGA Global HIV AIDS Program, share their reflections on their experience in the Global HIV Monitoring & Evaluation Support Team, GAMET. The GAMET Country Support Team (CST) consists of 11 international M&E specialists. The team made 85 field support visits to 30 countries and projects in Africa alone.

This innovative and high-profile program has been charged with developing strategies to use the principles and practice of M& E to help support programs in HIV mitigation and improvements. Their work has included the assistance of 25 countries to develop indicators and initiate M&E plans. Susan, Joe and their colleague David Wilson, who telecommutes from the Africa region, are really the ground-breakers in how to use Monitoring and Evaluation to support the overall effort to mitigate the problems associated with HIV/AIDS -- and note that their work was made possible through the vision and leadership of Dr. Debrework Zewdie, the Director of the Bank;s Global HIV/AIDS Program in the HDN Vice Presidency.

The video can be viewed in its entirety or via shorter clips sorted along the main themes (blue underlined), which emerged during the debriefing. The Knowledge and Learning Group organized this debriefing as part of AFTQK’s knowledge sharing and learning services. Jody Kusek conducted the de-briefing on March 02, 2004.

View full video (52:09 min), download mp3 audio

The Role of GAMET Promoting the use of M&E as a Reporting Tool

1.       Role of the GAMET Team is to learn how to use the principles and practices of Monitoring and Evaluation to help support programs in HIV mitigation. With the help of their country support team, the hard work of a team of consultants as well as David Wilson, the team has worked to develop strategies for ‘learning by doing’ in scaling up responses to HIV/AIDS (2:38 min).

2.     Partnership and Support of Key Bilaterals. The UNAIDS program was created through the ‘Joint Program on HIV/AIDS’, a partnership of nine agencies including the Bank. Through UNAIDS sponsorship, there was a lot of work among the agencies to come up with some unified guidance to countries on M&E (54 sec).

3.       Origins of the Program. The team was formed in July of 2002. Funding has taken place jointly by the Bank-and UNAIDS, however its primary source of funds is through UNAIDS (36 sec).

4.       The Key Word or Phrase is "Learn by Doing" This has clearly been recognized to be the method to achieve GAMET’s results–focus in progress against HIV/AIDS. “The only way to move forward in the field is to adopt this approach of learning by doing”, however the realization was reached that this needs to become a structured process, and one which is supported internally (5:00 min).

a.       Create a Structured Process for Learning by Doing UNAIDS and the Bank decided that creating a structured process for learning by doing in their work with HIV/AIDS would be best facilitated through a focus on the Bank and its position as lead donor. This would “put some ‘Oomph’ behind the interests of Monitoring & and Evaluation” (1:03 min).

b.       The Key to HIV/AIDS is Behavior Change. We know that the key to HIV/AIDS is changing behavior. However, what we don’t know about behavior change is much greater than what we do know. We have to do learning as we go (21 sec).

c. Participation is the Only way to Have a Contextual Understanding. If you haven’t done this, trying to inculcate without that experience what that means is a challenge. “The people who really get it, this learning by doing idea, using M&E information, are those who have actually participated in these exercises – it is because of participating in the exercises they do, that they have that understanding” (1:14 min).

d. Leverage Knowledge Through Learning by Doing is very transactional heavy to actually put people through it –how else can you leverage the knowledge that you have gained to be able to really leverage it? “help countries learn from each other. The most credible source of advice on how to design a national M&E system is going to come from another country who is designing an M&E system . you need top promote cross-country learning (2:21 min).

5.       Role of GAMET Coordinating and Harmonizing M&E Approaches. The role of the team was to pursue the harmonization and coordination goals of UNAIDS agencies and bilateral agencies such as the US government (56 sec).

6.      Why Don't Countries have Effective M&E Systems? There are several possible reasons. One is a consequence of their Institutional Administrative tradition. As an example, “the British and French civil service rules have not traditionally been performance based. The colonial administrative history did not seek a lot of results-measurement in normal ways of doing business” (35 sec).

7.       Aid the Managers to Steer and Guide their Own Programs. M&E is not for the manager to use to appease his critics, “It suits a different purpose. Once M&E actually gets into a form where it actually aids the manager to steer & guide their own programs so they can enhance their impact – then it becomes the catalyst for change in these organizations. That is what we are trying to do in GAMET, is to lay the foundation where M&E is understood to be a practical management tool rather than a reporting responsibility “ (43 sec).

a.       Use the Data to Improve Effectiveness, Not Just for Reporting The M&E Tradition in Development has been to ‘feed data’ to donor agencies and to donor agency owners. “Its been more about the extraction of information to report up and out of the country, rather than on the use of information within the country to improve effectiveness” ( 25 sec).

b.       Implications of the Reporting Approach are that the managers are asking themselves, ‘What does M&E do for me?’ They have to collect the information and report it to donors and to UNAIDS, in order to satisfy international reporting standards (21 sec) .

c.       Rename M&E to 'Motivation Empowerment' Avoid Seeing M&E as Something Negative. Countries often have the view that M&E feeds the donor – this can produce a lot of frustration and worry (26 sec).

8.      Apply M&E as a Motivating Force. One of the things we have learned is how motivating it is to even a poorly paid manager to say, ‘my program has achieved x or y - the intangible dimension of this is when people know what they are achieving” (19 sec).

9.       Bank's Role as a Facilitator The Bank's role is to give countries a voice and to “facilitate their ability to learn from each other" (16 sec).

 

Designing and Setting up M&E Systems

13.       Country Ownership of the M&E Process “This is not about the Bank coming in and providing the answer on HIV/AIDS in a particular country” (10 sec).

14.       Empowering District Over National-Level Decision-Making Processes. Decisions made at the national level will not address the correct priorities, the challenges will be different.. “Local people will come together and make decisions, which is based on local evidence but also on local preference. That’s where the mobilization of communities to participate and decide” (1:16 min)

a.       Focus on the Local Decision-Makers. “Who is ‘We’? ‘We’ are the local and national decision-makers, they are not the donor decision-makers” (20 sec).

b.       Implementation Should Take Place at the District Level. 25% of the funds also exist at the District level. This involves a coordinated effort to change - national indicators. Implementation organizations work in those districts so the district teams become management councils. District HIV AIDS Committee’s are called DAC’s. National AIDS Committees (NAC’s) are the coordinating agencies, they do not implement. The local agencies are responsible for implementing (1:28 min).

c.       Engage the Local Decision-Makers in the process of assessing performance. If you engage those who are actually implementing, this again taps into the motivational dimension “People know what they are doing, and if they know where they are going, they will be able to get there faster (41 sec).

15.       Successful Cases of M&E Programs: Cape Verde and Malawi What is involved is working with the National AIDS Commission and the National AIDS Commission Secretariat – who typically have an M&E Officer in place. Our job is to meet with them and help them decide on the appropriate M&E plan. “..our job is to help them build that operational plan, not just the indicators. but the identification of who will be responsible for collecting, how often they are collected where it will be reported, what decisions thatinformation will be trying to influence (2:06 min).

16.       Multi-Sectoral Partnerships. Results in Community Learning National AIDS Council and District level councils that work on HIV/AIDS. Districts have come together to use their multi-sectoral unions with the input of community organizations. Also, there are the Community HIV/AIDS initiatives – communities developing their own proposals and being funded by the national AIDS Commissions – that has 25% of the funds. This is one of the most important things that the MAPs are doing - investing in local innovations“ (3:29 min).

17.       Capacity-Building Through M&E. Within 48 hours of data collection, the individuals we have worked with have “tabulated their data and are beginning to use them for modifying their programs“ (28 sec).

a.       Support the Development of a National M&E Framework. The first element of Designing a Successful M&E Framework involves the following three strategies (1:40 min).

1.       The country needs to have a single M&E framework

2.       The country needs to know where they want to go, to have a key strategic document that guides their work., and to extract from that what they want to evaluate.

3. The country needs to figure out who needs to collect what information, how often.

b.        Maintain a focus on Management Dimensions. The team used a form of management coaching called Rapid Results Initiative, which was provided to key HIV/AIDS decision-makers at the CO level on how to set & achieve goals in a very short period of time. This technique, “takes them through the process of doing/setting a goal, and seeing it accomplished in a short, 100 day period. (30 sec) .

c.       Community Learning. This involves the development of tools that help local areas, districts, and NGO’s to get information on results. “M&E is for them, it is not for us. It will lead them to use their resources in a potentially better way and to build the capacity of their people.” It must be clear to the ministers that “we are there to support them, regardless of the state of their program –what is important is that information is available for them to scrutinize their programs and to make changes based upon the evidence (1:58 min).

19.      ' Are we Helping Manage for Results?' Our prime concern has to be whether we are helping our National, Country, District, Civil Society, local, and line ministry staff manage for results We have gotten hung up on whether the donor agencies are managing for results - they like results as much as we do (34 sec).

20.      Empower the Implementing Organizations. The best method is to empower organizations that are implementing with simple tools to collect information themselves. That they become so mobilized that they have now this qualitative experience in addition to the numbers, which has an impression on them. “That process is so fundamental to learning” (1:06 min).

21.       Management through Rapid Results Initiatives Coaching. The team should set its own goals, and get some coaching in whether the goal is attainable and how to go after it. There should be a focus on “the people who are really making the decisions, as opposed to those who are financing the decision-makers” (1:32 min).

Project Challenges Faced

22.       ' Indicator-itis’ is a disease characterized by an effort to say that M&E is just about listing indicators. Actually, decision-making is the hard-part’ (37 sec).

23.       Coordination of Tasks requires that we be more open, and less hung up on ownership of the information. There is a tendency for each of our agencies to say, ‘lets harmonize on my procedures.’ All of us have had to learn to compromise and be a little bit more open, and less hung up on ownership of the information (25 sec).

24 .      Make the Best Use of M& E Financial Resources. There isn’t a lot of experience building the effective use of M&E resources. ”Countries tend not to be willing to spend money on M&E, and - donor agencies tend not to want to spend money on M&E” (1:17 min).

25.       Partners and Stakeholders in the Authorizing Environment. Include the Bank, the Global Fund for HIV/AIDS, TB and Malaria, now the Bush Initiative. All stakeholders want to know what are the results. There is an “important balancing challenge is to balance this enormous desire among the authorizing environment for results“ (1:05 min).

26.       Harmonization of Indicators is Not Enough “They have harmonized and harmonized on indicators”, however “the real issue is in helping the staff in other agencies to overcome their tendency to be in reporting mode up, rather than the building capacity mode (43 sec).

27 .       Donors Should Harmonize their Efforts. “Are the donors really harmonizing or is there just a lot of talk”? (10 sec).

28.       Spend Time in Rural Field Settings. GAMET is committed to working directly with people who are implementing projects. “if you spend time just in the senior persons office, that really reinforced the idea that M&E is for reporting - or as a political system” (5:06 min).

 

The Future of M&E: Where are we Headed?

29.       Future of M&E Work. Extending and intensifying M&E efforts is the goal our work for the future. “we don’t feel yet that there is any one country of the 24 receiving MAP funds who has an operational M&E plan that is routinely producing evidence and information that we think it needs (28 sec).

30.       Mainstreaming M&E. There is a huge gap in the availability of resources such as talent and money. The Bank has recommended that countries should be spending 5 – 10% of their program resources on M&E. But if countries are not familiar with what M&E might mean, it is very easy to not budge that“ (55 sec).

31 .      Replication of Lessons Learned The more individuals we have at work using these methods consistently, the able they are to replicate this M&E method in neighboring countries. “We are trying to extend our philosophy and approach by co-training other agency staff” Another future direction is trying to spread this philosophy externally as well as internally (48 sec).

32.       Examples of Successful Program Replication. A colleague working in Southern Nepal has been remarkably successful at implementing an M&E system on a recurrent basis. “He has been able to convey to the local populations in very practical terms, the value of this information and how they can use it to make decision” (1:32 min).

33.       Maintain a Focus on Cost-Effectiveness. The aim of our collection of cost-data is to show that creating and implementing M&E systems is actually very inexpensive. The thing to examine is what is most cost-effective? (1:27 min).

34.       Additional Program Resources (3:21 min).

a.       Training Guidelines that Describe the M&E Process A manual has been created in collaboration with UNAIDS, ‘The Yellow Book.’ We’re now working on a brochure for GAMET called ‘frequently asked questions about GAMET’. In early May there will be a website, www..gametproject.org (1:02 min).

b.       Help the Country Counterparts to Define Best Practices. We have the responsibility in GAMET to produce with our country colleagues some reports which show lessons learned, and aid them to define Best Practices. Aiding them to have their lessons and their suggestions in an international arena for debate” (1:13 min).

c.       Cross-Cultural Resources We have a rich resource of multi-cultural colleagues available for translations and on the web, and through teaching aids at low cost in the UK ‘anyone is able to get it – it is the cheapest publication in the world. “we’ve used multiple resources (1:06 min).

Lessons Learned

35.       Cost Management for Program Ownership of cost obligations will need to be clearly established by the task team leader, country director, and sector manager. It is not just about cost – we want this to be something that really matters to them. “We’re more than happy to be a catalyst for change, with our rather limited resources to get started. But there will be costs that the country department and TTL will need to incorporate into their plans down the line“ (2:20 min).

36.       Create a Healthy Environment for Information Exchange within and Between Nations. “Our strategy is to go for the weak point and cover that. The weak point is, how do we get countries learning from each other about performance management, and that is for us cutting edge“ (32 sec).

37.       Concluding Remarks (3:24 min).

a.      M&E's as a Tool to Motivate and Empower. Its greatest asset is its ability to increase motivation and empower its user to think critically. “What we can help do is help signal that results themselves don’t have to be bad even if they are low, that the process of learning is itself exciting and motivational “ (1:36 min).

b.      M&E as a Tool to Create New Forms of Leadership. Monitoring and Evaluation is used correctly is able to “Catalyze change - people to do things they wouldn’t otherwise do. It gives teams eyes, helps them see the direction that they can go in. Leadership - is using your eyes to move towards your vision” (1:48 min).