00:00 Very good.
00:00 Welcome everyone to this,
00:02 uh,
00:02 to this session.
00:03 We are very pleased to,
00:04 to start the conversation today about scaling up and system change.
00:09 And it's going to be a very interesting session.
00:12 We have two parts.
00:13 The first one is a presentation of a recent publication,
00:16 and,
00:17 and let me introduce who are going to speak today with us.
00:20 And then the second part of the,
00:21 of the sessions for those of you who can stay
00:24 in person only
00:25 is a hands-on exercise on,
00:27 on how to actually
00:28 start thinking in a different way,
00:30 this issue of scaling up,
00:32 which is not just doing more of the same,
00:34 right?
00:34 It's a different logic into,
00:35 into it.
00:36 So uh we're gonna have uh opening remarks from Arturo.
00:40 Herrera,
00:41 who is our
00:42 global director for governance here at the World Bank,
00:46 um,
00:46 and then we're gonna have a presentation by the three main authors of the book.
00:50 The first one is Isabel Guerrero.
00:52 Many of you know
00:53 Isabel is an economist.
00:55 He's a psychoanalyst,
00:56 so be careful about what you say.
00:59 Oh,
00:59 don't say
01:03 what is about the what you forget that is very important.
01:08 She's also the co-founder of IMAGO Global Grassroots,
01:12 which helps organizations scale up.
01:14 Since 2014,
01:16 10 years now,
01:17 she has taught the course Scaling Up and Sis exchange
01:20 for Development Impact at the Harvard School of Government.
01:23 Prior to this,
01:24 Isabel worked for 30 years at the World Bank.
01:27 Where she had several country director positions,
01:29 including Bolivia,
01:30 Ecuador,
01:30 Peru,
01:31 and Paraguay,
01:32 and later Mexico,
01:33 Colombia,
01:34 and India.
01:35 Then she was vice president for the South Asia region for 5 years.
01:39 She holds a master's degree from the London School of Economics and graduated
01:43 from the Washington Psychoanalytic Institute.
01:46 Follow me,
01:46 Isabel.
01:47 We're gonna have uh Sijante
01:49 Rocale.
01:50 He's a knowledge manager
01:52 at Imago Global Grassroots where he builds the knowledge function.
01:56 He co-authored a recent theoretical paper
01:58 on the adaptive evaluation methodology.
02:00 And advises in Mago's evaluation projects.
02:03 He previously worked
02:04 at the Behavioral Development lab,
02:07 establishing JPAL South Asia's
02:09 first project in the state of Goa.
02:12 He holds a master's degree in
02:14 Public Administration and International Development from
02:16 the Harvard Kennedy School and a master's in Economics from Columbia University.
02:20 You could have done better,
02:21 right?
02:25 Thank
02:25 you for
02:25 joining us
02:27 and uh.
02:27 Finally,
02:28 uh,
02:29 Jossy,
02:29 uh,
02:30 Fassbender is a program manager at Iago Global Grassroots
02:34 focused on advancing,
02:35 scaling up and adaptive evaluation projects in Latin America and,
02:39 uh,
02:39 global office.
02:40 Previously,
02:41 Jossy worked at the World Bank and the Inter-American Development Bank.
02:43 Bank in implementation and evaluation of projects in jobs,
02:46 rural development,
02:47 and agriculture.
02:48 She holds a master's degree in Public Administration
02:51 and International Development from the Harvard Kennedy School.
02:54 I'm Roby Serovic.
02:55 I will be your host today.
02:56 I'm the manager of the Global Unit on governance here at the World Bank.
03:00 So let's start with,
03:01 uh,
03:01 with Arturo and Michael,
03:02 you,
03:02 you can.
03:05 Improvise.
03:05 Michael Wolcom
03:07 used to work at the World Bank for many years,
03:10 and he actually did a wonderful,
03:12 uh,
03:13 work,
03:13 break,
03:13 uh,
03:14 groundbreaking work on poverty
03:16 at that time,
03:16 if I remember correctly,
03:18 Michael,
03:18 and part of the work we did together with Michael was on,
03:21 on Mexico and the need to actually,
03:23 uh,
03:24 break monopolies down,
03:25 uh,
03:25 not an easy thing to do.
03:27 In,
03:27 uh,
03:27 some of our countries in Latin America.
03:29 So there was a huge conference in,
03:32 not in terms of the number of people,
03:33 but the quality
03:34 on,
03:35 uh,
03:36 basically promoting competition,
03:37 uh,
03:38 in Mexico.
03:38 And I remember when I worked with,
03:40 with Michael on this,
03:41 um,
03:42 we were putting together the agenda,
03:43 and we said,
03:44 who are we going to invite?
03:45 And Michael said,
03:46 well,
03:46 we need to have
03:47 on the room people from that.
03:49 And they said people from I met for this type of thing,
03:51 and he said,
03:52 well,
03:53 they have all the credibility on this,
03:54 so if we are going to,
03:56 to,
03:56 to,
03:57 to open a real conversation all this,
03:59 they need to be in the room.
04:00 After that,
04:00 Michael,
04:01 uh,
04:01 passed to a better life,
04:03 and,
04:03 uh,
04:03 he
04:04 retired from the World Bank and became a full professor at Harvard,
04:07 uh,
04:08 and,
04:08 uh,
04:09 you know,
04:09 living in different countries around the world.
04:11 So
04:11 part of Imago as an advisor if I understand correctly,
04:14 I'm full-time now,
04:15 full-time now.
04:18 Full time.
04:18 Very good.
04:19 All right,
04:20 let's start now with uh maybe Arturo with,
04:21 with the introductory,
04:23 uh,
04:23 remarks from this session.
04:26 Arctor's joining in person.
04:28 He's joining in person.
04:29 I thought he was traveling
04:31 a little bit later today.
04:32 Ah,
04:32 so he's traveling today.
04:33 All right,
04:33 he'll be here.
04:35 OK.
04:35 So,
04:36 shall we start?
04:36 What,
04:37 what do you think,
04:37 uh,
04:37 with the wedding?
04:40 But
04:41 Should start,
04:43 well,
04:43 it would be nice to know who is in the room in terms of what,
04:46 what areas do you work and we can use a little bit of time.
04:50 So,
04:50 so name and the where,
04:51 where do you work just to get a sense.
04:54 Yes,
04:55 Zen Choudhry,
04:56 I'm in the poverty division,
04:57 but we are the behavioral unit of the World Bank.
05:01 And me see Yong Kim,
05:02 a senior specialist in contacting,
05:04 working with Matthew and with other a call.
05:07 My name is Sarah.
05:08 I work at the Labor DIC with Elena Carranza.
05:11 I think it's a couple of us.
05:14 Hello.
05:14 Good to see you.
05:15 Guzmano Lassa.
05:16 I'm a senior government specialist now in the global unit with the
05:19 Peruvian team.
05:22 I,
05:22 I'm Matthieu.
05:23 I'm a senior economist in Ruby's unit as well working on governance.
05:26 I'm the program manager for Coalition for Reform.
05:29 Exactly,
05:29 yes,
05:30 I mean,
05:30 I'm gonna get more chairs in case they will be welcome.
05:33 Uh,
05:34 I'm Kiara Brunke.
05:34 I'm the practice manager for Public institution data
05:36 and analytics in the governance of global practice.
05:41 Oh,
05:42 the woman.
05:43 Uh,
05:43 I'm Ahab Tofi.
05:44 I'm from MTI,
05:45 not completely foreign to this practice,
05:47 um,
05:48 and what is the,
05:48 is the macro,
05:49 is the,
05:49 is the old EP.
05:51 Oh yes,
05:51 apologize.
05:52 It's all these acronyms ever,
05:54 uh,
05:54 in grad school,
05:55 uh,
05:55 we did some workshops on systems change in complex systems,
05:59 very good,
05:59 um,
06:00 and,
06:01 um,
06:01 the theme that we explored was,
06:03 um,
06:03 uh,
06:03 climate change.
06:04 How can we?
06:06 So,
06:07 uh,
06:07 this triggered,
06:08 uh,
06:08 memories or flashbacks or traumas in grad school.
06:12 And now I want to come back here,
06:13 do things differently
06:16 with your post-traumatic stress.
06:20 I'm Guillermotar.
06:22 Uh,
06:22 you also work in the labor GSG with the
06:27 Lara,
06:27 and it is.
06:27 I'm I'm
06:28 external
06:35 specialist
06:40 Carwin.
06:41 Carmen Del Rio and
06:42 and the governance team um working on partnership and communications today.
06:47 Hi,
06:47 I'm Diana Swin.
06:49 I'm in social protection and jobs also with Eliana Carlo.
06:53 And
06:53 I'm Elizabeth McBride.
06:55 I'm a knowledge and advocacy consultant with WiFi and um
06:59 also working on a book about
07:01 sort of a
07:02 guidebook for business leaders adapting to the new world of capitalism,
07:05 or one of the concepts we're writing about in it is a coalition building.
07:09 Oh,
07:09 very good.
07:11 I wanted to call it the Ion the Women reports Finance Initiative,
07:14 which is an HD
07:16 on Pablo Guerreros,
07:17 senior governance specialist at the public administration we need.
07:22 Great.
07:22 We have
07:23 to
07:27 careful.
07:29 So Arturo,
07:30 we,
07:30 we start with your introduction,
07:32 uh,
07:32 to the session.
07:34 Yeah,
07:34 no,
07:35 it's,
07:35 it's,
07:35 it's,
07:36 it's not gonna be about this.
07:37 I was,
07:38 uh,
07:38 as I was coming here,
07:39 I,
07:40 I realized that the first person I met the World Bank
07:43 ever was actually again
07:47 we,
07:47 we were in a dinner in,
07:48 in Mexico City,
07:50 I mean
07:51 that was right after,
07:54 after I,
07:54 I met up.
07:56 Some presentations in New York.
07:58 I talked with her what,
07:59 what I,
08:00 uh,
08:01 she was our country director.
08:02 What I,
08:03 what I did and then she said,
08:05 so why don't you present it in Washington.
08:07 So I came to Washington
08:09 and uh I've been living in New York.
08:11 I thought it was easier
08:12 to catch a cab here than in New York,
08:15 uh,
08:15 which was not at the time.
08:17 So I,
08:18 so she asked R to pick me up and I,
08:20 and I declined.
08:22 So I,
08:22 I end up,
08:23 uh,
08:24 I end up running.
08:26 So I came sweating
08:29 as today.
08:30 Uh,
08:31 it's becoming a partner,
08:33 uh,
08:33 but what is interesting is that,
08:35 uh,
08:36 the second time we met
08:37 she was hosting
08:39 a presentation for me here.
08:41 Now we're hosting a presentation for you.
08:46 That's perfect.
08:46 So let's go to the presentation and then,
08:48 uh,
08:49 we will have time.
08:50 Until 9:45 for this,
08:52 uh,
08:52 session,
08:53 right back.
08:54 Yes,
08:54 and,
08:54 uh,
08:55 so if we,
08:55 if we have time,
08:56 hopefully we'll have a round of questions Q&A,
08:59 and then the second part of the session will be only for those who are in the room,
09:02 and so on,
09:03 starting at 9:45 or so.
09:04 So Isabel,
09:05 great,
09:06 good.
09:06 Well,
09:06 very happy to be here and see a lot of friends around the table,
09:10 um.
09:11 Uh,
09:11 we're going to go,
09:12 uh,
09:12 through this presentation.
09:13 I will start,
09:14 uh,
09:15 thinking about systems and then
09:17 see and Josie and Michael will,
09:19 will
09:20 work through and applied,
09:21 uh,
09:22 as,
09:22 uh,
09:23 you were saying systems can be traumatic,
09:25 but they can also be very easy to,
09:27 to get your,
09:28 your head around it.
09:30 Uh,
09:30 so why do we care?
09:31 This is a scaling up framework where we first make
09:35 clear,
09:36 uh,
09:36 that what is the vision,
09:37 what is the gap that somebody's trying to
09:39 Address through an intervention,
09:41 a program or a project,
09:43 and then really important to understand the system failure.
09:47 Why haven't others been able to solve that problem that you're looking to solve,
09:51 which could be
09:52 the market or the government or social entrepreneurs.
09:55 Understanding that failure is a really important part of your success later.
09:59 Then you get into the model and,
10:01 you know,
10:01 uh,
10:01 and experimenting or or theory of change,
10:04 how you're going to have the social impact that you think
10:07 that you're having.
10:08 And your business model,
10:10 your intervention,
10:11 your minimum viable intervention,
10:13 the essence of that intervention,
10:15 and that's where we work a lot with
10:18 organizations that can take actually several years,
10:20 and then only when,
10:21 when you found that model,
10:23 you start spending money and building the foundation
10:26 so that you can um er you know,
10:28 make the best use of your runway
10:31 by testing the model before building that
10:33 foundation and building evidence as you see
10:36 starts from the beginning.
10:37 the vision,
10:38 uh,
10:38 the best organizations are those that start with the
10:41 evidence from the beginning and then at the end,
10:44 of course,
10:45 that becomes part of the learning and helping you to,
10:48 uh,
10:49 to make sure that you are doing what you thought you were doing.
10:52 Uh,
10:52 we have the framework with many questions.
10:54 I just went through them,
10:56 why isn't the system producing the results that you want to produce.
11:00 So,
11:00 we look at systems.
11:02 I started teaching this course on scaling up 10 years ago.
11:06 And it made a lot of sense then.
11:09 Suddenly with everything that has happened
11:12 with COVID,
11:12 the disruptions,
11:13 the climate change,
11:15 it,
11:15 it seems sort of strange to be focusing so much only on scaling up,
11:20 and that's why we have really
11:22 started to put much more emphasis on systems because systems are failing.
11:26 Because we see that around us many sectors are being disrupted by tech,
11:32 um,
11:33 and therefore understanding that environment before knowing
11:36 whether you're scaling up within a system
11:38 or you have to change the system to scale development impact is important.
11:43 So the first time
11:44 The first question is that,
11:46 understand where you are,
11:48 where your program is relative to that system.
11:51 Then to help clarify the model and
11:53 interventions and strengthen the theories of change.
11:56 As you will see in the clinic afterwards,
11:58 when you start to map the system,
12:00 sometimes you realize you cannot go at it alone.
12:03 You need to bring other stakeholders,
12:05 other partners that
12:06 have other
12:07 interests but also other skills into the The System for change and
12:11 that's like coalitions for Change is a very good example of that.
12:15 And then
12:16 to understand what's the pathway to scale that you're choosing
12:20 and do you need to think about alternative pathways because
12:24 as we will see,
12:25 systems are very resilient.
12:27 Uh,
12:27 they are,
12:28 uh,
12:29 produced,
12:29 they're,
12:30 they're designed to produce the results that you see.
12:32 So,
12:33 uh,
12:34 what do we mean by a system?
12:36 Nature is the most amazing,
12:37 you know,
12:38 example of a system.
12:39 You think about the trees,
12:40 you think about all the roots that are underneath,
12:43 all the mycelium that is underneath.
12:45 That system
12:46 is an awesome system,
12:48 very resilient.
12:49 A tree will die and for
12:51 decades sometimes,
12:53 it will be fed from the roots underneath,
12:55 from the system of roots under,
12:56 underneath.
12:57 Same is true for the,
12:58 for the,
12:59 for the human body,
13:00 you know,
13:00 the human body is a system totally uh interrelated.
13:04 So the parts,
13:05 the sum of its parts
13:07 is,
13:07 is much more than,
13:09 than each part uh independently.
13:11 So there's a property of the whole in a system,
13:14 which is really important to understand when you're trying to change that system.
13:18 So the properties of a system
13:20 come from how
13:22 the different parts interact,
13:24 rather than just from the parts themselves.
13:26 And so,
13:27 So,
13:27 when you want to change a system,
13:29 you need to shift the conditions that are holding that system in place.
13:34 Um,
13:34 and as I said,
13:36 since systems are designed to produce exactly the results
13:39 that they
13:40 are producing when you're writing your book on business for
13:44 this whatever you're seeing is actually
13:48 an awesome system working to replicate itself every time.
13:52 Some of the
13:53 Uh,
13:53 properties of a system is that,
13:55 uh,
13:56 a system is nested,
13:57 and if you go in,
13:58 you will find another system which is also complex.
14:01 So finding the boundaries to the system that you want to change is part of the,
14:05 of the challenge.
14:07 It's non-linear,
14:08 some sometimes you have a little change in one part
14:11 and it causes unexpected large uh sys uh uh large
14:16 impacts on the overall system.
14:18 I remember
14:19 when I was uh the VP for South Asia,
14:22 we had an amazing pro uh project um
14:25 for road safety for trucks in Pakistan.
14:29 And so we thought that it would be great
14:31 uh to have stops because the
14:34 truck drivers will be able to stop,
14:37 have a rest at night,
14:38 and then continue
14:39 uh later and reduce um
14:42 Uh,
14:43 reduce
14:44 accidents.
14:45 And what happened is,
14:47 without us knowing the nonlinear part,
14:49 there was a huge increase in venereal diseases
14:53 among truck drivers,
14:54 which if you think about it is,
14:55 of course,
14:56 but we were not thinking systems
14:58 when we were designing that project.
15:00 It's unpredictable,
15:02 linear predictions always fail.
15:04 It has feedback loops that help actually
15:07 reinforce and sustain that system.
15:10 Or destabilize you when you want to,
15:13 to get a systems change,
15:15 you need to disrupt,
15:17 you need to destabilize,
15:18 and you need to know where to do it.
15:20 And then
15:21 systems learn,
15:22 they are adaptive,
15:23 they will learn from
15:25 uh uh shocks and then adapt to those
15:28 shocks like what has happened with COVID actually
15:31 it's a very good example.
15:32 So there are um
15:34 3 levers for system change.
15:38 Um,
15:38 the first level is the one that we usually work on,
15:41 policies,
15:42 regulations,
15:44 resources.
15:45 Uh,
15:45 that is the structural level,
15:46 it's very explicit,
15:48 we usually go to that for change.
15:50 But if you find that a system is being really
15:52 tough and resilient and it doesn't produce the change and so we have so many wicked
15:58 Development problems if that happens,
16:00 you need to go one step further,
16:01 which is the relational change,
16:03 which has to be with relations between uh players and the power dynamics in
16:09 um in the system.
16:11 Those are much more powerful leverage points to change a system.
16:15 That's why coalitions for change are so important.
16:18 And then finally,
16:18 the,
16:19 the biggest,
16:20 the biggest leverage is the shift in mental models.
16:23 That is the one that really brings transformation,
16:26 and that's why all these um
16:29 advocacies are important,
16:30 all these movements are important.
16:32 Over time they change mental models in society,
16:36 and within that
16:38 the most important mental change,
16:40 mental shift that you can make is your own.
16:42 Uh,
16:42 as Gandhi said,
16:43 be the change that you want to see outside.
16:45 That is the first step,
16:47 because,
16:48 because the system is designed to produce the changes that it's,
16:51 the,
16:52 the,
16:52 the realities that it's producing,
16:55 you have a role in that and understanding what's
16:57 your role is the first step for system change.
17:00 So that is a very quick um overview of systems and I'm going to pass on to Michael to
17:06 introduce us to the next,
17:07 I'm trying to,
17:08 yeah.
17:10 So that's
17:11 why we got into it and Isabel has sketched
17:13 some of the
17:14 conceptual aspects of systems that people.
17:18 You,
17:18 you said it with some pain,
17:19 um,
17:20 but what I'd like to
17:22 emphasize that we're going to spend the rest of the session on
17:25 is actually practical tools
17:28 that actually we have developed in the field
17:31 to engage with problems.
17:32 Either one of them is actually,
17:33 we actually
17:34 had to do
17:35 online in the field and the other really,
17:36 really
17:37 um in the field.
17:38 Now we've always used,
17:40 or we've discovered we were,
17:42 We discovered we were,
17:43 we're doing stuff and then we discovered it's called human-centered design.
17:46 Um,
17:46 so we
17:47 use a lot of that,
17:48 we do a lot of mapping and interpretation,
17:51 and as Isabel highlighted,
17:53 a key part of this is going from the systems diagnostic
17:57 to exploring where is the resistance,
17:59 where are pathways to change
18:01 through the system,
18:02 and how that then maps to
18:05 The classic domain of theories of change and
18:08 dog frames,
18:09 and then
18:09 it almost always you have to iterate.
18:11 Let's go forward.
18:13 We're going to,
18:14 um,
18:14 Sedat and Josi are going to present
18:17 two
18:17 kinds of techniques that we've developed in the form,
18:20 in,
18:20 in,
18:20 in the field.
18:21 One is what we call
18:22 a systems dynamics map that emphasizes
18:26 mapping
18:27 relationships and functions within the system.
18:29 The other is starting from
18:32 uh an aftermath of the system.
18:34 The only thing I'd like to highlight as you go in,
18:36 these are two prisms
18:38 on
18:39 the same system,
18:40 and the question is,
18:41 it's often a very pragmatic choice
18:43 of where you start.
18:44 So with that,
18:45 so that,
18:45 over to you.
18:46 Great,
18:47 thank you.
18:48 So,
18:48 we,
18:49 I'm gonna walk through an example of our work,
18:51 uh,
18:52 we did with UNDP Tunisia
18:54 and the,
18:55 the,
18:55 the development challenge we were dealing with was intimate partner violence,
18:59 and this is prevalent
19:00 in many developing countries,
19:02 but in this case,
19:02 we worked on Tunisia.
19:04 And the existing intervention strategy was one that's very common in
19:08 multilaterals and in governments where you
19:10 have these portfolio of interventions,
19:12 often in different departments and areas.
19:15 In this case,
19:15 it was justice,
19:16 security,
19:17 and women's empowerment,
19:19 but
19:19 They're uncoordinated.
19:21 It's unclear how they connect and it's unclear how all of them together
19:25 are causing the system to change or transform in the direction we want to.
19:29 And so our task here was to understand
19:32 how these different parts of the system are interrelated
19:35 and also how are the gaps in the intervention portfolio
19:39 and which feedback loops that are,
19:41 are causing this problem to be more persistent,
19:44 uh,
19:45 are addressed by these interventions and so.
19:48 The very first step,
19:49 and this is often the first step in
19:52 when we do a diagnostic,
19:53 is to really understand the problem
19:55 much more deeply.
19:56 And so this was really a joint effort with
19:59 UNDP Tunisia and Imago.
20:01 So,
20:02 UNDP Tunisia did a lot of work interviewing 30+ stakeholders.
20:05 So different actors.
20:07 In the system.
20:07 So from the Ministry of Child and Health,
20:10 from the Ministry
20:11 of Justice,
20:12 Security,
20:13 they interviewed NGOs working very closely with um
20:17 people who,
20:19 with,
20:19 with intimate partner violence and victims of violence.
20:21 They also interviewed
20:23 victims of violence and spoke to sociologists.
20:25 Sanny Mago spoke to a lot of people in UNDP Tunisia,
20:29 personnel who were working on projects of intimate partner violence.
20:34 Uh,
20:34 and so,
20:35 so what we're seeing here is a problem space that's a culmination of,
20:38 of that work.
20:39 And so roughly we found
20:41 three big areas of
20:43 proximate causes of intimate partner violence,
20:45 and these actually aligned very closely to,
20:48 uh,
20:48 Isabel's,
20:49 what Isabel showed which were the three leverage points in the system.
20:52 So the first,
20:54 institutional,
20:55 this,
20:55 this really relates to policies and rules.
20:58 So
20:58 the judiciary where we found,
21:00 um,
21:01 that,
21:02 you know,
21:02 judges often.
21:04 Put cases on hold expecting the family to withdraw,
21:07 uh,
21:07 because of,
21:08 expecting victims to withdraw because of family pressure.
21:11 Uh,
21:12 that's the police where there's issues of trust and reassurance
21:14 of women going to the police and social services,
21:17 which is shelters.
21:18 There's societal factors,
21:20 and this
21:21 is,
21:21 I think,
21:21 more in the domain of mental models that Isabel just presented.
21:25 It's about,
21:26 it,
21:26 it's about societal norms,
21:28 patriarchy,
21:29 the institution of marriage,
21:31 pressures to preserve family image,
21:33 and so on and so forth.
21:34 And then there's individual factors,
21:36 things like,
21:37 uh,
21:38 a woman's own agency,
21:39 their ability to earn income,
21:41 uh,
21:41 their support from immediate family members,
21:44 and so on and so forth.
21:45 So,
21:45 these were the three pillars.
21:47 From
21:47 which we developed our
21:49 understanding of the system.
21:50 So that's the boundary of the system.
21:52 There's of course a much larger system that envelopes this
21:55 system that we're dealing with with intimate partner violence.
21:59 And so
22:00 there's of course many factors.
22:01 COVID-19 was important then and you know,
22:04 other,
22:04 other factors.
22:06 So from here we began mapping the system,
22:09 but we use those three pillars and causes to
22:12 understand how different variables interrelate with one another.
22:16 And so the first is,
22:17 we always start with the,
22:19 the,
22:19 the main outcome.
22:20 So in this case,
22:21 the main outcome
22:22 was the number of IPV instances,
22:24 the likelihood of a case being pursued with,
22:27 and also the number of convictions.
22:28 And because this was a problem,
22:30 the hypothesized levels
22:32 were,
22:32 were low in the case of number of convictions,
22:35 high in the case of a number of instances.
22:37 We also mapped institutional factors which I already described
22:41 pertaining to the legal,
22:43 uh,
22:43 legal delays,
22:44 you know,
22:44 trust with the police.
22:46 And,
22:46 and their hypothesis levels.
22:48 There's societal factors uh pertaining to
22:51 um beliefs about resolving marital problems internally
22:55 and we found them to be high.
22:56 All of these hypothesized levels,
22:58 of course,
22:58 were built
23:00 from,
23:00 um,
23:01 our interviews and,
23:02 and discussions with various stakeholders.
23:05 And then there's individual factors around agency,
23:07 awareness,
23:08 so on and so forth.
23:09 But I think the task of the system dynamics mapping,
23:12 and this may be familiar to you,
23:14 given you've,
23:14 you've done some of this work,
23:16 is to find the interrelationships between these variables.
23:19 And so we start as an example with one.
23:22 So,
23:22 if you have a low likelihood of a case being pursued.
23:26 Uh,
23:27 the number of convictions is likely to be low,
23:29 and this is a positive relationship in the sense
23:32 that a low variable leads to another low variable.
23:36 Now if we take it a step further,
23:38 right,
23:38 uh,
23:38 low number of convictions
23:40 means there's low awareness,
23:42 uh,
23:43 among women about their legal rights,
23:45 partly because they don't see it in the media,
23:47 their neighbors don't know it.
23:48 So the low number of convictions contributes to this low awareness
23:52 and this low awareness in turn
23:54 leads to a further,
23:55 uh,
23:56 uh,
23:56 a reduced likelihood of a case being pursued
23:58 and persisted with because they have low awareness.
24:01 And so what we have just mapped.
24:02 Is a feedback loop that explains why IPV is a persistent problem in the system.
24:08 It's one of the many feedback loops that explains that this awareness,
24:11 this lack of awareness is an
24:13 adverse self-reinforcing feedback loop
24:16 that,
24:17 that explains why this problem has not changed.
24:19 And so we did this mapping with all.
24:22 All of the relationships and if you look closely,
24:24 you'll be able to see,
24:26 you can draw many circles of different feedback loops that
24:29 explain different hypotheses of why this problem is persistent.
24:33 But,
24:33 but the,
24:34 the goal of this was not to understand just
24:36 different interactions between different parts of the system,
24:39 but to see
24:40 where our interventions coming in
24:42 and where do they address some of these feedback loops.
24:45 And so we did a lot of detailed analysis,
24:47 but this was a broad landscape analysis of where multilaterals.
24:51 We're working in Tunisia on the problem of intimate partner violence
24:55 and you see,
24:56 most of the projects
24:57 were on the,
24:58 on,
24:59 on rules and policies and UNDP was focusing a lot on justice and security,
25:03 but majority
25:05 were,
25:05 were on rules and,
25:06 and policies
25:08 which often is,
25:09 you know,
25:10 the first level but not the most powerful as Isabel was,
25:13 was mentioning.
25:14 Uh,
25:14 the hardest to change our mental models and so very few
25:17 projects on societal factors and one of the things we found.
25:21 Was that a lot of the societal interventions did not focus on attitudes to men.
25:25 So that was one of the big gaps we found in our analysis.
25:29 Uh,
25:29 one of the things we did was look at
25:31 individual projects and how they enter the system,
25:34 but also which subsystem they affect and what feedback loops.
25:37 And so,
25:38 a quick illustration of that is here.
25:40 So
25:41 suppose we have interventions to improve access to women's entrepreneurship.
25:45 So that immediately affects women's income.
25:48 Right?
25:49 But
25:49 if you follow the arrows from women's income
25:52 to the main,
25:53 uh,
25:54 outcome variables we started.
25:56 What you see is the feedback loop it effects.
25:58 So,
25:59 women's income,
26:00 if you reduce,
26:00 if you increase women's income,
26:02 you reduce dependence on spousal income,
26:05 which was one of the main reasons women tolerate violence.
26:07 So reducing
26:09 dependence on spousal income
26:11 reduces tolerance for violence,
26:12 which means fewer number of IPV cases,
26:14 but also fewer number of cases,
26:17 um,
26:17 a higher likelihood of cases being pursued.
26:20 And,
26:20 uh,
26:21 so this is just another illustration
26:23 of that.
26:24 That was the subsystem of the larger system,
26:27 and this is an initial theory of change which can then be further fleshed out.
26:31 So the idea of doing all of this was really to
26:35 understand how these different interventions
26:37 from different departments are interconnected.
26:39 Often they address one or many feedback loops
26:42 and how this can lead to a greater understanding of,
26:45 you know,
26:46 what is the theory of change.
26:47 Uh,
26:48 that this effects,
26:49 and from here I'll pass on to Josie.
26:51 He'll go on to system after another.
26:53 Thank you.
26:54 So yeah,
26:54 I'll present,
26:55 like Michael said,
26:57 another reason how you can look also at assistance,
26:59 and this one is you start by mapping all
27:02 your stakeholders and then the relationships between them.
27:04 In this case,
27:05 it was,
27:06 uh,
27:06 work that we were doing with local governments in Brazil.
27:09 Uh,
27:09 it was the uh recipe government,
27:11 and they were trying to respond
27:14 to the weed development challenge and that they were.
27:17 facing a lot of learning deficits in reading and math due to COVID-19.
27:21 And there were so many things changing at the same time,
27:24 and the government was trying to explore different alternatives of how to solve
27:28 these learning gaps.
27:29 So they were just exploring different policies.
27:31 It was very hard to say,
27:32 OK,
27:33 we're going,
27:34 we're gonna go into one policy and really
27:36 uh do an impact assessment to know if this is working or not.
27:39 So we really needed to.
27:41 Understand the whole and really try to get an
27:44 idea of how are all these policies working together,
27:47 but also it was important to first
27:50 understand uh for each of the interventions,
27:52 how was the system uh working
27:54 in relation to,
27:55 to that.
27:56 So one of the policies that we were trying to help them assess was that
27:59 they were trying to implement teaching at the right level,
28:02 which is uh a program that was developed by RTA in India.
28:06 So of course it's,
28:06 it also needed to have some adjustments to be implemented
28:10 in Brazil,
28:10 but basically,
28:11 it,
28:12 the way that it works is that you,
28:14 you regroup students based on the initial learning levels that you're observing.
28:18 And according to that,
28:19 you give them the material to their initial learning level,
28:22 and the idea is that then they are going to be able to
28:25 recover and be at the grade level that they are currently assigned.
28:29 Um,
28:30 so what we did was we first really make a big
28:34 effort to understand who should be there mapping the system.
28:38 And we think that
28:39 uh outdoor mapping is very important,
28:41 especially when we are doing fieldwork,
28:43 it's important that you have people from the school,
28:45 people from the government,
28:47 because everyone understands the system in a different way.
28:49 But everyone knew,
28:50 OK,
28:51 at the center of the intervention is the student.
28:54 This is the,
28:54 actually the,
28:55 the last beneficiary,
28:56 the person that we want to affect.
28:58 So they put the student in the center and then we started
29:01 mapping for all the stakeholders that are connected to the student,
29:04 if we are thinking,
29:05 if we are thinking of the teaching at the right level intervention.
29:08 So we have
29:09 At the school level,
29:10 we have the school directors and coordinators,
29:12 but then we also have teachers.
29:14 And then it's important to even go into that system and separate the levels.
29:18 We have family of commu or community of students.
29:21 We have the Department of Education itself,
29:23 and we have CAEI which was
29:25 a research organization which was helping them
29:28 develop the assessments for this intervention.
29:31 So,
29:31 for example,
29:32 uh,
29:33 we map,
29:34 the very important thing in the system
29:36 mapping when you focus on actors is to
29:39 focus um mapping the relationship but both ways.
29:42 So for example,
29:43 if CAG,
29:44 that is a research organization,
29:46 is developing formative assessments,
29:48 which are gonna give feedback to understand the students uh
29:52 learning.
29:53 Then the students need to participate in these assessments.
29:56 Then with this information,
29:58 then CET gives the information
30:00 to the school directors and coordinators so that they can analyze
30:03 uh how the students are improving learning levels or not.
30:06 And then with them,
30:07 we map all the different relationships.
30:10 So it was like,
30:11 everyone just really understand,
30:13 OK,
30:13 how does this work or it doesn't work,
30:15 but like,
30:16 basically just understanding all the different connections which even
30:19 include how is the family connected to the students,
30:21 how is the Department of Education connected to families,
30:24 so at every level.
30:26 And this in a way gave us
30:28 a process mapping as well.
30:29 No,
30:31 you understand what is the process so that this
30:32 intervention actually work in the system that you're working
30:35 in very simple terms,
30:36 it was OK.
30:37 First,
30:37 the government needs to give the the guidelines for
30:40 teaching at the right level to the schools.
30:42 Then the schools diagnose the student learning levels.
30:46 Then with this,
30:47 uh,
30:47 the teachers are going to be able to regroup the students,
30:50 and then the students receive teachings,
30:52 uh,
30:53 based on the teaching at the right level methodology.
30:56 And this
30:57 will give you an initial way of thinking of what is the theory of change.
31:01 Of course,
31:02 there are some intermediate outcomes that are missing
31:04 that it's like after following all these steps,
31:06 then you get some,
31:07 to some intermediate outcomes that
31:09 um make it possible that the students are able to reach
31:12 at the expected
31:14 learning level for the grade level,
31:16 and then you are going to be able to decrease learning gaps after the pandemic.
31:20 But then the interesting thing to map with them
31:24 was that
31:25 even if they could see that there were many parts that
31:28 they were actually working as they are supposed to work,
31:31 there were also many other parts where
31:33 the system was just not responding.
31:35 Like
31:36 there were many things that even if the
31:38 government had a very clear understanding of how things
31:41 uh will need to happen
31:43 inside of the schools,
31:44 that things were just not happening.
31:45 So that's why it was important.
31:47 that the coordinators that were in contact with the schools were
31:50 the ones that were in the room and could say like,
31:52 actually,
31:53 it's not happening that the teachers have any kind of meetings
31:57 with the directors to access the information and to know if actually
32:00 uh students are improving their learning levels.
32:02 Also,
32:03 there are no meetings so that they can then
32:05 improve the way that they are doing this uh intervention with the students.
32:09 So then,
32:10 this was the initial step,
32:12 so that with them we could work on a revised theory of change,
32:15 and then we also developed a
32:18 process tracing methodology,
32:19 which is something that we do in adaptive evaluation,
32:21 but that basically
32:23 helps you understand for each step of the theory of change,
32:26 what is happening,
32:27 what is not happening,
32:28 and then regularly going to the schools and creating this.
32:32 Feedback mechanisms between people that has to be aware of this,
32:36 that includes researchers,
32:38 implementers,
32:38 people in the government that can make decisions
32:40 of the pedagogy that needs to change.
32:43 So that actually you don't need to wait so long to see the final outcome change,
32:47 but actually understand
32:49 on the way,
32:50 like what is working and what is not
32:52 working and be able to improve your implementation.
32:55 So this is a very fast example but it's like
32:59 uh very connected to what we are gonna try to be working with you today
33:03 but before going into that or opening questions
33:06 I wanted to pass again to Isabel because
33:09 we wrote a book where we explain these
33:11 tools and also
33:13 other ways of thinking about.
33:17 So for everybody in the room,
33:18 all,
33:19 all connected,
33:20 OK,
33:21 this is the latest publication of Iago scaling up development,
33:24 uh,
33:25 impact,
33:25 uh,
33:26 you can find it in many places,
33:27 right?
33:27 Amazon only.
33:28 Amazon only,
33:29 OK.
33:30 And if you refer to this conversation,
33:31 you're gonna have it.
33:35 we go and chapter 3 is on systems change and then chap the last 3 chapters are on
33:40 adaptive evaluation which is
33:42 how do you accompany this process through uh what we call a process of
33:46 meaning making,
33:48 uh,
33:48 about what is happening in a way that
33:50 In fact,
33:51 evaluations are part of the solution,
33:52 but a,
33:53 a small part,
33:54 most of it is about iterating,
33:56 helping to pivot,
33:58 giving just in time information
34:00 through a set of tools that are described there.
34:03 Great.
34:03 Good.
34:04 Thank you very much for the presentation,
34:05 the 4 of you.
34:06 So we have a few minutes for Q&A maybe now before we go to,
34:10 into the
34:10 hands-on
34:12 uh
34:13 exercise later on.
34:14 Anyone wants to come in?
34:16 Maybe that's the question.
34:17 Thank you very much.
34:18 Actually,
34:19 I like this,
34:20 uh,
34:20 conceptualization as well.
34:22 So,
34:23 and the example are very compelling.
34:27 Yet I'm trying to think all the way through
34:30 the example of education actually you've highlighted numerous factors
34:35 in order to succeed,
34:37 and Brazil has a certain context,
34:40 of course.
34:40 Also we work a lot because they have a lot of
34:42 data now they're much more advanced in terms of monitoring,
34:45 and I was wondering
34:47 how would that play out in countries that are less endowed
34:51 in terms of skills
34:54 it's often.
34:55 You know,
34:55 we have tried,
34:56 I don't know,
34:57 uh,
34:59 Bigger SWAT teams initiatives.
35:02 I think
35:03 your conceptualization can then play in different ways,
35:06 but we've seen example of that,
35:08 and it,
35:08 it's just I'm trying to think how we can
35:14 That sometimes there is accountability.
35:15 There has to be some level of accountability that I
35:18 would like you to pull out more at some point
35:21 of also achieving the results,
35:23 and all these are instruments,
35:24 but somebody needs to be really responsible and precisely for what you said,
35:28 Isabel,
35:28 at the beginning,
35:29 this level of accountabilities are becoming
35:32 looser in many countries and with CEOs and advanced economies,
35:36 it's all very confusing these days.
35:38 I wonder also if you could tell a little bit about that.
35:41 Yeah,
35:41 great question.
35:42 Should I answer?
35:43 Yeah,
35:43 so,
35:44 um,
35:44 we use this with SEWA.
35:46 SEWA is an organization of 2 million women that come from the informal market who
35:51 they,
35:51 they
35:52 don't even know how to read and write,
35:54 and we do the system mapping with them.
35:56 Uh,
35:57 because as you will see in the clinic,
35:59 you can do it in a very playful way,
36:01 uh,
36:02 where
36:03 what we do is facilitate this by asking the right questions
36:06 and then they are doing the mapping and it works beautifully.
36:10 So I think in very low capacity environments you can do it.
36:14 Um,
36:14 the important thing is to help them
36:17 choose who are the people who need to be in the room
36:20 and then have somebody who's going to be the problem holder
36:23 and who eventually is going to be the champion through that.
36:25 Usually
36:26 it would be the organization that wants to work on solving that problem,
36:30 who needs to hold it,
36:31 uh,
36:32 in order for the accountability of the work as you
36:35 go through different iterations because
36:38 you will need to do it several times,
36:39 it's not a one,
36:40 a one shot thing,
36:42 yeah.
36:42 I can also compliment.
36:44 So I think that's,
36:45 for example,
36:45 in Brazil it was very important for us also to determine,
36:48 OK,
36:49 inside of the government who are our champions,
36:51 who actually believes in this and wants to do this and
36:54 wants to be part of the core team that is gonna keep like pushing this forward.
36:58 And then we also have people from CAI who is
37:00 the research organization who is going to be doing this,
37:02 so you just identify the.
37:04 Who,
37:05 what,
37:05 and when,
37:06 and it was very important to say,
37:08 OK,
37:09 every 2 months people are going to meet and discuss these results so
37:13 that we can keep understanding how the system is changing or not.
37:16 I know that we have also faced,
37:18 uh,
37:18 a lot of restrictions because if the government changes and it's like difficult to
37:23 again get a new person,
37:25 but so.
37:25 But you also need a lot of
37:27 ownership from the government and they need to understand why it's important.
37:30 So we kept trying to
37:32 involve the government and say,
37:34 look,
37:34 we are here to try to help you make decisions,
37:37 and then in that,
37:38 in that way,
37:38 they were OK,
37:39 actually,
37:39 this makes sense,
37:40 this is just
37:41 kind of complementary to the,
37:43 to what we are doing already.
37:45 And also I think something
37:47 For the low skill or like
37:49 uh countries that don't have a lot of data like
37:52 we don't rely a lot of administrative data.
37:54 We try,
37:55 we try to understand what administrative data is available that
37:58 we can use so that we don't build it again.
38:01 But then we also develop very simple questionnaires and we also Understand
38:05 when they are already going and interviewing teachers,
38:08 so that it's not like we are adding a new questionnaire that they don't want to have,
38:12 it's just,
38:13 hey,
38:13 how are you working right now and how
38:15 in this way that you're working,
38:17 we can just do it in a more structured way
38:19 so it's easier for you and it's also more sustainable.
38:24 Thank you,
38:24 Justiceabel others.
38:25 Yes,
38:25 Jose.
38:27 I'm curious if you can talk more about the issue of mental models and how they affect
38:32 the
38:33 The initial step that you had there that
38:35 it's uh understanding the problem even more.
38:39 Owning the problem and,
38:40 and in many societies in many,
38:42 many governments lack the,
38:45 the self-awareness to understand that there is something that needs to be changed
38:50 and,
38:50 and they,
38:51 there is not a cultural kind of value of,
38:53 of
38:53 say,
38:54 OK,
38:54 acknowledging that things are failing,
38:56 therefore
38:57 you,
38:58 you have a big block there,
38:59 yeah.
39:00 Great question.
39:01 So I,
39:01 I think first of all,
39:03 that's why we talk about the wicked development problems,
39:05 because we go into those problems that are really resistant to change.
39:09 Some problems don't need this,
39:10 but when you're trying to do something and
39:12 You try again and again and it,
39:14 and the problem persists,
39:15 that's the time where it's good to go in and
39:18 you know,
39:18 I,
39:18 I think,
39:19 uh,
39:19 it,
39:20 it's the,
39:20 it's the pain of not seeing the solution which actually makes people open up
39:25 to the awareness that there might be something that I'm not quite seeing.
39:30 And the second thing is that we've discovered that,
39:32 well,
39:33 there's no surprise,
39:33 uh,
39:34 that
39:34 the many times government,
39:37 uh,
39:37 people at the front line are very frustrated.
39:40 Because they have gone there with uh,
39:42 with the,
39:43 for,
39:43 for doing public service and they find that
39:46 the
39:46 environment where they are working is one where it's very bureaucratic,
39:50 very difficult to innovate,
39:51 to take risks,
39:53 and so
39:54 making them our allies in that,
39:56 uh,
39:57 so that they can actually deliver what
39:59 they were even sometimes asked to deliver but are unable to do so because of the
40:04 old bureaucratic system which doesn't allow for the innovation.
40:08 Yeah,
40:09 I wanted to add as well.
40:10 So,
40:10 I think the,
40:12 the really persistent social problems and I think intimate partner violence
40:15 is an example of it is where often mental models are,
40:20 are really driving uh the problem.
40:23 Uh,
40:23 it's also where coalitions for change
40:26 become
40:26 very important,
40:27 and a lot of the awareness movements,
40:30 collaboration movements,
40:32 uh,
40:32 to raise awareness often are the interventions that try and
40:36 change,
40:37 uh,
40:37 how you think about,
40:38 you know,
40:39 You know,
40:39 some of these mental models and sometimes
40:41 it's,
40:42 uh,
40:42 it's a shock,
40:43 uh,
40:44 an incident or an
40:45 external shock that,
40:47 that,
40:47 that is like COVID-19
40:49 that really changed how we think about interacting in meetings like,
40:53 like this meeting,
40:55 um,
40:56 that often
40:57 feels very disruptive in the beginning,
40:59 but is the groundwork
41:01 for rethinking and changing mental models and
41:04 throughout history there have been a lot of shocks.
41:06 That made us reassess how,
41:08 how we,
41:08 how,
41:08 how we think about the way the world should be.
41:10 So I think this is a moment like that.
41:13 We're living in a moment like that.
41:15 I think we have a few questions.
41:17 How many,
41:17 uh,
41:17 we have one question online.
41:18 One question online.
41:19 I think that on this topic something interesting is
41:22 you said at the beginning systems are designed to reproduce themselves in a way,
41:25 right,
41:26 not to be changed.
41:27 And then we,
41:28 this is the World Bank,
41:29 so it's an institution where a lots of economies uh exist and the,
41:32 the,
41:33 the tendency is to say,
41:34 well,
41:35 people will change if you provide the right incentives
41:38 and what we are saying is something is something different.
41:40 It's not just about incentives,
41:43 uh,
41:43 because changing your mental model is not about an incentive.
41:46 It's about the realization.
41:47 Maybe it's about,
41:48 uh,
41:48 maybe at that moment it's something that goes against
41:51 everything that you think and thought during your life
41:54 until you want to do it.
41:55 So
41:56 what's in there for you?
41:56 I mean,
41:57 it's suicidal to your identity in a way,
41:59 and you still do it.
42:01 So
42:02 yeah,
42:02 we're all economists,
42:03 the four of us,
42:06 and,
42:06 and I think,
42:07 I think it recoveries,
42:08 yeah,
42:11 there's,
42:11 there's some problems which are easy to fix with incentives,
42:14 but for us the most interesting ones are those wicked ones that are not,
42:18 yeah.
42:19 OK,
42:19 very good.
42:20 So who is in line?
42:21 Perfect.
42:21 Linnea,
42:21 do you wanna come in,
42:22 please?
42:25 Yes,
42:25 thank you so much.
42:26 Thank you for fantastic presentations.
42:28 It's such an,
42:29 such an interesting topic.
42:31 I am,
42:31 um,
42:33 I have many questions,
42:34 but the question I wanted to pose now
42:36 was,
42:36 uh,
42:37 being realistic about the resources you need to conduct these kind of mappings.
42:41 What,
42:42 can you give us an idea of
42:43 The,
42:44 um,
42:44 the
42:45 amount of time needed,
42:46 kind of the team needed,
42:48 the process,
42:49 uh,
42:49 the resources,
42:50 the,
42:50 just to give us a very practical
42:53 lay of the land,
42:54 what,
42:54 what is needed to do this,
42:55 please.
42:56 Thank you.
42:57 Thank you.
42:57 And maybe we collect any other remaining question before
43:00 we go to the next part of the session.
43:02 Uh,
43:03 I would have a question.
43:03 So you showed us two examples that were really kind of in the social sector.
43:08 Um,
43:08 I mean,
43:08 I come from governance and we're in
43:10 love with institutions and institutions I think.
43:12 Do you have any examples in the book or maybe
43:15 from the top of your head just kind of mentioning
43:16 working within public sector,
43:19 within,
43:19 for example,
43:20 uh,
43:20 a tax administration unit or,
43:22 um,
43:23 you know,
43:23 installing a new system in the financial.
43:25 Ministry,
43:26 for example,
43:27 and just,
43:28 just to add one more
43:29 is where is that you see this more useful in sector-specific problems or
43:34 horizontal problems like a fighting hunger,
43:37 right,
43:37 like in Brazil did a few years,
43:39 gender issues,
43:40 climate change where you need to actually work
43:42 across the,
43:43 the vertical sectors that governments have.
43:45 So I,
43:45 I'll answer that and Michael,
43:47 I'll ask you to talk about.
43:48 The Gates grant that we're working on convergence,
43:51 um,
43:52 so because I think the,
43:53 the Gates grant answers both the question on resources and the question on,
43:58 uh,
43:59 examples of working within government,
44:01 uh,
44:01 but in terms of,
44:02 of how it's useful in the sector,
44:04 so
44:05 it is
44:05 very interesting how to see
44:07 that our water and sanitation problem
44:11 has
44:11 A lot of links with other sectors
44:14 or I remember when we were working on infant mortality in Bolivia,
44:17 one of the biggest breakthroughs was to say
44:20 what really matters.
44:21 And so even though you go in a sector,
44:24 the wicked development problems usually have
44:27 a very broad intersectional connection.
44:30 So
44:30 I think,
44:31 um,
44:32 yeah,
44:32 it,
44:33 it's very useful.
44:35 Yeah.
44:37 Uh,
44:37 we find you can tailor
44:39 what you do
44:40 to the context that you have,
44:42 and
44:42 the examples that,
44:43 um,
44:44 Josie gave,
44:45 or we're working,
44:46 we're going to illustrate that for those who stay later,
44:48 that actually
44:49 doing the system mapping,
44:51 you can actually do it in one workshop
44:53 and already get insights.
44:55 At the other extreme,
44:58 we are in a,
44:59 in a
45:00 project,
45:00 which is actually a three-year project where Gates is supporting
45:03 an evaluation,
45:04 adapt,
45:05 we've got an adaptive evaluation
45:06 of a very complex intervention.
45:09 Which is trying in
45:10 two states of India
45:12 to
45:13 activate
45:14 two huge programs that the Indian government has.
45:17 One about building
45:18 women's groups from top down,
45:20 from the bottom up,
45:21 and one about um
45:23 parenting employment,
45:24 to bring them together to work differently.
45:26 That's a case
45:27 where you've got
45:28 wonderful goals
45:30 and a very complex system which is very
45:32 resistant.
45:33 Now
45:33 that is
45:34 a
45:35 3 to 4 year program with a lot of money.
45:38 Um,
45:38 but both as a what,
45:39 what,
45:39 what we'd like to give a signal,
45:41 uh,
45:41 many,
45:41 many years ago I worked for many years in the World Bank.
45:44 And I,
45:44 and
45:45 so I have a sense of the kind of
45:47 very varied resources that you have
45:49 when you're working on a project,
45:50 when you're working on an analysis.
45:52 And this,
45:53 this is either
45:54 no more
45:55 nor less.
45:56 It's coming in with a different mentality
45:59 and with different tools,
46:01 which takes you to
46:02 different questions.
46:03 But to the Recifu project
46:06 we were working on for.
46:07 Over 2 years
46:09 with
46:10 it wasn't
46:11 that expensive because we were working
46:13 with an existing,
46:15 existing groups,
46:16 working with the governmental system,
46:18 uh,
46:18 yes,
46:18 tracking the stuff,
46:19 having
46:20 working in 9 schools to get insights in the system.
46:23 The Tunisia project
46:25 we worked on
46:26 over
46:27 6 months,
46:28 but again it wasn't.
46:30 Very substantial
46:31 financial resources,
46:33 but we were doing a lot
46:35 that required resources of
46:37 the
46:37 working with the UNDP groups.
46:40 Some
46:40 understood the ideas,
46:41 some didn't understand them at all,
46:43 and we were doing some,
46:44 a lot of design work as we went along.
46:46 So it was 6 months,
46:47 fairly,
46:48 uh,
46:48 not by far from full-time,
46:50 but,
46:50 um,
46:50 uh,
46:51 quite,
46:51 quite a lot of,
46:52 quite a,
46:53 quite intensive work to build that and to build the understanding.
46:56 If we did it again,
46:56 it'd be much quicker.
46:58 Well,
46:59 you could also,
46:59 in addition to for project preparation where you know the World
47:02 Bank will need to for a project of the bank,
47:04 let's say,
47:05 right,
47:05 uh,
47:05 come up with the,
47:06 with the resources
47:07 you can
47:08 eventually build this up into the project design
47:13 totally and then it's going to be paid by the,
47:15 by the loan proceedings,
47:16 so by the government itself,
47:18 so then the resources become less.
47:19 It's important,
47:20 uh,
47:20 yes,
47:21 good.
47:22 All right,
47:23 yes,
47:23 one more question.
47:24 Yes,
47:24 um,
47:25 so I was curious if you could talk a little
47:27 bit about how to institutionalize some of these things,
47:30 right?
47:30 So many parts of the system sometimes are very person dependent,
47:34 so you're there,
47:35 you talk to them,
47:36 or even project dependent,
47:38 so you map for one project,
47:39 but then.
47:41 Have you seen anything,
47:42 you know,
47:42 once the bank leaves,
47:44 once,
47:44 once the project ends,
47:46 how are many of those things that are advanced
47:48 in one project actually institutionalized for the next project,
47:51 so that a lot of these
47:53 things,
47:53 uh,
47:54 are
47:55 repeated within the government.
47:56 And maybe that,
47:57 I know it's a
47:58 very hard question,
47:59 um.
48:01 Uh it is a hard question and we've experienced that a lot.
48:04 It's no different from,
48:06 um,
48:07 we work a lot in
48:08 India.
48:09 And you get a reform initiative and the bureaucrat leaves and it stops.
48:12 It's no different from that as a general problem
48:15 for
48:16 any of the stuff that we're engaged with.
48:18 How you do it is actually
48:20 would be
48:21 building the practices and processes
48:24 within the governmental system or within the,
48:26 within the non,
48:27 the non-governmental system.
48:29 So
48:30 one of the reasons that we emphasize that
48:32 you can use this and take it to something very standard,
48:35 which is the log road.
48:36 This is a way of getting a better log frame,
48:38 and working with that log frame
48:40 is that will be a way of kind of building into standard project preparation
48:45 and monitoring
48:47 how to do this,
48:48 which is also why
48:49 increasingly,
48:50 when we're involved in the follow-up,
48:52 we try to work with administrative systems.
48:55 So that the counterparts see they can use the data they're collecting
48:59 already to begin to understand
49:01 how different parts of the system
49:03 uh are
49:03 are or are not working.
49:05 Yeah,
49:05 I think also in the case of Brazil,
49:08 uh,
49:08 as we were doing it,
49:09 we were also trying to do capacity building,
49:12 so it was also important that they really understood how it's gonna
49:15 be done and like why it was done in that way.
49:18 And for example,
49:19 in just in the case of Brazil,
49:20 it was very important for us that we had a local partner which was CEI which
49:24 they are in all of Brazil doing assessments and they started
49:29 in recipient and then they saw the value and they're like,
49:31 oh,
49:31 OK,
49:31 let's do the same now in Minas Gerais.
49:33 So now we are going to start doing the same Minaira which is also going to involve
49:37 government of Minas Gais and the idea is that
49:39 people from the government should.
49:41 Always
49:41 participate and co-create with that so that they see that this actually
49:46 makes sense for them in the way that they want to use it
49:49 and then they can just adapt the tools themselves because we,
49:52 we are not planning to stay there forever,
49:54 right?
49:54 Like maybe it's going to be 1 year,
49:55 2 years and then they use it and I one reason that's one reason
50:02 we haven't gone to new jobs here.
50:04 You made
50:05 mathematical
50:06 er system
50:07 analysis,
50:08 partly because that tends to obscure too much
50:11 and it's opaque
50:12 for users.
50:13 er we,
50:14 we do something that anybody can actually understand.
50:17 Uh,
50:18 I,
50:18 I'll,
50:18 I'll,
50:19 so we have a project,
50:20 uh,
50:20 in India,
50:21 uh,
50:21 which is an innovation that Sewa did for women entrepreneurs,
50:25 uh,
50:26 to move from nano-enterprises to
50:29 more,
50:29 uh,
50:29 you know,
50:30 sustainable enterprises,
50:31 and what we did was,
50:33 uh,
50:33 work on demons.
50:34 Demonstration centers so in each we're working on 3 states.
50:38 In each state we have 5 demonstration centers
50:40 and then the government is doing 100
50:43 demonstration centers and that is what we call
50:45 an intermediation function where we're there for 2-3 years
50:49 helping adapt this to the government because.
50:52 The government will have things that are less flexible than CEWA had,
50:56 but also you can tap into so many other resources than CEWA didn't have.
51:00 And so
51:01 there we have a large team in place in each of these three states
51:06 working with the government to adapt it,
51:08 even some of the people we're hiring.
51:10 are then going to be in the government when the project finishes
51:13 and so that's the only way that you can really scale through to 8 million women
51:19 from,
51:19 you know,
51:20 a very small,
51:21 in the end it's a small grant from Gates
51:23 relative to the impact that it's going to have,
51:25 yeah,
51:26 very quickly,
51:26 so I think.
51:28 I think it's,
51:28 it's very,
51:29 I wouldn't underestimate how difficult that is.
51:31 That's like a very difficult challenge.
51:33 But,
51:33 um,
51:34 uh,
51:34 in the book,
51:35 we have some examples also.
51:37 So one of the examples we,
51:38 the famous examples is of the cash transfers in Mexico.
51:42 And I think the lessons from there for me were about
51:46 how they used evidence as a way to convince a lot of stakeholders,
51:50 but also if
51:51 If you're able to demonstrate something works and people have,
51:54 you have support of the people for it,
51:56 I think that also,
51:57 and that,
51:59 so that,
51:59 so that's one element is community support evidence,
52:02 and I think the third element is trying to
52:05 make it embedded in the government system such that the administrative
52:08 and bureau bureaucratic processes are simpler and easier so that it.
52:12 It's not something,
52:13 it's not a,
52:14 like a virus attacking the government system.
52:17 It's embedded in the government system,
52:18 which makes it easy for future political parties
52:22 to just,
52:23 you know,
52:23 have it,
52:24 maybe rebrand it,
52:25 but still have,
52:25 you know,
52:26 the,
52:26 the core of the program.
52:27 Um,
52:28 thank you.
52:28 So we are almost ready to go to the second part of this,
52:31 uh,
52:31 session.
52:32 Artur,
52:32 anything you want to say before we go to the
52:34 hands-on exercise?
52:36 Really happy to have you here.
52:38 Very good.
52:39 Thank you.
52:40 And those of you who are connected,
52:41 thank you for connecting so far.
52:43 We are now going to disconnect and,
52:45 and move on to,
52:46 to the next part of the session.
52:47 Thank you.
52:47 The,
52:48 the recording of the session is also gonna be on the website,
52:50 the questions for
52:52 for website with all of our blogs and resources as well.
52:54 So feel free to contact us.
52:56 Thank you.
52:57 All right.
52:57 OK.
52:58 Second part.
53:00 Uh,
53:00 we should predict him.
53:02 OK,
53:03 yes,
53:04 um.
53:07 Like you want to transition?
53:09 Yes,
53:10 so,
53:11 I mean,
53:11 we're gonna
- add-style
- lp-body-content
On June 10, 2024, the World Bank’s Coalitions for Reforms (C4R) team, in collaboration with IMAGO Global Grassroots, supported by the World Bank’s Governance & Institutions Umbrella Program, organized an event titled “Scaling Up and Systems Change – A Webinar and Interactive Clinic on Building Coalitions.” This event delved into the concepts and tools necessary for building coalitions aimed at systems change. It included discussions on the theory behind systems thinking in the context of scaling up, supplemented by a practical case study using system diagnostic tools. (View Presentation).
The event featured insights from Isabel Guerrero, Siddhant Gokhale, and Jossie Fahsbender, co-authors of the book “Scaling Up Development Impact.” This book draws on a decade of experience in aiding grassroots organizations to scale their impact, integrating learnings from a Harvard Kennedy School course. By merging concepts from economics, psychology, systems thinking, design thinking, and adaptive evaluation and management, the book offers a comprehensive framework for scaling up with practical relevance.
Explore further resources about the book:
- Scaling Up Development Impact Book
- Book Excerpt at Stanford Social Innovation Review
- Book Launch with the Center for Global Development
- Book Launch with the Scaling Up Community of Practice
About IMAGO
IMAGO Global Grassroots works with organizations around the world to build capacity and scale impact while taking into account the complexities of the different contexts and systems in which they operate.
SPEAKERS
MODERATOR
ROBY SENDEROWITSCH, Practice Manager for Public Administration, Governance Global Practice, World Bank
SPEAKERS
ISABEL GUERRERO, Co-Founder and Executive Director, IMAGO Global Grassroots, Harvard Kennedy School
SIDDHANT GOKHALE, Knowledge Manager, IMAGO Global Grassroots
JOSSIE FAHSBENDER, Program Manager, IMAGO Global Grassroots
SPEAKERS’ BIOS
Isabel Guerrero is an economist, a psychoanalyst, and the co-founder of IMAGO Global Grassroots, which helps organizations scale up. Since 2014, she has taught the course “Scaling Up and Systems Change for Development Impact” at the Harvard Kennedy School of Government. Prior to this, Isabel worked for 30 years at the World Bank, where she had several country director positions, including Bolivia, Ecuador, Peru and Paraguay, and later Mexico, Colombia, and India. Then, she was Vice-President for the South Asia region for five years. She holds a master’s degree from the London School of Economics and graduated from the Washington Psychoanalytic Institute.
Siddhant Gokhale is a knowledge manager at IMAGO Global Grassroots, where he built the knowledge function. He co-authored a recent theoretical paper on the Adaptive Evaluation methodology and advises IMAGO’s evaluation projects. He previously worked at the Behavioral Development Lab, establishing J-PAL South Asia’s first project in the state of Goa. He holds a master’s degree in public administration and international development from the Harvard Kennedy School and an MA in economics from Columbia University.
Jossie Fahsbender is a program manager at IMAGO Global Grassroots, focused on advancing scaling up and Adaptive Evaluation projects in the Latin American and Global office. Previously, Jossie worked at the World Bank and the Inter-American Development Bank in the implementation and evaluation of projects in jobs, rural development, and agriculture. She holds a master’s degree in public administration and international development from the Harvard Kennedy School.