Module 3 Video: Improving Program Impacts in the Context of the Forcibly Displaced

videoType

dynamic-media

videoDmUrl

keyFrameImage

timestamp

00:00 Good evening,

00:00 depending on where you are.

00:02 And welcome

00:03 to the

00:04 3rd module of the Learning from the

00:06 Evidence on Forest Displacement Training Program.

00:09 This training is organized by the UK government,

00:11 UNHCR World Bank Building the Evidence on Forced Displacement Research Program

00:15 in collaboration with the World Bank UNHCR

00:17 Joint Data Center on Forced Displacement.

00:19 My name is Domenico Tavaso,

00:20 I'm a senior economist at the Joint Data Center,

00:22 and today's session is an impact evaluation,

00:25 its application to forced displacement.

00:27 The module will be moderated by Marcus Holland,

00:30 a reserve manager in the Development Impacted department of the World Bank,

00:34 to whom I will leave the floor in one minute.

00:37 But before we get started,

00:38 I would like to cover some

00:39 housekeeping rules.

00:41 Um,

00:41 this meeting is recorded.

00:44 And the recording will be posted online,

00:46 uh,

00:46 in the Building the Evidence,

00:48 uh,

00:48 Program website.

00:49 All the material will be

00:51 showing you today,

00:52 the slides,

00:53 and the list of reference will also be available on the website at the end of this,

00:56 uh,

00:57 at the end of this,

00:58 this module.

00:59 Uh,

01:00 we encourage you to leave your camera on should you wish to do so,

01:03 as it's always nicer for speakers to see some faces while they're talking,

01:08 uh,

01:08 but we also kindly ask you to keep your,

01:10 uh,

01:10 microphone mute when you are not talking.

01:13 The session will last 90 minutes.

01:15 There will be a Q&A at the end of it,

01:18 uh,

01:19 but you're also welcome to use the chat to post your questions and,

01:22 uh,

01:23 the,

01:23 our speakers will try to address your question as we go through the module.

01:27 Uh,

01:27 in case of pressing question,

01:29 uh,

01:30 we kindly ask you to wait until the,

01:32 the speaker has finished,

01:33 uh,

01:33 his presentation to pose,

01:35 uh,

01:35 to pose your question.

01:37 Feel free to introduce yourself in the chat,

01:40 mention your name and affiliation,

01:42 where you're based from.

01:43 Let's use the chat as an interactive space

01:46 and

01:47 in case you can do that,

01:48 we'd also kindly ask you to resume,

01:50 rename yourself uh in Zoom,

01:53 uh,

01:53 so as to show your name and affiliation to,

01:56 to all of us.

01:58 Again,

01:59 welcome

02:00 to all of you

02:01 to this 3rd module,

02:03 and I now leave the floor to Marcus,

02:05 uh,

02:05 to moderate this session and kick it off properly.

02:08 Marcus,

02:09 over to you.

02:12 Thank you so much,

02:13 Domenico,

02:13 and,

02:14 and welcome everybody.

02:15 Um,

02:16 great to,

02:16 great to be connected um to you,

02:18 to you all.

02:20 Uh,

02:20 before beginning,

02:21 just want to say

02:23 thank you again to the UK for financing the

02:25 World Bank UNHCR program on building the evidence on,

02:28 on forced displacement

02:29 and also a big thanks to the

02:31 Joint Data Center,

02:33 um,

02:33 as well as the World Bank's FCV Group for organizing this,

02:36 um,

02:36 specific training session.

02:38 And then,

02:38 you know,

02:39 finally,

02:39 to you all,

02:40 uh,

02:40 for joining the second session on

02:42 doing better enforcibly displaced context,

02:45 uh,

02:45 improving outcomes through trial and adopt impact evaluation.

02:48 And I realized that that's

02:49 You know,

02:50 a title that's slightly different from what was in the course materials,

02:53 and in the next few minutes,

02:54 I'll,

02:54 I'll try to motivate

02:56 uh the reason why.

02:58 Um,

02:58 next slide,

02:59 please.

03:00 No.

03:01 So just,

03:02 just to begin,

03:03 uh,

03:03 to motivate the session more broadly,

03:05 you know,

03:06 I want to say something which maybe

03:08 will sound a little bit provocative,

03:10 but uh I hope many of you will

03:12 agree,

03:12 it is actually not.

03:14 And that is that the way that we've uh traditionally designed and managed

03:18 humanitarian as well as development programs

03:21 um

03:22 is really not up to

03:24 To,

03:24 uh,

03:25 to today's challenges,

03:26 right?

03:26 So one could even go as far as to say

03:28 some of the traditional approaches to the design,

03:30 to the management of programs

03:32 are,

03:32 are obsolete.

03:34 Um,

03:34 why is that,

03:35 right?

03:35 The traditional approach is akin to,

03:38 uh,

03:39 we could say planting 1000 flowers and hoping that some will,

03:42 will bloom.

03:43 So it's one where we often work largely in isolation.

03:46 Uh,

03:46 we use preset designs that are fully decided on in advance of implementation.

03:51 There's,

03:51 there's little real-time information or updating

03:54 to guide this kind of adaptive program design,

03:57 uh,

03:57 and evaluation,

03:58 if it's done at all,

03:59 is,

03:59 is kind of an afterthought in many cases.

04:02 And so the end result here is really,

04:04 you know,

04:04 inefficient projects which perform

04:06 below,

04:07 below their potential.

04:09 Next slide,

04:09 please.

04:12 Uh,

04:12 now at DIME,

04:13 DIM is the Development Impact Group,

04:15 the Development Impact department of the World Bank.

04:18 Over the past two decades,

04:19 we've developed something that we call

04:22 trial and adopt technology,

04:24 uh,

04:24 which is a way of increasing returns to humanitarian as well as development finance

04:28 and,

04:29 and really to achieve better outcomes on the ground.

04:32 And this technology is based on

04:35 programmatic approaches and partnerships.

04:37 It embraces flexibility in design.

04:39 It puts data and digital technologies at the center of operations,

04:43 and it encompasses iterative trials

04:45 to test ideas and adapt implementation so that we can

04:49 get

04:49 the job

04:50 done.

04:51 In other words,

04:51 it allows us to test whether going left or right

04:54 or up or down gets us to better results,

04:57 and

04:58 What we have seen from this is that

05:00 employing this technology gets us more efficient

05:02 investments and higher and increasing impacts.

05:05 Next slide,

05:06 please.

05:07 And,

05:07 and so just how high,

05:08 right?

05:08 Well,

05:08 we've piloted this approach in about 160 projects,

05:12 and we found that small investments in this impact evaluation,

05:16 trial and adopt technology

05:17 can increase the impact of our investments by more than 50%.

05:22 So really,

05:23 really big,

05:24 big returns.

05:25 Uh,

05:25 next slide,

05:26 please.

05:28 We call this a

05:30 technology,

05:30 right,

05:31 not an approach,

05:31 not a model,

05:32 but a technology

05:33 because it applies a scientifically

05:35 rigorous and well-defined process to resolving

05:39 development challenges.

05:40 So we begin with investing in data

05:42 to know in what and where to invest.

05:45 We trial ideas

05:46 and adopt those which work best not only in theory,

05:49 but,

05:49 but actually in practice,

05:50 and this is really the trial and adopt part of,

05:52 of,

05:53 of the process.

05:54 Um,

05:55 we work strategically and programmatically to

05:57 mainstream learning and achieve better outcomes,

05:59 and,

05:59 and we do this with our partners in country and,

06:02 and across the world

06:04 to strengthen local capacity to manage programs and policies for impact.

06:09 Um,

06:09 and at the heart of this work really is this idea of impact evaluation,

06:13 and we define impact evaluation very specifically

06:16 as evaluation that establishes a direct

06:19 cause-effect relationship

06:20 between an intervention

06:22 and

06:22 outcomes.

06:24 Next slide please.

06:27 Um,

06:27 so today's session will focus on the intuition behind

06:30 impact evaluation and we'll showcase studies from four countries,

06:34 uh,

06:34 from Colombia,

06:34 Afghanistan,

06:35 ***,

06:35 and Bangladesh

06:37 that highlight some of the questions we can explore using,

06:41 um,

06:41 different

06:42 types of methods.

06:44 Each of these,

06:45 of course,

06:45 contributes to building the evidence on

06:46 forced displacement and beyond this to understanding

06:50 how we can design policies to improve the material and social well-being

06:54 of highly vulnerable populations.

06:57 Um,

06:57 and this actually is,

06:59 is the essence of the

07:01 program on fragility,

07:02 conflict,

07:02 and violence that DM,

07:04 that DIM runs,

07:06 where we focus on building resilience to shocks,

07:09 building human capital and sustainable livelihoods,

07:11 and improving physical

07:13 and social

07:14 connectivity.

07:16 And this program complements excellent sort

07:18 of analytical and diagnostic work done

07:22 using data from the JDC and also,

07:24 for example,

07:25 by our colleagues in the FCV Group

07:27 or in the poverty and equity Global Practice where they have just

07:30 released a series of reports on the welfare implications of forced displacement,

07:34 which I think are are recommended reading.

07:36 And our contribution within,

07:37 within this broader ecosystem is really,

07:40 you know,

07:40 the,

07:40 the what works piece.

07:41 It's understanding what works

07:43 and how

07:44 to achieve better outcomes in each

07:46 specific

07:47 context.

07:48 We do this by working with governments

07:50 and leveraging internal and external partnerships.

07:52 Um,

07:52 for example,

07:53 we're working with the World Food Program across multiple countries

07:56 to understand how can we optimize um different types of,

07:59 of WFP programming.

08:01 Um

08:03 And we'll soon be launching something called

08:04 the Global Alliance for Development Impact,

08:07 uh,

08:07 which will be a platform to promote the intensive use

08:10 of data and a scientific approach to learning to help

08:13 us reach targets such as the sustainable Development Goals,

08:17 really through more intelligent use of,

08:18 of what are ultimately very scarce resources,

08:21 and that's really what this is all about.

08:23 It's,

08:23 it's testing what works.

08:24 It's understanding how to make each policy,

08:27 each program,

08:27 each intervention the best version of itself.

08:30 Um,

08:30 so with that,

08:31 I want to thank you all again for,

08:32 for being here.

08:33 Um,

08:33 we'll have a bit of time for,

08:35 for Q&A at the end,

08:36 as,

08:36 as Domenico mentioned,

08:37 um,

08:38 but hopefully this can also be a start of a conversation with many,

08:41 uh,

08:42 with many of you,

08:42 and I look forward to that discussion.

08:44 Um,

08:45 and now I want to

08:46 hand over to my colleague,

08:47 Nosheen.

08:48 Uh,

08:48 Nosheen is a research analyst at DIM focusing on FCV gender and education,

08:53 and

08:53 she'll begin by administering a short knowledge quiz

08:56 and then take us through,

08:57 uh,

08:57 experimental and non-experimental,

08:59 experimental methods.

09:00 So,

09:00 Nosheen,

09:01 over to you.

09:03 Thank you,

09:03 Marcus.

09:04 Um,

09:05 so,

09:05 hello,

09:05 welcome everyone.

09:07 So,

09:07 I wanted to thank Marcus for the introduction

09:10 of the concept of impact evaluation,

09:13 trial and adopt technology.

09:15 But before we dive into all this interesting material,

09:19 wanted to take a quick

09:20 Temperature around the room

09:22 and really

09:23 try and gain an understanding

09:25 of what do we mean by these

09:27 concepts and methodologies.

09:30 So in the chat we have linked to a knowledge exercise.

09:34 It shouldn't take more than 5 to 6 minutes and.

09:38 It'll,

09:38 it's completely anonymous,

09:39 and we will also have a chance

09:42 at the end of the session to revisit

09:44 re revisit these questions

09:46 to see how our session has influenced

09:49 what we understand by these concepts.

09:52 So,

09:52 in the chat,

09:53 you will find a link.

09:57 And it's 9:10 right now,

09:59 so let's come back by 9:15,

10:02 and we can

10:03 move ahead with the session.

10:05 Thank you.

13:50 So it's 9:15 right now

13:53 and let's go ahead and start

13:56 with the session.

13:57 So

13:58 let's start today

14:00 thinking about a quick overview of the different types of

14:03 evaluation or analysis that we can do.

14:06 So globally,

14:07 both governments,

14:09 non-government partners,

14:11 what they're interested in

14:12 is trying to improve people's lives

14:15 and they do this through different projects,

14:17 programs

14:18 that may be related to improving people's standards of living,

14:22 improving.

14:22 Access to health,

14:24 education,

14:25 water and sanitation,

14:27 and other types of outcomes.

14:29 So if,

14:30 if we really want to know about how these programs are working,

14:34 we can use different types of analysis

14:37 and the types of questions we want to answer

14:40 really motivates or

14:42 allows us to use

14:43 which methodology will work best.

14:46 So the first one,

14:47 the.

14:47 Project M&E or process evaluation

14:50 is typically used if you want to measure project outputs,

14:53 implementation,

14:55 if you want to see things that are being delivered as planned

14:58 and

14:59 such,

15:00 then we have an impact evaluation which measures

15:04 the causal impact of a program or intervention,

15:07 which is the topic of our session today,

15:09 and we will dive into this a bit later.

15:13 We also have qualitative evaluation.

15:16 So the thing with qualitative evaluation is it provides an in-depth understanding

15:20 of the interventions process,

15:23 it delves into the why and how questions,

15:25 and really uses things such as FGDs or key informant

15:30 interview to get a more comprehensive understanding.

15:33 We also have cost benefit analysis,

15:36 which looks at

15:38 or measures the benefits of a decision or action

15:41 subtracting or minusing the costs associated with taking that action.

15:46 We really want to be able to see uh what are the benefits

15:50 after offsetting the costs.

15:52 But a lot of times it's not always possible to monetize the benefits.

15:57 In those situations,

15:59 we are able to also do a cost effective analysis,

16:02 which measures the relative costs of

16:04 outcomes of two or more

16:06 actions or interventions,

16:08 and we can compare and see which one is the most cost effective.

16:13 So,

16:13 moving on to impact evaluations,

16:16 as Marcus mentioned,

16:18 we are not just simply interested in evaluating a single interventions,

16:23 but we want to look at the program as a whole.

16:26 So when we're thinking of the trial adopt technology,

16:30 uh,

16:30 we can use this example here.

16:32 So how can we increase women's employment?

16:34 And there are various ways.

16:36 To go about this,

16:38 so we can think of this or that,

16:40 for example,

16:40 we can think of a cash grant,

16:42 we can think of a cash grant combined with training,

16:45 and then depending on what we find,

16:47 we can move in the left or right direction.

16:49 What we wanna do is to be able to scientifically measure

16:53 the impact of alternative ideas

16:55 and choose the one that works best in practice and that can be scaled up.

17:01 So

17:01 thinking about impact evaluations,

17:04 first,

17:04 let's break it down a bit and go through some of the concepts.

17:08 So outcome,

17:09 outcome is what we observe,

17:10 measure,

17:11 and want to affect.

17:12 For instance,

17:13 employment,

17:14 earnings,

17:15 revenue,

17:15 these can be considered outcomes.

17:18 Then we have impact.

17:20 So impacts are the change in outcomes that we observe.

17:23 Which are caused by the program.

17:25 So if you think employment increased by 10%,

17:30 that 10% changes the impact of the program.

17:33 We also have the cause-effect relationship.

17:36 So

17:36 when we think about cost,

17:38 we use this term because we want to explain

17:41 that any effect we find is attributed only to the program and the program itself.

17:47 So when we put all these terms and concepts together,

17:52 we can think of some research question or

17:55 what do we really want to answer?

17:57 So what is the impact of an intervention

18:00 or a set of variations of an intervention

18:03 on a set of Outcomes

18:04 and more specifically if we want to

18:07 go deep into a subject we can look at

18:10 what is the impact of a youth skills training

18:14 or skills training combined with soft skills training

18:17 on labor market job entry.

18:22 So,

18:23 thinking of the example I just highlighted,

18:26 let's see what we have in the graph here.

18:29 So,

18:29 we are interested in looking at employment outcome

18:33 before and after skills intervention.

18:35 So,

18:35 we are starting small.

18:37 So here

18:38 in the

18:38 graph,

18:39 we see

18:40 Group one,

18:41 which is receiving a technical.

18:43 Skills training,

18:44 and we see that before and after change.

18:48 So I want you all to go to the poll

18:51 EV.com/ Nashin Khan588.

18:55 This has been also posted in the chat

18:57 and answer the question,

19:00 did the intervention have a positive effect on employment?

19:07 I'll give one minute and then we can come back to the session.

19:48 So in the poll,

19:49 you can see there are 4 response options.

19:52 Did the intervention

19:53 have a positive effect on employment?

19:56 Yes,

19:56 no,

19:57 no effect or cannot conclude.

19:59 So go ahead and select the response.

20:19 OK,

20:19 we have started to get some answers.

20:27 OK,

20:28 thanks,

20:28 thanks for those who have answered the

20:30 poll.

20:31 So I see that most of you have selected the response cannot conclude,

20:36 while some people have selected

20:39 the response,

20:40 yes.

20:41 So right now,

20:41 63% of the respondents

20:44 have selected cannot conclude,

20:46 and that is the correct answer.

20:49 So why are we selecting this option?

20:52 So when we see this before.

20:54 After comparison,

20:56 we see there Group One has increased employment,

21:00 but we don't really know what's causing this increase in employment.

21:03 Is it the program

21:05 or is it because the economy is doing really well?

21:09 There's a bustling economy and there are more jobs in the market,

21:12 so now employment is higher.

21:14 So that's why

21:16 when we,

21:17 we can't just look at before and after,

21:18 but

21:19 here we have another.

21:22 graph where we look at

21:24 what do we see when we have not just the before and after comparison,

21:28 but we also have

21:30 both a comparison group who do not receive the program,

21:34 and we have group one which receives the program.

21:43 So now that

21:45 we have,

21:45 we can see both the comparison group and group one,

21:49 we can see Group One has a higher employment rate than the comparison group.

21:54 This is because

21:55 the program had a positive effect.

21:59 So now we can take this one step further.

22:03 Here we also look at employment outcomes before and after an intervention,

22:08 but here we have two variations of the interventions.

22:11 So for group one,

22:13 we see that they are receiving only technical skills,

22:17 whereas group two are receiving technical skills combined with

22:22 soft skills training.

22:23 So in this example,

22:26 I want you to answer the following question.

22:29 Between

22:30 comparison group,

22:31 Group 1 and group 2,

22:33 which has the larger effect on employment.

22:36 So,

22:37 you can also go back to the same link

22:40 and you will be able to answer this question,

22:44 which group

22:45 has a

22:46 more positive effect or larger effect on employment?

23:19 OK,

23:20 I see that people have started responding to the question.

23:24 81 to 82% of the respondents are saying they have selected group 2,

23:31 6% group one.

23:34 And a few have selected the comparison group or cannot conclude.

23:38 So overwhelmingly you have selected the right answer,

23:41 which is group two.

23:43 So from this example we can see that

23:46 between the three groups,

23:47 group 2 has the largest impact on employment,

23:51 and we can only see this because we are able to

23:54 have a comparison group

23:56 which.

23:57 Looks at

23:58 they all have the same levels of employment before the intervention

24:02 and after the intervention we are able to see the changes

24:06 and it also accounts for any other change that may be going on

24:10 so

24:11 we can use IES to learn and to get better

24:15 and.

24:16 One of the main things that we can understand from this example

24:20 is that when we're thinking about

24:22 impact evaluations and comparing different groups,

24:26 it's really important to have this comparison group,

24:29 because in our first figure,

24:31 without the comparison group,

24:32 we were not able to tell whether the program had an effect

24:36 or whether other things were going on,

24:38 which resulted in the improvement.

24:40 So,

24:41 when we're thinking about counterfactual or comparison group,

24:45 we can't simply,

24:46 it will,

24:47 it'll be really tempting to compare

24:49 those who received the program with those who don't receive the program,

24:53 but

24:54 we cannot always do so.

24:56 This is because of several factors,

24:58 because anyone who chooses or

25:01 is motivated to apply and enroll in the program

25:04 may also be more motivated to look for jobs

25:07 and more motivated to end up with higher employment.

25:11 So simply comparing these two groups will not be enough.

25:15 So you want to be able to isolate

25:17 that motivation and have a proper counterfactual group.

25:21 So in our,

25:21 so in our case of the examples of the skills training,

25:25 you want to be able to have a group of youth with similar.

25:28 Characteristics to those who are participating in the skills training program

25:33 but who do not actually end up participating

25:36 and this is the only way

25:38 that we

25:39 will not end up a situa in a situation where the estimates or

25:43 the outcomes are biased.

25:45 So

25:47 how do we go about doing this?

25:50 So in our search for a counterfactual,

25:52 we can think about two main methods.

25:55 So first is the experimental methods,

25:58 which uses random assignment to create the same groups

26:02 with the same characteristics on average.

26:05 And if you think about our graphs that we just showed,

26:08 all three levels like comparison group,

26:11 group 1,

26:11 group 2,

26:12 all three started off at the same level.

26:15 So we can use this randomization

26:18 to

26:18 Ensure that all groups have the same characteristics

26:22 at the baseline.

26:23 We also have,

26:24 and when we're not able to do randomization,

26:27 non-experimental methods,

26:30 which also try and use statistics and assumptions

26:33 to create groups with the same average preprogrammed characteristics.

26:38 But this is a bit more difficult.

26:40 So we will dive into this,

26:42 but first,

26:43 let's think about experimental methods and randomization.

26:47 So what are some steps we can do to randomize?

26:51 So first,

26:52 we choose a sample for impact evaluation,

26:55 and this is a

26:57 group of

26:58 all,

26:58 all program participants.

27:00 So if we go back to our

27:02 youth skills program,

27:03 it can be all the youth in the community.

27:06 But

27:07 we,

27:07 when we think about these program participants,

27:10 there might be different eligibility rules.

27:13 So for instance,

27:14 you.

27:14 You may be able to

27:16 uh define a program for youth who are above a certain age,

27:20 who have a certain educational background,

27:23 or who's come from a certain

27:25 vulnerability.

27:26 So,

27:27 let's look at how this looks pictorially.

27:30 So you start off with the sample of all possible program participants,

27:35 and then you have

27:37 one group

27:38 who are eligible,

27:39 like I mentioned,

27:41 and then.

27:41 They have,

27:42 they fit the age criteria,

27:43 they fit the education criteria,

27:46 they fit the vulnerability criteria or any other criteria that the program we have.

27:50 So this group on the right forms are IE or impact evaluation sample,

27:55 and then you have another group who are not eligible,

27:58 so those are not part

28:00 of our program or our study and so on.

28:03 Now,

28:04 within the IE sample,

28:05 you go about doing the randomization.

28:08 So what is randomization?

28:10 This is,

28:11 this can be done in two ways.

28:12 So you can have a public lottery,

28:15 um,

28:15 and do it physically in person,

28:18 similar to a coin flip,

28:19 or you can do it electronically through

28:22 different software.

28:23 But regardless of the method you use,

28:26 you are able to create groups with

28:29 the same characteristics

28:31 through randomization.

28:32 So in the example here,

28:34 we have Village A

28:35 and

28:37 You are able to flip the coin

28:39 and then see which group they're fitting into.

28:42 So here,

28:43 village A goes into group one that's receiving the intervention.

28:47 Then you have village B and you randomize or flip a coin,

28:51 and they follow or go part of the comparison group.

28:55 So this is.

28:57 This looks quite easy in practice

28:59 and um,

29:01 but there are several ways

29:03 um why randomization,

29:05 like maybe harder

29:07 in practice to implement,

29:09 but there are

29:10 different ways to go about this.

29:12 So,

29:13 when we're thinking about randomization,

29:16 it's oftentimes you have to convince governments or

29:20 program partner or.

29:21 Implementer

29:22 who

29:22 will be doing this,

29:24 why we need to do randomization or

29:27 why,

29:27 why this is the best

29:29 uh mechanism for evaluating the impacts of a program.

29:33 So you identify the critical causal links,

29:36 but this is not just you coming in and evaluating the program.

29:40 This has to be in consultative or engaging

29:44 process where.

29:45 You

29:45 work with clients,

29:46 you work with partners,

29:48 everyone comes together for this iterative process.

29:52 So you facilitate

29:53 a discussion on the challenges

29:56 that you face,

29:56 and you want to identify possible solutions.

30:00 So you propose those possible solutions

30:02 and you test high potential or feasible solutions

30:06 through multiple variations of an intervention.

30:09 And then finally,

30:10 once you are able to

30:12 randomize and implement these programs or the variations,

30:16 you can adapt and scale up the best solution.

30:19 So when we're thinking about randomization,

30:22 we,

30:23 there are many real world situations where randomization makes sense,

30:27 and there are also some situations where it's not possible to randomize.

30:31 So let's focus on the former.

30:34 So when you're trying to justify

30:36 why you may need to randomize,

30:38 there can be several reasons.

30:40 So first,

30:41 budget constraints.

30:43 So when government or an NGO is implementing a program,

30:47 it's not It always possible to

30:49 have full coverage at the same time.

30:52 So random assignment is one way which ensures a fair and transparent process.

30:58 For example,

30:59 if you have

31:00 300 villages who are eligible

31:03 and you only have funding for 150 villages

31:07 at once,

31:07 then you can use

31:09 randomization to offer 150

31:12 for the program and 150 the comparison.

31:15 You may also have another situation

31:18 where you have.

31:20 Funding for all 300 villages,

31:23 but you're not able to implement this at the same time.

31:26 For example,

31:27 if you think of a cash transfer program,

31:30 and this is FCV context where there's not mobile money transfer available

31:35 and there's not enough government capacity to hand out these transfers

31:40 to all 300 households at the same time,

31:44 so you can also select a phase-in randomized.

31:48 where

31:49 both have the same chance,

31:51 both groups have the same chance to go first,

31:54 but you have

31:55 one group

31:56 out of the 300,

31:57 150 that gets the program,

32:00 for example,

32:00 in the first year,

32:02 and the second group

32:04 serves as comparisons in the first year,

32:06 but then ends up receiving the program in the second year.

32:10 And I won't get deep into the cases because these will be covered by

32:14 our other.

32:15 Participants,

32:16 but

32:16 there are different ways.

32:17 You also have situations

32:20 when there's limited pre-existing data,

32:22 especially in these FCV contexts,

32:25 so you have very limited evidence on which alternative works best.

32:29 So you may be able to use random assignment to pilot variations

32:33 and scale up the one that is the most successful.

32:37 And

32:37 last but not least,

32:39 it's also important to highlight here

32:41 that a pure.

32:42 Comparison group is not always possible.

32:45 So for our

32:47 dimes as Sahel work,

32:49 we have offered cash transfer to one group in the drought season,

32:54 but and then we have the

32:56 comparison group serves as the one

32:59 that receives cash transfer in the

33:02 lean season.

33:03 So it's not always comparing

33:06 those who do not receive it at all,

33:07 but you can have different mechanisms.

33:11 So,

33:11 um,

33:12 if it's not always possible to randomize,

33:15 like I mentioned before,

33:17 we can use non-experimental methods,

33:20 but these are also tools that can be used

33:23 to measure impacts in a rigorous manner.

33:26 But one thing to highlight here that sometimes these are.

33:29 Often more expensive

33:31 or it can be very uh exhaustive,

33:34 uh,

33:34 requiring

33:35 more data and tools,

33:38 so,

33:38 but they can be used to come up with rigorous impacts.

33:42 So we are um a bit short on time,

33:45 so I'll highlight these very quickly.

33:48 But we will go into these further in the case studies.

33:51 So the first is the regression discontinuity design,

33:55 then we have a difference in difference model,

33:57 and finally,

33:58 we have propensity score matching.

34:01 So what do we understand by regression discontinuity design?

34:06 So,

34:07 when we're thinking about situations when we cannot randomize,

34:11 we can think of a government social programs

34:14 like anti-poverty programs that targets households

34:18 below um given poverty index or income.

34:22 We can think of pensions where

34:24 it's targeted to population above a certain age.

34:28 Or you can have an education program where scholarships

34:32 are given to students with high scores on standardized tests.

34:36 So in all these cases,

34:38 we can see that there is one threshold

34:41 or a cutoff

34:43 which identifies whether a group is eligible or not eligible for the program

34:49 and what regression discontinuity design does.

34:53 it allows us to compare individuals at the threshold or cut-off point

34:59 because we can make the assumption that these

35:02 individuals are very similar and have the similar characteristics.

35:06 So for those individuals who are at the threshold,

35:10 we can check their outcomes

35:12 and see whether a program had positive or negative or what kind of impact we have.

35:19 Then

35:20 we look at

35:21 a difference in difference model.

35:23 So,

35:24 difference in difference is usually a tool that's used

35:28 for natural experiments.

35:30 For example,

35:30 if you have,

35:31 um,

35:32 like the COVID pandemic or Ebola pandemic,

35:36 and you wanna see

35:37 what happens to labor market outcomes

35:39 before and after an event,

35:41 um,

35:42 one of the most famous papers here

35:44 I've highlighted is a card in.

35:46 Kruger paper

35:47 which looked at the impact of an increase in minimum wage

35:51 by comparing employment in

35:53 the fast food sector in New Jersey and in Pennsylvania,

35:57 but one of the key things to remember with this model

36:00 is that

36:02 while the two groups we are comparing can be different,

36:05 they evolve similarly

36:08 in the sense that they follow a parallel trend.

36:11 So in this graph here,

36:14 We have two time periods,

36:16 T0

36:17 and

36:18 T1,

36:19 so

36:20 the impact or the intervention happens

36:23 in between T0 and T1.

36:25 So for

36:26 those that are in the comparison group,

36:29 And

36:30 those that are in the intervention group,

36:32 we can see that they have a similar trend,

36:35 but once we have an impact,

36:37 then we are able to measure

36:40 in what would have happened,

36:42 whether before or after the program.

36:47 So the third mechanism we can use is propensity score matching,

36:52 and we don't use this just by itself,

36:55 but it's usually used

36:56 as a robustness check,

36:58 and we can use it,

37:00 um,

37:00 it,

37:01 which complements our current analysis.

37:04 So what it does is matches individuals based on observable characteristics.

37:09 Before an intervention takes place.

37:11 And it,

37:12 what it does is measures the difference between the match groups

37:16 after the intervention is finished.

37:18 For example,

37:19 if you have a situation

37:21 where

37:22 you,

37:22 you have um

37:24 similar characteristics that can be observed

37:27 and you try and accumulate as much information.

37:31 as you can,

37:32 and you have um several variables.

37:35 What you can do is record all these characteristics

37:38 and be able to measure the difference on these observable characteristics.

37:43 So what it does is it allows you to

37:46 estimate the impacts in a more robust way,

37:49 and you can attribute it to a certain program.

37:53 So there are challenges with this,

37:55 um,

37:56 but it's often expensive because you need a lot of observations,

38:00 and

38:01 the more observations you have,

38:03 the better your match is.

38:05 But these are some techniques that we can use,

38:08 um,

38:08 when we're not able to randomize.

38:12 So I'll stop here.

38:14 Um,

38:14 this screen just shows

38:16 some of the case studies that will be presented today,

38:19 and Marcus also mentioned,

38:21 but we have

38:23 4 case studies,

38:25 Colombia and Afghan refugees returning from Pakistan,

38:29 conflict-blown in rural areas in ***,

38:32 as well as looking at idleness camps in Bangladesh.

38:35 And what this does is touches upon some of The impact evaluation

38:40 methodologies that I focused on

38:42 to explain what we found in these programs.

38:45 So,

38:46 thank you for your participation.

38:48 Uh,

38:48 I know there was a lot of material,

38:50 but we also have a Q&A session,

38:53 so we can go back and touch upon this,

38:56 and this will also be shared and there's a list of references

38:59 we can go back to,

39:01 to touch upon these methods.

39:03 Thank you.

39:06 Great,

39:06 thank you.

39:07 Hold,

39:07 hold,

39:08 hand it over to Marcus.

39:10 Perfect.

39:10 Thank you so much,

39:11 Nosheen,

39:12 for,

39:12 for taking us through that intuition behind impact evaluation

39:15 as well as,

39:16 uh,

39:17 providing a brief overview of,

39:19 of some of the many methods that we use,

39:22 fully realizing,

39:22 like

39:23 Nosheen said that this was a lot of material.

39:25 Typically we would cover this over several sessions and several days,

39:28 uh,

39:29 when we run workshops for,

39:30 uh,

39:30 for our clients and partners,

39:32 so.

39:33 We appreciate that there was a lot of information

39:35 to take in.

39:36 I think the most important thing to retain there is that idea of,

39:39 you know,

39:39 having a

39:40 A real valid comparison group,

39:43 um,

39:43 and there are a few different ways,

39:45 um,

39:45 through which we can create such a group so that

39:47 we can really estimate the causal impact of a program.

39:50 Um,

39:51 with this,

39:51 we're going to go over to our case studies,

39:55 um,

39:55 beginning with,

39:56 uh,

39:56 Arlen.

39:57 So Arlen is an economist at DM.

40:00 Where he focuses on fragility,

40:02 conflict,

40:02 and violence as well as health,

40:04 and he's going to present a case on the

40:07 impacts of reparations for victims of the conflict in,

40:10 in Colombia.

40:11 Um,

40:11 this is an interesting case.

40:13 Uh,

40:13 it leverages the rollout of the program

40:15 as well as multiple sources of data to understand

40:19 uh impacts across,

40:20 across multiple

40:21 dimensions.

40:22 So without further ado,

40:23 Alan,

40:23 over to,

40:24 over to you.

40:27 Thank you so much,

40:28 Marcus.

40:28 I'm gonna start sharing the screen.

40:32 So as Marcus was,

40:34 was mentioning,

40:35 um,

40:36 this is a,

40:37 an impact evaluation of a program in Colombia.

40:39 So to put it

40:40 in the wording that she was mentioning,

40:42 we are trying to evaluate the effect of an implementation

40:45 or of a program that is reparations for victims of the conflict,

40:49 and we try to look at the effects on the outcomes

40:52 which in this case are going to be outcomes

40:54 that are going to be related to the well-being

40:56 of these recipients years after receiving these reparation.

41:01 This is going to be part of what Nain was

41:04 suggesting as non-experimental methods where we have to leverage several

41:08 ingredients and we actually have to be very lucky in

41:11 the characteristics of the population to be able to accomplish

41:15 and uh the requirements of this type of

41:18 um of methodologies.

41:20 Um,

41:21 but before getting too deep into The methodologies,

41:24 let me

41:25 just give you a brief motivation of why we started to study more uh reparations

41:30 and,

41:31 and the policy question that we had in mind is,

41:33 uh,

41:33 whether

41:34 reparations can help victims of human rights violations rebuild their lives.

41:39 More than 30 countries around the world have implemented these,

41:42 uh,

41:43 reparation programs to recognize and address.

41:45 The harm suffered by

41:48 victims during war,

41:49 conflict,

41:49 and under authoritarian regimes,

41:53 and they are seen as a way to transfer

41:56 uh money to these victims to help them,

41:59 uh,

42:00 to compensate the harm suffered during these,

42:02 um,

42:02 these events and allow them to recover

42:05 and,

42:05 and,

42:06 and,

42:06 and improve their well-being.

42:08 Uh,

42:08 these reparations are thought to be

42:10 fundamental for justice and recovery.

42:13 And

42:14 uh they are thought to be really important

42:18 in particular given how most victims are usually

42:21 in very poor and vulnerable situations.

42:24 And despite this,

42:25 uh,

42:26 thought,

42:26 uh,

42:27 potential effects of these reparations,

42:29 there were,

42:30 there was unfortunately not a lot of evidence on the effect

42:33 that these reparations may,

42:34 may have on this,

42:35 um,

42:35 this population,

42:36 and there was a lot of controversial.

42:38 Discussions about whether these reparations could have an effect and

42:42 given how expensive these reparations were

42:45 and how the characteristics of these populations were most,

42:48 which are mostly in this case,

42:51 victims of forced displacement and were not,

42:54 would potentially not able to make a good investment

42:57 of this money given the characteristics of this population

43:01 caused by the victimizations

43:03 some.

43:04 Of the people that was against these reparations were claiming that

43:07 people will be too traumatized.

43:10 They wouldn't have the social networks,

43:12 they wouldn't have like the social

43:14 or family structure that would allow them to

43:16 make a productive investment of these reparations.

43:19 And because of these discussions,

43:21 there was not still like a,

43:23 um,

43:23 a well widely accepted view of the potential effect that

43:27 these reparations were having on the individuals that resist.

43:30 And part of the reason was

43:32 uh why we don't have or we didn't have like a good evidence on this is that

43:35 first there was not a case in which we had like a

43:38 good counterfactual group or a case where we had like a randomized,

43:42 um,

43:42 assignment of these reparations so we didn't have a good counterfactual

43:46 and in cases where we didn't have a good counterfactual having uh

43:50 information of these victims

43:52 and rich administrative good data about this.

43:55 Victims was scarce and we didn't,

43:58 we weren't able to

43:59 uh look at these um effects.

44:02 So what we do is that we take a look at the case of Colombia,

44:05 um,

44:06 since this was a retrospective,

44:08 uh,

44:08 analysis in which we didn't have a,

44:11 a random assignment and we and random assignment was,

44:14 uh,

44:14 not feasible in this particular case.

44:16 What we have to do is that we have to rely a lot on administrative data.

44:20 And we,

44:20 we see,

44:21 and we use information from Colombia which

44:23 committed to compensating 7 million victims of

44:26 the internal armed conflict

44:28 in Colombia,

44:29 victims of the guerrillas,

44:30 paramilitary groups,

44:32 and state forces,

44:33 and the target was to compensate these victims by 2031.

44:37 But before getting too deep into the

44:39 methodology and what we do,

44:41 Let me give you a little bit of context of what the case of Colombia.

44:44 Colombia has had the most prolonged internal

44:46 armed conflict in the Western Hemisphere,

44:49 and there has been a battle between,

44:51 um,

44:51 guerrillas,

44:52 parliamentary groups,

44:53 and state forces for more than

44:55 70 years now.

44:56 This conflict has victimized

44:58 around 9 million victims during

45:00 the last 3 decades.

45:02 That is almost 11 in 5 Colombians is a victim of the conflict,

45:07 and it has

45:08 had

45:08 important implications and devastating implications for

45:11 most of the municipalities or most of the

45:13 uh geographic coverage of the uh of the

45:17 country with more than 90% of the municipalities

45:20 suffering some type of victimization

45:22 in,

45:23 uh,

45:23 in most rural and most border areas

45:26 being the ones that are affected the most by this conflict.

45:30 When I refer to a victim of the conflict in Colombia,

45:32 I mostly refer to victims of forced displacement,

45:35 which make for 90% of the victimizations

45:38 in,

45:38 in this case and make for the largest number

45:40 of forcibly dispelled displaced population around the world.

45:44 Uh,

45:45 but I also refer to victims of murder and,

45:47 and,

45:48 uh,

45:48 relatives of people that were forced disappeared

45:51 and among other types of victimizations,

45:53 but most of these victims are

45:55 victims of displacements.

45:57 About the time for these victimizations,

45:59 most of these victimizations occurred

46:01 in,

46:02 in,

46:02 in the early 2000s,

46:04 and the government

46:06 of Colombia has tried to deliver some type of

46:09 uh compensation of humanitarian aid to these victims.

46:13 And in order to provide this humanitarian aid in 1997 and they started.

46:17 Collecting victims,

46:18 uh,

46:19 uh,

46:20 information through the victim's registry,

46:22 which is one of the main tools or one of the main sources of information that

46:27 we used to be able to collect information about these victims.

46:31 In 20,

46:31 in 2008,

46:33 the government started compensating some victims

46:35 whose relatives have been murdered.

46:37 Uh,

46:38 during the conflict,

46:39 but importantly,

46:40 as part of the transition into peace in 2011,

46:43 uh,

46:43 in 2011,

46:44 Colombia introduced the victims' law,

46:47 a victims law that introduced reparations to the victims of the conflict,

46:51 and,

46:52 and this victim law

46:54 targeted to compensate the 7 million victims that I was referring earlier.

46:58 Um,

46:59 so far,

46:59 uh,

47:00 this,

47:00 under this victims law,

47:02 uh,

47:02 more than 1 million victims have been compensated.

47:05 These

47:05 compensations or these reparations are substantially large than make for

47:09 around 3 times the annual household income for these

47:12 recipients.

47:13 And importantly for us,

47:14 we can rely on rich administrative data

47:17 to do

47:18 a definitive analysis.

47:20 We have information on the 7 million people

47:22 that are eligible to receive the compensation.

47:24 We have information on

47:26 The 1 million individuals that have received the reparation,

47:30 and we have panel data,

47:31 which means that we can

47:33 follow individuals

47:35 on administrative data before and after they received the reparation.

47:39 So we can use

47:41 as comparison groups,

47:42 both individuals that are eligible to receive

47:45 the reparation and haven't received it,

47:47 and individuals that have received it,

47:49 and we can compare those individuals before and after receiving the reparation.

47:53 And we have information about several aspects of these

47:57 victims' lives.

47:58 We are able to provide information about and and results about

48:01 their work and living standards.

48:03 We are able to uh take a look on what are the health effects of these reparations,

48:07 and we are also able to take a look on what are the effects on the

48:10 next generation's human capital accumulation,

48:13 uh,

48:14 which is important to understand what is the effect on the,

48:16 uh,

48:17 children of these victims.

48:18 When we take a look at the effects.

48:21 Importantly,

48:22 in this graph,

48:22 we have the differences between those that received the reparation

48:26 before and after receiving the reparation.

48:28 On the left-hand side of this graph,

48:30 we observe the difference

48:32 before receiving the reparations.

48:34 And luckily for us,

48:35 like in this case,

48:36 what we observe is that

48:37 There are no differences

48:39 before they receiving the reparations which makes

48:41 in this case for a good counterfactual.

48:43 And what we see on the right are the differences between those

48:47 that receive the reparation and those that don't receive the reparation.

48:51 In this case,

48:52 we're talking about the effects on formal employment.

48:54 So what we observe here is that since these

48:57 lines here are distant from zero and they don't intersect with zero,

49:02 we we analyze this as a negative effect in this case.

49:05 So victims one quarter after receiving the reparation

49:09 are less likely to be working in the formal sector.

49:12 And this is an effect that attenuates over time

49:15 such that 3 years after receiving the reparation,

49:18 this coefficient,

49:20 the last coefficient is very close to 0

49:23 and it's actually

49:24 Um,

49:25 uh,

49:25 smaller than the one that we had before,

49:27 such that 3 years after receiving the reparation,

49:30 victims are 1.6% less likely to be working in the formal employment.

49:35 This effect is driven

49:37 mainly by

49:38 young workers,

49:39 uh,

49:39 shifting out of low paid and risk employments.

49:42 So when we take a look at what is happening to

49:44 these victims that are not working under the formal employment,

49:47 what we observe is that

49:48 they are actually taking more time

49:50 using the money as a better outside,

49:53 um,

49:53 um.

49:54 Outside resource

49:55 to look for better jobs,

49:57 to get better jobs,

49:59 such that 3 years after receiving the reparation,

50:02 they're actually working for better paid jobs

50:05 that are less risky

50:06 and that allow them to have larger incomes.

50:08 And not only that,

50:09 when we take a look at business creation

50:12 and other type of results,

50:13 what we find is that they are also creating more businesses.

50:16 They're able to consume and buy more land.

50:18 They're able to pay off their their old debts,

50:21 and they're able to buy more durables over time.

50:24 These effects

50:26 add to a,

50:27 a,

50:27 a positive effect on health outcomes such

50:29 that victims 3 years after receiving the reparation

50:33 are less likely to have an emergency,

50:35 emergency department visit,

50:37 are less likely to have a hospitalization which we read

50:40 as a positive effect on health.

50:42 And not only that,

50:43 the

50:44 children of the victims are more likely to have

50:47 a,

50:47 a undergraduate enrollment and they are also performing better in the high,

50:52 high school exit exam,

50:53 which

50:54 Makes for a case of victims invest in this money not only

50:57 on their consumption,

50:59 not only on their businesses,

51:00 but also

51:01 on their children's education,

51:02 which makes for potential

51:04 better and longer term outcomes.

51:06 So

51:07 as key messages with rich administrative data

51:10 and a,

51:11 a,

51:11 a potential feasible design and

51:14 could allow for differences in differences,

51:16 estimations that could allow for

51:17 uh uh identifying the causal effect of victims.

51:20 In this case,

51:21 we find that reparations improve victims.

51:23 Lives in several outcomes

51:26 including uh consumption and living standards,

51:28 improving their health,

51:30 their health,

51:30 and also improving their

51:32 outcomes for the next generation.

51:34 In forthcoming exercises,

51:37 try to look at the effects on other outcomes and also

51:39 uh using an RCT to see what happens when we accompany these reparations

51:44 with other variations of the programs including

51:46 psychological workshops and investment advising,

51:49 uh,

51:49 workshops and

51:51 that's

51:51 all that I have for today.

51:55 Thank you so much,

51:56 Arlen,

51:56 for sharing these uh powerful and important results.

51:59 Uh,

52:00 we're going to go right to the next case study.

52:02 I wanted to encourage all of you,

52:04 however,

52:04 if you have any questions for any of the presenters at any time,

52:09 um,

52:09 if you want to put them in the chat,

52:11 um,

52:11 we can hold on to those until the Q&A.

52:13 Um,

52:14 so please,

52:14 um,

52:15 don't hold your questions,

52:16 just,

52:16 just go ahead and,

52:17 and stick them in the chat.

52:18 Uh,

52:19 with that,

52:19 I want to turn it over to Paula,

52:21 who is an impact evaluation specialist in the World Bank's fragility,

52:25 conflict and Violence Group

52:26 and is one of the engines,

52:28 I think,

52:28 behind the building evidence on force displacement program.

52:32 Paula's going to

52:34 present to us a non-experimental study

52:38 again,

52:38 which looks at

52:40 The impacts of changing the size of the cash transfer provided to Afghan returnees,

52:45 um,

52:45 using a regression discontinuity approach.

52:48 So with that,

52:48 Paula,

52:49 over to you.

52:51 Thank you,

52:51 Marcos.

52:52 I'll uh uh share my screen now.

52:55 And uh yes,

52:56 as you mentioned,

52:57 this is uh uh part of the Building the Evidence Program.

53:00 This study is part of the Building the Evidence Program.

53:03 And um it,

53:04 it is a result of a collaboration between the World Bank and the UNHCR

53:08 with the uh goal of building evidence on

53:11 the return experience of uh Afghan refugees.

53:14 I am presenting

53:15 um

53:16 a work that was led by my colleagues,

53:18 Hisham,

53:19 uh,

53:19 Nandini,

53:20 and uh Christina.

53:22 So,

53:23 between 2016 and 2018,

53:25 there was a,

53:26 a large scale return of um Afghan refugees uh from Pakistan to Afghanistan

53:32 and uh uh 458,000 registered with the UNHCR and 380,000

53:38 were uh were not registered.

53:40 And this is on top of uh um

53:42 uh 1.6 million returnees uh from Iran.

53:46 A data sharing partnership was set up between the World Bank and the UNHCR

53:50 with the objective of uh using uh administrative data

53:53 that was being collected to understand the location,

53:56 socio-economic,

53:57 uh,

53:57 characteristics,

53:58 livelihoods,

53:58 and needs of,

53:59 uh,

54:00 recent returnees,

54:01 uh,

54:01 to Afghanistan.

54:03 In the context of this,

54:04 uh,

54:04 uh,

54:05 data collaboration,

54:06 uh,

54:06 there was also a sudden,

54:07 uh,

54:08 programming,

54:09 uh,

54:09 change that took effect,

54:10 uh,

54:11 and became the opportunity to

54:13 Uh,

54:14 be able to rigorously evaluate the UNHCR

54:17 cash reintegration assistance provided to returnees,

54:20 uh,

54:20 using existing administrative data and,

54:23 uh,

54:23 um,

54:24 as,

54:24 uh,

54:25 Nao Xin explained,

54:26 uh,

54:26 regression discontinuity,

54:27 uh,

54:28 design,

54:28 uh,

54:28 method.

54:31 Um,

54:32 so as I mentioned,

54:33 uh,

54:34 administrative data sources were,

54:36 were used,

54:37 uh,

54:37 and this included the population profiling verification

54:40 and response data collected in Pakistan,

54:42 um,

54:43 uh,

54:43 back in 2011,

54:45 and then UNHCR's voluntary repatriation,

54:48 uh,

54:48 form.

54:48 Data that's collected as uh returnees enter after they cross the border

54:55 at collection points

54:57 and uh the UNHCR post-return monitoring surveys that

55:00 were done in 3 waves and covered,

55:02 uh,

55:03 a sample of over 4000 houses.

55:06 There,

55:06 there were some new data collections as well that were done,

55:09 um,

55:10 namely,

55:10 namely,

55:11 uh,

55:11 phone surveys

55:12 and,

55:13 um,

55:13 qualitative interviews,

55:14 uh,

55:15 interviews done with the,

55:15 uh,

55:16 10 different individuals,

55:17 uh,

55:17 during COVID and,

55:18 uh,

55:19 repeatedly.

55:20 Um,

55:21 so there were a few analytical products that came out of the

55:24 analysis of this data and the one we're focusing on in this presentation

55:29 is the impact evaluation.

55:32 Um,

55:33 so as I mentioned,

55:34 as,

55:35 um,

55:35 uh,

55:36 the UNCR has in place an unconditional cash transfer program to,

55:39 um,

55:40 returnees from Pakistan that register,

55:42 uh,

55:43 upon,

55:43 uh,

55:43 uh,

55:44 return.

55:45 And,

55:45 uh,

55:46 uh,

55:46 the amount that that's transferred,

55:48 uh,

55:48 was until

55:50 Uh,

55:50 July 1st,

55:51 2016,

55:52 about uh $200.

55:54 There was then the introduction,

55:55 some of the sudden introduction of an increase

55:58 uh that was announced by the UNHCR commissioner

56:01 uh on June 29th and took effect

56:04 was implemented on July 1st.

56:05 So,

56:06 um,

56:08 Uh,

56:08 quite suddenly and immediately after

56:10 the decision was taken

56:12 and similarly,

56:13 this increase,

56:14 uh,

56:15 remained,

56:15 uh,

56:16 effective for nine months.

56:17 Uh,

56:17 after that there was a sudden decrease,

56:19 uh,

56:20 because of budgetary constraints

56:21 and so a return to the $200 allowance.

56:25 Um,

56:26 so,

56:27 uh,

56:27 what this sudden policy change and unexpected,

56:30 uh,

56:30 made possible was the comparison between

56:33 Uh,

56:33 households that had returned just before the,

56:36 uh,

56:37 decrease,

56:37 uh,

56:38 so before April,

56:40 uh,

56:40 a specific date in April 2017,

56:43 and,

56:43 uh,

56:44 those that,

56:44 um,

56:45 um,

56:46 received the,

56:47 the lower

56:47 allowance amount because they,

56:49 they returned after,

56:50 uh,

56:51 the decrease took effect.

56:52 And since there was no way to tell when this would

56:55 happen as it was unexpected and um quite suddenly introduced,

56:59 um,

57:00 these,

57:01 uh,

57:01 these two groups,

57:02 we can think of these two,

57:03 these two groups as uh randomly receiving one or the other amount and so,

57:07 uh,

57:07 as a

57:08 comparable groups.

57:11 So this analysis was done and then uh the findings were quite uh interesting.

57:16 After 60 on average 60 months after return,

57:20 um,

57:21 the households that received um the higher uh

57:25 amount were 17% um percentage points more,

57:28 more likely

57:30 to have invested in durable assets such as a house.

57:33 Uh,

57:34 those that received the higher amount were also 30%

57:37 touch points more likely

57:39 to have obtained,

57:40 uh,

57:41 legal documentation for all household members.

57:44 Uh,

57:44 there was also different spending pattern between,

57:47 um,

57:48 those that received,

57:49 uh,

57:49 the different amounts,

57:50 uh,

57:50 in that those that received the lower amount were more likely to spend it

57:54 on immediate,

57:55 uh,

57:55 consumption needs.

57:56 So there was an impact there also on,

57:58 um,

57:59 So the size of the,

58:00 of the transfer affecting the spending patterns

58:02 and choices of households and individuals.

58:05 Um,

58:06 and instead there were no um differential impacts

58:09 on employment or enrollment of children in school.

58:12 So overall,

58:13 this impact evaluation and um this study

58:16 uh generated new evidence on uh what a larger

58:20 uh cash assistance amount can do in terms of,

58:23 uh,

58:24 uh,

58:24 facilitating,

58:25 uh,

58:25 the,

58:26 um,

58:27 The return

58:29 and

58:30 post-return outcomes for refugees returning.

58:35 Uh,

58:35 this chart,

58:36 I'll,

58:36 I'll skip over,

58:37 uh,

58:37 but shows,

58:38 uh,

58:38 the strong impacts on house ownership,

58:40 uh,

58:41 which I just mentioned,

58:42 documentation,

58:43 and,

58:44 um,

58:45 spending on,

58:46 uh,

58:46 on food items,

58:47 um,

58:48 and as well as the non,

58:50 non-impacts on,

58:50 uh,

58:51 employment,

58:52 uh,

58:52 and,

58:53 um,

58:53 um,

58:54 enrollment of children in school.

58:57 Um,

58:58 I have this graph here to show how,

59:00 uh,

59:01 uh,

59:01 yes,

59:01 that nicely shows the differences.

59:04 So,

59:04 uh,

59:05 the zero in this,

59:06 uh,

59:06 uh,

59:06 graph on the

59:08 On the right is the date in which the different so the

59:13 reduction in the cash assistance amount was introduced and so you can see that

59:17 households are returned earlier and had higher,

59:20 higher amount,

59:20 have much higher

59:21 household um ownership at 43%,

59:24 whereas those that received the lower amount are returned after the

59:29 um

59:30 uh change,

59:31 uh,

59:31 programming change,

59:32 um,

59:33 had a 17% uh um

59:36 house ownership instead.

59:37 And uh the similar lines in this next graph shows

59:40 uh what I was explaining as the non uh non-impact,

59:44 the non-differential impacts on

59:46 employment outcomes that are the same regardless of the amount received.

59:50 So to wrap up,

59:51 um,

59:52 this was,

59:52 uh,

59:52 this is an example of an impact evaluation,

59:54 uh,

59:55 employing a regression discontinuity,

59:57 uh,

59:58 method,

59:58 um,

59:59 and,

59:59 uh,

1:00:00 with the cutoff being the date the programming change was implemented.

1:00:04 Um,

1:00:04 this,

1:00:05 uh,

1:00:05 this can happen a lot and,

1:00:07 uh,

1:00:07 uh,

1:00:08 as programs are implemented there is

1:00:09 It's a sudden change and this can be

1:00:11 exploited

1:00:12 if the data is available uh to uh compare outcomes and uh evaluate the programs.

1:00:19 And,

1:00:20 uh,

1:00:21 it was nice about this evaluation as well as ours that they

1:00:24 used existing administrative data so there was no need for new data collection

1:00:28 and also,

1:00:29 uh,

1:00:30 it delivered important findings,

1:00:32 uh,

1:00:32 six months,

1:00:33 uh,

1:00:33 post-return.

1:00:34 Um,

1:00:35 on,

1:00:35 uh,

1:00:35 um,

1:00:36 what it does to receive a larger,

1:00:38 uh,

1:00:38 cash reintegration assistance amount.

1:00:41 And,

1:00:41 uh,

1:00:42 uh,

1:00:42 thanks everyone for the attention and I'll stop

1:00:44 there and I'm happy to take any questions.

1:00:49 Thank you so much,

1:00:50 Paula.

1:00:50 I think this is a,

1:00:52 a very nice illustration of how

1:00:55 we can use unexpected

1:00:57 changes in,

1:00:57 in policies

1:00:59 to actually understand uh what the real impact is.

1:01:02 So,

1:01:02 so great case.

1:01:04 I'm going to turn it over to um Chloe now.

1:01:06 Chloe is a research officer um here at DIM uh where she uh leads our

1:01:11 FCV

1:01:12 program and also works on our social protection and humanitarian portfolios,

1:01:16 uh,

1:01:16 including leading our collaboration with the World Food Program.

1:01:20 And Chloe's going to present,

1:01:21 um,

1:01:21 two cases back to back.

1:01:22 So the first is a randomized controlled trial of a livelihood support program

1:01:26 for host communities and displaced persons in,

1:01:28 in ***.

1:01:29 And this is an example of embedding

1:01:31 an impact evaluation in a World Bank-supported project

1:01:34 from the design stage on.

1:01:36 So working with the government team from design

1:01:39 to finish to understand what are the impacts of this program.

1:01:42 And after that,

1:01:43 Chloe will present a study in a program involving Rohingya refugees in Bangladesh,

1:01:48 which was designed to understand

1:01:51 the additional psychosocial value of work beyond just the,

1:01:54 the monetary benefits.

1:01:55 So,

1:01:56 Chloe,

1:01:56 over to you.

1:01:58 Thank you.

1:01:59 Um,

1:01:59 good morning.

1:01:59 Good afternoon,

1:02:00 everyone.

1:02:01 So we're heading over to ***,

1:02:02 uh,

1:02:03 after the presentations on Colombia and Afghanistan,

1:02:06 um,

1:02:07 where I'm going to present results from an impact evaluation on,

1:02:10 um,

1:02:10 entrepreneurship support component that was provided to both

1:02:13 displaced populations,

1:02:15 but host populations in

1:02:16 quite fragile areas of the country.

1:02:18 We have many people that are working on this impact evaluation,

1:02:21 including other colleagues at DM,

1:02:23 but also our partners at um Trinity College in Dublin.

1:02:28 So,

1:02:28 the context of ***,

1:02:29 I'm sure everyone knows,

1:02:30 unfortunately,

1:02:31 is a very fragile one.

1:02:34 a big part of the population lives below the national poverty line,

1:02:38 and it consistently ranks one of last in any development index,

1:02:43 including the famous HDI.

1:02:45 It's also highly exposed to conflict and climate shocks,

1:02:49 and yet,

1:02:50 regardless of this level of poverty and these recurring shocks,

1:02:53 it hosts

1:02:54 first many refugees from um neighboring countries that

1:02:57 are also themselves very much affected by conflict.

1:03:00 But in addition,

1:03:01 there's a lot of internal population movements as these shocks happen.

1:03:05 So,

1:03:05 we're in a context where we have an initially very vulnerable country,

1:03:09 um,

1:03:09 and this vulnerability is further exacerbated by these population flows.

1:03:13 And as most of you here know,

1:03:15 I suspect,

1:03:16 um,

1:03:17 until recently,

1:03:17 a lot of the research

1:03:19 that has been done

1:03:20 are on exposed impacts of displacements,

1:03:23 right?

1:03:23 So how did the displacement

1:03:25 affect potentially the host country or the populations.

1:03:28 However,

1:03:29 there's very little research that has been done.

1:03:31 Um,

1:03:32 how to design better programs

1:03:34 to improve the lives of displaced and vulnerable host populations.

1:03:38 And so this is what this initiative between,

1:03:40 you know,

1:03:40 the World Bank,

1:03:41 UNHCR funded by the UK is really trying to think through.

1:03:45 So,

1:03:45 what can be done to improve economic

1:03:47 and social outcomes of these population groups.

1:03:50 And so the project that provided us with the opportunity to

1:03:53 look a bit into this question is the World Bank funded

1:03:55 uh refugees and Host Community Support Project,

1:03:58 PARCA,

1:03:58 and it's French acronym,

1:04:00 that um is implemented by the Nigerian government

1:04:03 with input actually from,

1:04:04 from UNHCR and it aims at improving access to basic services

1:04:08 and economic opportunities for refugees,

1:04:11 IDPs,

1:04:11 and hosts alike.

1:04:13 So,

1:04:13 a very important component of this project is that it is

1:04:16 agnostic to the population it's serving.

1:04:19 It's also in Niger's most fragile regions.

1:04:22 So there are two main components of this project.

1:04:25 The first one being

1:04:26 a labor intensive public work component,

1:04:28 which provides short-term cash transfer in exchange for labor,

1:04:32 which aims at building infrastructure.

1:04:34 It targets the ultra ultra poor of these communities,

1:04:37 and again,

1:04:38 regardless of your displacements or host status,

1:04:41 anyone could have access to it.

1:04:42 The second component that I'm going to be focusing on today,

1:04:46 which is what the impact evaluation was studying,

1:04:48 is on an entrepreneurship support package

1:04:51 that was provided to

1:04:53 the slightly less,

1:04:54 or,

1:04:55 again,

1:04:55 we're still in a very vulnerable context,

1:04:57 so that has to be taken into account,

1:04:59 regardless of status,

1:05:00 and this package included both um business trainings,

1:05:04 coaching,

1:05:05 savings,

1:05:06 etc.

1:05:07 And then a $200 cash grant,

1:05:10 conditional in attending these trainings,

1:05:11 and this cash grant was to be used to invest in these

1:05:14 new entrepreneurship opportunities that were provided to the population.

1:05:19 So,

1:05:20 what will we learn by looking at the impacts of this entrepreneurship package?

1:05:24 Well,

1:05:24 first,

1:05:25 evidence has shown that similar livelihood

1:05:27 programs do provide strong and sustainable impacts

1:05:30 in a wide range of contexts.

1:05:32 So we know it works,

1:05:33 um,

1:05:34 especially in ***,

1:05:35 um,

1:05:36 which had its own adaptive social protection program

1:05:38 implemented by the World Bank and the government.

1:05:41 Which took place with similarly vulnerable populations,

1:05:44 but in slightly more stable areas of the country,

1:05:47 and that only targeted non-displaced populations.

1:05:50 And so the contribution that we hope to make is,

1:05:53 do these programs,

1:05:54 these similar livelihood programs,

1:05:56 work in very volatile and dynamic contexts where there's a lot of population flows.

1:06:01 In addition,

1:06:02 we're also going to try to see if these results differ by population groups,

1:06:06 that being displaced,

1:06:06 refugees,

1:06:07 internally displaced,

1:06:09 or hosts.

1:06:10 And

1:06:11 thinking ahead,

1:06:13 um,

1:06:14 these results and these types of studies

1:06:16 could potentially help,

1:06:17 um,

1:06:19 unfortunately with the recent coup that the country faced,

1:06:22 we,

1:06:22 we expect maybe further repercussions on these fragile population groups,

1:06:25 and maybe this study could further help them

1:06:28 development and humanitarian partners alike that are connected here,

1:06:31 prepare their upcoming response.

1:06:35 So,

1:06:35 the

1:06:36 Pera project

1:06:37 took place in the regions of Tillabiri,

1:06:39 Tawa,

1:06:40 Marradi and Difa on the left-hand side.

1:06:42 In red,

1:06:42 you see the villages that were targeted by the project.

1:06:45 Um,

1:06:46 and as you can see,

1:06:46 if you corroborate that with Aled's

1:06:48 most recent,

1:06:49 uh,

1:06:50 conflict map in the region for the last 5 years,

1:06:52 you can see that,

1:06:53 uh,

1:06:54 the

1:06:55 project is very much taking place in the most conflict prone areas of the country.

1:07:00 So,

1:07:01 how does this work in practice?

1:07:02 Going back to the first session that you saw,

1:07:05 um,

1:07:05 this impact evaluation supported the Parker project right from its design.

1:07:09 So in this case,

1:07:10 we were able to

1:07:11 join the project team as it was developing

1:07:14 the different components and thinking about how to implement and deliver them.

1:07:17 So first,

1:07:18 we focused our study sample

1:07:20 for the impact evaluation only on defiant liability,

1:07:23 um,

1:07:24 and we

1:07:24 focused on 170 eligible villages

1:07:27 to receive the entrepreneurship package.

1:07:30 Then we allocated them by lottery to either receive this package in the 1st phase

1:07:35 or in a 2nd phase,

1:07:36 which is what enabled us to create

1:07:38 these comparison groups that Noxin mentioned earlier.

1:07:41 And also it helped with project implementation logistics,

1:07:43 since not all villages could be targeted at once.

1:07:46 Second,

1:07:47 in the villages where we had more eligible poor individuals

1:07:51 than the program could target,

1:07:53 so over subscription,

1:07:54 we did an individual public

1:07:56 um lottery to decide who would get this entrepreneurship support package.

1:08:00 So it was fair,

1:08:01 it was public,

1:08:02 and then it decided who was going to get this package versus who wouldn't.

1:08:07 This is what enabled us to measure these causal impacts across

1:08:10 the individuals,

1:08:11 individuals who received the entrepreneurship package,

1:08:14 to individuals in the same communities that did not get the package,

1:08:18 and then to individuals in communities where the package had yet

1:08:21 to be delivered.

1:08:23 Moving on from uh the design phase,

1:08:26 this impact evaluation also helped

1:08:29 um

1:08:30 Pcast

1:08:31 uh implementation,

1:08:32 right?

1:08:32 So we supported

1:08:33 the project throughout its life cycle.

1:08:35 So in June 2020,

1:08:37 uh,

1:08:37 the implementing NGOs

1:08:39 worked with the communities to identify who

1:08:41 are the poor and ultra-poor households.

1:08:43 As I mentioned earlier,

1:08:44 this entrepreneurship package focused on the poor,

1:08:46 not ultra-poor.

1:08:48 Now,

1:08:49 If you look on the left side,

1:08:51 only 3% of the population is considered non-poor,

1:08:53 so

1:08:54 these are very relative uh vulnerability levels.

1:08:57 So I'm not gonna go through each step,

1:08:58 but basically what you need

1:09:00 to kind of remember is that we helped with the implementation once we assigned

1:09:04 um people um to the village lottery,

1:09:07 or to the public or individual um lottery,

1:09:10 the entrepreneurship support package was delivered.

1:09:12 We collected in an end line in June,

1:09:14 so very,

1:09:15 very recently.

1:09:16 Um,

1:09:17 and then the remaining villages received the package,

1:09:20 so everyone got support at the end of the project.

1:09:24 So what are some um

1:09:26 preliminary contributions of this impact evaluations?

1:09:29 Well,

1:09:29 first,

1:09:30 remember that this is a very fragile part of the country

1:09:33 with little existing data at conception.

1:09:35 So as the project was designed,

1:09:37 um,

1:09:37 we could have some ideas about certain points,

1:09:40 but we We weren't exactly sure.

1:09:41 So,

1:09:41 for example,

1:09:43 where are the displaced?

1:09:44 Are they everywhere?

1:09:45 Um,

1:09:45 are they more concentrated in certain communities or regions?

1:09:48 That was a big question.

1:09:50 Second,

1:09:51 a common,

1:09:52 maybe misconception is that the displaced are

1:09:54 potentially worse off than the host population.

1:09:57 Um,

1:09:58 and this could impact their capacity maybe to absorb,

1:10:00 you know,

1:10:00 the project.

1:10:02 Um,

1:10:02 and finally,

1:10:03 there could be tensions between different population groups.

1:10:05 Again,

1:10:05 we're in an area of the country that's

1:10:08 Very prone to conflict,

1:10:09 and so we could imagine that there are tensions or social cohesion issues.

1:10:13 And however,

1:10:14 what was interesting is that at baseline we got

1:10:17 Very important and slightly different information.

1:10:20 First,

1:10:21 where are the displaced?

1:10:22 So you don't need to focus too much on the graphs,

1:10:24 but

1:10:25 we saw that about 40% of our population is

1:10:28 displaced in DFA of our impact evaluation population,

1:10:32 versus only 22% in Tillaberi.

1:10:34 This is still high,

1:10:35 but this means that Tillaberi has slightly less levels of displace,

1:10:38 and on top of that,

1:10:39 Tillaberi

1:10:40 only has internally displaced and no refugees.

1:10:43 This could have impacts

1:10:45 on

1:10:46 This could have implications on the impacts of the program in these regions,

1:10:49 cause we're talking about different population groups.

1:10:53 As I mentioned earlier,

1:10:54 we imagine also potentially vulnerability differences

1:10:56 between the displaced and their hosts,

1:10:58 and thus their capacity

1:10:59 to maybe appropriate this project components.

1:11:02 However,

1:11:02 we found that

1:11:03 all population groups are extremely vulnerable,

1:11:06 regardless of whether they are hosted displaced.

1:11:08 This is just one example with food security that you see here,

1:11:11 but all our different groups barely meet the international thresholds.

1:11:17 We could say the same about mental health,

1:11:19 so this is,

1:11:19 um,

1:11:20 in terms of depression threshold,

1:11:22 all of our populations are very,

1:11:24 unfortunately,

1:11:24 very close to to these depression indicators.

1:11:28 And then in terms of tensions,

1:11:29 as I mentioned earlier,

1:11:30 there's actually quite little tension between our different groups,

1:11:33 um,

1:11:34 less than 10% reports some form of conflict,

1:11:37 and by conflict we include

1:11:39 discrimination,

1:11:40 verbal abuse.

1:11:41 Um,

1:11:41 it's still there,

1:11:42 but it's,

1:11:42 it's not

1:11:44 very different than elsewhere,

1:11:45 and so this could also

1:11:47 make us rethink a bit some of the social cohesion

1:11:49 implications that we expect to get from the project.

1:11:53 Or any other projects,

1:11:54 by the way.

1:11:55 So the key messages of this study in ***,

1:11:57 well,

1:11:58 first trial and adopt,

1:11:59 as Marcus mentioned,

1:11:59 or impact evaluation exercise

1:12:01 can really help throughout the project life cycle.

1:12:04 It's not just useful to measure pro project impacts,

1:12:07 but also it can help facilitate the project design as

1:12:10 we have to think about how to implement it,

1:12:12 especially in fragile contexts where implementation

1:12:14 and delivery is quite complex.

1:12:16 But also the data that comes in,

1:12:18 even from the beginning.

1:12:19 Beginning in this case at baseline can help us rethink interventions,

1:12:22 make the project a bit more adaptive,

1:12:24 and anticipate a bit more where we want to go.

1:12:27 Now,

1:12:27 I'm sure what people are most interested in and what are the results of this project.

1:12:31 So we have

1:12:32 very,

1:12:33 very preliminary findings.

1:12:34 As I said,

1:12:35 the endline data collection happened just

1:12:37 last June.

1:12:37 The project is closing right now and we've started our analysis.

1:12:41 But right now,

1:12:41 it suggests that this entrepreneurship package had positive effects.

1:12:44 Effects

1:12:45 on household income,

1:12:46 employment,

1:12:47 and economic activities.

1:12:48 We're fine-tuning these numbers and hope to share them with you very soon.

1:12:52 But more importantly,

1:12:53 right now,

1:12:53 we're gonna further this study a little bit more into other outcomes,

1:12:56 as we said,

1:12:57 psychological well-being,

1:12:58 vertical and horizontal trust,

1:13:00 social interactions,

1:13:02 and especially going into this breakdown by population group,

1:13:04 which is what

1:13:05 a lot of people are interested in.

1:13:07 Um,

1:13:08 so that was it on ***.

1:13:10 I'm going to,

1:13:12 uh,

1:13:12 acknowledge,

1:13:13 of course,

1:13:13 there's slides,

1:13:14 everyone will have access to them,

1:13:15 to,

1:13:15 to the donors and the people that funded this project,

1:13:18 um,

1:13:19 and because we only have a few minutes,

1:13:20 and we really want to give you time to

1:13:22 ask questions,

1:13:23 I'm gonna spend a brief minute on,

1:13:25 on Bangladesh,

1:13:26 um,

1:13:27 and the psychosocial value of work in worryinga refugee camps.

1:13:31 This work was done by um a colleague of ours,

1:13:33 Erin Kelly,

1:13:34 but uh I'll be representing her today.

1:13:38 So,

1:13:39 moving over to a different context,

1:13:40 now we're in a context of refugee camps,

1:13:42 so populations aren't mixed anymore,

1:13:45 um.

1:13:46 What motivated this study,

1:13:48 right?

1:13:48 So,

1:13:49 we know that there are refugees worldwide.

1:13:51 Everyone on this um training knows this,

1:13:54 and that 70% of them face restrictions on their right to work.

1:13:57 And though there are conventions protecting these rights,

1:14:00 most countries either

1:14:02 don't sign them or have expressed reservations,

1:14:05 which thus contributes to high incidence of poverty amongst

1:14:08 refugees because they aren't able to access employment.

1:14:12 Now,

1:14:13 in addition,

1:14:14 we know very little about the impacts of being employed on mental health,

1:14:18 though this is quite needed both um in

1:14:20 terms of displacement response as displaced populations,

1:14:24 uh we can suspect often have,

1:14:26 you know,

1:14:26 mental health um constraints,

1:14:29 but it also can provide evidence on,

1:14:31 you know,

1:14:31 other unemployment schemes such as cash for work or the universal basic income.

1:14:37 Now,

1:14:38 what is the value of employment

1:14:41 beyond its direct monetary benefits?

1:14:43 So what we want to know here is,

1:14:44 can employment help refugees,

1:14:47 um,

1:14:48 help improve refugees' mental health,

1:14:50 basically.

1:14:52 So the context is we're going to focus on being a camps in Bangladesh,

1:14:56 and Bangladesh is not party to the 1951 convention,

1:14:59 and the refugees cannot work.

1:15:01 In addition,

1:15:02 there were very little informal job opportunities

1:15:04 due to mobility constraints,

1:15:07 military checkpoints,

1:15:08 etc.

1:15:09 There's some jobs provided in camps by NGOs,

1:15:12 and this is what we're going to be looking into right now.

1:15:16 So,

1:15:17 Going into the actual design,

1:15:19 um,

1:15:20 this I impact evaluation took place in the

1:15:22 Kutupalon refugee camp that is divided in 149

1:15:27 blocks or clusters,

1:15:29 and so we use these blocks to randomly allocate,

1:15:32 um,

1:15:33 people into different groups.

1:15:34 The first one being a cash for work group,

1:15:36 so 83 blocks received

1:15:38 this opportunity to work,

1:15:40 um,

1:15:40 small labor for,

1:15:41 for,

1:15:42 uh,

1:15:42 money.

1:15:43 It was about $5 for 3 or 4 days of work per week.

1:15:47 A second group where we're gonna provide just cash,

1:15:50 not work,

1:15:51 um,

1:15:52 to people for the same amount,

1:15:53 so about $5

1:15:54 and then our comparison group

1:15:56 that did not receive cash for work or cash,

1:15:59 just a very minor incentive to respond to the survey.

1:16:03 So how do we measure the psychosocial value of work?

1:16:06 Well,

1:16:06 we're gonna compare

1:16:07 people that get cash for work,

1:16:09 so that get to work and they get money for it,

1:16:11 that are compensated to people that

1:16:14 receive nothing,

1:16:15 at least from this component.

1:16:17 And then,

1:16:18 in addition to that,

1:16:18 what we really want to understand,

1:16:20 right,

1:16:20 is what is the value of work

1:16:22 beyond its pecuniary value?

1:16:23 What is the value of work beyond

1:16:25 being

1:16:27 compensated by money.

1:16:28 What does it do for your mental health?

1:16:30 And so it's for this,

1:16:31 we're going to compare people that receive cash for work to people that

1:16:34 receive the same amount of cash,

1:16:35 but may not be working.

1:16:38 And so the type of work um is both accessible to men and women,

1:16:42 it's,

1:16:42 you know,

1:16:43 very low skilled work,

1:16:44 uh,

1:16:45 it's not too good,

1:16:46 not too bad,

1:16:46 it provides,

1:16:47 you know,

1:16:47 it's outdoors,

1:16:48 people can move,

1:16:49 it's a bit repetitive,

1:16:51 it doesn't require too much time and attention,

1:16:53 but still a bit.

1:16:54 Um,

1:16:55 it has a clear purpose,

1:16:56 but it's secondary to typical employment,

1:16:58 so it's not like being a teacher,

1:16:59 right?

1:17:00 And it's offered by these NGOs.

1:17:03 So for example,

1:17:04 it can be,

1:17:04 you know,

1:17:05 some cleaning tasks in the community,

1:17:07 uh,

1:17:08 to,

1:17:08 you know,

1:17:09 sewing classes,

1:17:10 etc.

1:17:11 so they're,

1:17:11 they're minor labor jobs.

1:17:13 So what are the results?

1:17:15 Um,

1:17:15 well,

1:17:15 we see,

1:17:16 um,

1:17:17 that work,

1:17:18 relative to no work,

1:17:19 uh,

1:17:20 and cash,

1:17:21 improves mental health.

1:17:22 So

1:17:22 it provides a 50% increase in the likelihood of not being depressed,

1:17:27 which is

1:17:28 quite big.

1:17:29 It's similar to more or less to a 44 week long psychotherapy program

1:17:33 that's currently being implemented in these camps.

1:17:36 Now what we're really interested in is what is the

1:17:38 added value of working compared to just receiving cash,

1:17:41 and so if we isolate.

1:17:44 These two groups,

1:17:45 we see that work yields significantly larger improvements,

1:17:48 up to 4 times in psychological well-being than just receiving cash.

1:17:53 It also has implications on physical health,

1:17:55 20% decrease on feeling unwell,

1:17:57 and improvements on cognitive

1:18:00 function.

1:18:02 What are the key messages?

1:18:04 Well,

1:18:04 first,

1:18:05 uh,

1:18:05 the psychosocial benefits of work exceed those in cash in these settings,

1:18:09 particularly for men.

1:18:11 Now,

1:18:11 we can suspect that it's because women are

1:18:13 a little bit less idle with,

1:18:15 you know,

1:18:15 household chores and childhood and children.

1:18:18 This is being studied right now.

1:18:19 These are just hypotheses.

1:18:21 It's very important for policy to

1:18:23 Discussions on promoting the right to work,

1:18:25 um,

1:18:26 especially in the case of refugee camps,

1:18:28 where

1:18:29 there's a lot of,

1:18:30 you know,

1:18:31 not much can happen,

1:18:32 there's not a lot of work,

1:18:33 there's a lot of idleness,

1:18:35 and scope for leisure activities is limited.

1:18:37 So this provides a,

1:18:38 you know,

1:18:38 something to do.

1:18:40 And so this study has currently been extended

1:18:42 to.

1:18:43 This hypothesis in other contexts,

1:18:44 other camps or non-camps,

1:18:46 and trying to really isolate what elements of

1:18:49 of work actually yield the greatest psychological benefits.

1:18:53 Is it

1:18:54 the fact that people are being trained in something new?

1:18:57 Is it the fact that people

1:18:58 are busy?

1:18:59 Is it because

1:19:01 they feel a sense of purpose,

1:19:02 etc.

1:19:03 And of course,

1:19:04 trying to document these impacts on intra-household dynamics,

1:19:07 um,

1:19:08 bargaining power,

1:19:08 intimate partner violence,

1:19:09 as women are also participating in these programs.

1:19:13 Um,

1:19:13 thank you.

1:19:14 I think we're gonna go into QAA Q&A,

1:19:17 I'm sorry.

1:19:18 So,

1:19:18 um,

1:19:20 I'll

1:19:21 let maybe my colleagues and myself,

1:19:23 if there are questions for me,

1:19:24 answer

1:19:25 questions.

1:19:26 I'm gonna mute myself so that

1:19:28 maybe

1:19:29 the moderator could take over.

1:19:32 Thanks so much,

1:19:33 Chloe.

1:19:33 Really interesting presentations,

1:19:34 and I think these are both good cases that show

1:19:38 why

1:19:39 integrating

1:19:40 this kind of

1:19:43 Uh,

1:19:43 analytical impact evaluation approach into the fabric

1:19:47 of the project,

1:19:48 the program from day one

1:19:49 really allows us to understand,

1:19:51 to understand and to answer a much

1:19:53 broader range of questions.

1:19:54 So when we think about what our questions will be before the program starts,

1:19:58 we can actually

1:19:59 design our data collection strategy and work

1:20:01 with program implementation in such a way that we have

1:20:03 very good quality answers to those at the end,

1:20:05 which is not always true if we wait until the program is over.

1:20:09 Um,

1:20:09 with this,

1:20:10 I want to turn it over to,

1:20:12 to Q&A.

1:20:13 Um,

1:20:14 if anybody

1:20:15 wants to come in,

1:20:17 uh,

1:20:18 preferably,

1:20:18 you know,

1:20:19 unmute yourself and,

1:20:20 and,

1:20:20 um,

1:20:21 and go ahead and speak.

1:20:22 Uh,

1:20:23 if not,

1:20:24 um,

1:20:24 you can

1:20:25 type your question in the chat,

1:20:26 and we do have a,

1:20:27 a couple of,

1:20:28 of questions to,

1:20:29 to get started that already came through in the chat.

1:20:31 First of all,

1:20:33 Um,

1:20:34 for,

1:20:35 for Paula.

1:20:38 I've just lost the question.

1:20:40 Here we go.

1:20:41 Uh,

1:20:41 Valeria Lentini from UNHCR

1:20:43 was interested in

1:20:46 How you measured mid-term or long-term effects on other indicators,

1:20:50 uh,

1:20:51 beyond the increase in transfers allowed for higher investment,

1:20:54 uh,

1:20:54 allowing for higher investment in durables.

1:20:56 Uh,

1:20:56 Valeria,

1:20:57 perhaps you can repeat the question if you want,

1:20:58 if,

1:20:59 if that wasn't clear.

1:21:00 Uh,

1:21:01 Paula,

1:21:01 if,

1:21:01 if you're able to answer this question,

1:21:03 that would be great.

1:21:04 And then,

1:21:04 um,

1:21:05 for Chloe,

1:21:05 there was a question on how,

1:21:07 uh,

1:21:08 depression

1:21:09 is,

1:21:09 is measured.

1:21:13 Thank you.

1:21:14 I already replied in the,

1:21:15 in the chat,

1:21:16 but I was thinking about the very long term in uh because we are always interested in,

1:21:21 in sustainable solutions,

1:21:23 right?

1:21:23 So maybe in,

1:21:24 in 16 months is is great,

1:21:27 but also more in the future since you found no

1:21:31 differences in,

1:21:32 in the labor market or

1:21:34 kids' enrollment.

1:21:35 So I was interested on that

1:21:37 and thank you very much,

1:21:38 very interesting,

1:21:39 the 3 presentations.

1:21:43 Thanks.

1:21:43 Uh,

1:21:44 thank you,

1:21:44 Valerie.

1:21:44 This is a great question.

1:21:46 And,

1:21:46 uh,

1:21:47 there was actually an attempt to look at the long-term effects.

1:21:52 Uh,

1:21:52 although it was really hard to reach households because in the meantime,

1:21:56 they changed,

1:21:57 uh,

1:21:57 phone numbers and so it was really hard to reach them and the sample,

1:22:02 uh,

1:22:02 of the planned sample,

1:22:03 I think less than 20%,

1:22:05 uh,

1:22:05 were reached.

1:22:06 And so it wasn't,

1:22:07 that wasn't enough data to be able to say,

1:22:10 like to say with certainty,

1:22:11 statistical,

1:22:12 um,

1:22:13 um.

1:22:14 Yeah,

1:22:15 uh,

1:22:15 power,

1:22:16 um,

1:22:16 that,

1:22:17 uh,

1:22:18 uh,

1:22:18 we were observing anything.

1:22:20 So an attempt was done and,

1:22:22 uh,

1:22:22 um,

1:22:23 yeah,

1:22:24 um,

1:22:24 it wasn't successful because of the difficulty in,

1:22:28 uh,

1:22:28 tracking,

1:22:29 um,

1:22:30 households,

1:22:31 uh,

1:22:31 in that situation.

1:22:32 Those are returned and,

1:22:33 uh,

1:22:34 specifically because of the

1:22:36 change in,

1:22:36 uh,

1:22:37 in phone numbers.

1:22:38 Um.

1:22:40 And you know,

1:22:40 this was,

1:22:41 um,

1:22:42 as I explained,

1:22:42 this study happened

1:22:44 because there was this policy change and there was

1:22:47 administrative data but it wasn't set up for the,

1:22:49 from the start

1:22:50 to look specifically at this.

1:22:52 So possibly uh questions would have been asked differently

1:22:55 or uh more of an effort would have been done

1:22:58 to uh Uh,

1:22:59 collect phone numbers and track them or like the number of the friend,

1:23:03 the neighbor,

1:23:04 uh,

1:23:04 but,

1:23:05 uh,

1:23:05 since,

1:23:05 uh,

1:23:05 the data was there and,

1:23:07 uh,

1:23:07 all the analysis was done retrospectively,

1:23:09 this was the,

1:23:10 yeah,

1:23:11 this is what,

1:23:11 uh,

1:23:12 what happened.

1:23:12 So it's not impossible to reach people on the phone,

1:23:15 but um,

1:23:15 it was,

1:23:16 uh,

1:23:16 challenging this,

1:23:17 um,

1:23:18 um,

1:23:19 situation.

1:23:21 Uh,

1:23:21 over back to you,

1:23:22 uh,

1:23:22 Marcus.

1:23:26 Thank you.

1:23:27 On the,

1:23:27 the measurement of some of the psychological indicators,

1:23:31 both in Osheen and,

1:23:31 and Chloe have,

1:23:33 have answered in the chat.

1:23:35 We have,

1:23:36 I think,

1:23:36 just a minute or two left if there's anybody else that wants to come in with a,

1:23:40 with a comment or a question.

1:23:42 And then we'll close.

1:23:47 And maybe it's

1:23:49 fine.

1:23:51 Uh,

1:23:51 no,

1:23:51 go ahead,

1:23:52 uh,

1:23:52 Valeria.

1:23:53 I am,

1:23:54 go ahead.

1:23:55 There's enough time.

1:23:56 Valeria Yanoti,

1:23:58 please,

1:23:58 then I can go because I already see.

1:24:01 Thank you.

1:24:01 Thank you.

1:24:02 Uh,

1:24:02 first,

1:24:02 I just discovered some

1:24:04 typos in my questions,

1:24:07 so please forgive me.

1:24:08 Uh,

1:24:10 one of my biggest concerns is

1:24:12 the Ecuadorian Colombian border.

1:24:15 Uh,

1:24:15 it's a very dangerous place and is

1:24:18 where the conflict is still active.

1:24:20 The Colombian conflict is still active.

1:24:23 And now

1:24:24 that

1:24:25 specific part of Ecuador has become very

1:24:28 dangerous as well because of organized crime.

1:24:31 So some refugees,

1:24:33 uh,

1:24:34 they prefer,

1:24:35 or,

1:24:35 or asylum seekers,

1:24:36 they prefer to

1:24:37 hide

1:24:38 instead of,

1:24:39 of showing themselves or to get access to

1:24:43 everything that the Ecuadorian

1:24:46 government has to offer.

1:24:47 Uh,

1:24:48 the Ecuadorian Constitution says that.

1:24:51 Regardless,

1:24:52 the um,

1:24:54 Migration status,

1:24:55 they have access to public

1:24:58 education,

1:24:58 public health,

1:25:00 but some people uh prefer not to access that

1:25:04 because they,

1:25:05 they are too close to the conflict and too close for what they have suffered.

1:25:11 So I,

1:25:11 my question is how can we reach that kind of person

1:25:15 to design programs or to know how can we

1:25:18 help as much as possible

1:25:20 and to gain trust

1:25:23 uh in Ecuador that we are not going to,

1:25:25 to disclose any information to Colombia.

1:25:29 So that,

1:25:29 that's my,

1:25:30 my concern.

1:25:31 I don't know if if it is pertinent or not for the course,

1:25:34 but.

1:25:35 Yeah,

1:25:36 thank you.

1:25:38 Thank you,

1:25:38 Valeria.

1:25:39 I think it's very pertinent and I'm,

1:25:40 I'm sorry

1:25:41 uh for missing the question that you had placed in the chat.

1:25:44 Um,

1:25:46 I'll offer a very brief answer and then turn it

1:25:48 over to my colleagues who may have more information.

1:25:52 I think this is a really important and an interesting question,

1:25:54 and I think this is

1:25:56 exactly the kind of question that we would

1:25:58 be thinking about at the outset of a program.

1:26:01 Um,

1:26:02 I am not aware

1:26:03 of studies that look specifically at

1:26:07 Different strategies

1:26:08 for encouraging persons to come forward that may

1:26:11 want to remain anonymous in other domains,

1:26:13 but I think this is exactly the kind of thing that one could think about,

1:26:17 um,

1:26:18 testing to see which sort of strategy is,

1:26:21 is the most effective in doing so.

1:26:23 Um,

1:26:24 so I think this is exactly the kind of question that,

1:26:26 that we often,

1:26:26 uh,

1:26:27 think about.

1:26:28 Uh,

1:26:28 unfortunately,

1:26:29 um,

1:26:30 I don't have a,

1:26:31 a

1:26:32 A ready answer for you there,

1:26:33 but perhaps others want to come in.

1:26:36 I may come in

1:26:37 and

1:26:38 I'm probably not being like too insightful in terms of um

1:26:43 Uh,

1:26:43 any strict evidence or rigorous evidence from an impact evaluation,

1:26:48 but there have been different discussions about the

1:26:51 way to target victims of the conflict,

1:26:53 and both because some victims want to remain anonymous and also

1:26:58 because targeting specific victims or telling a portion

1:27:01 of the population that you're not worthy,

1:27:04 the aid,

1:27:05 it can also create

1:27:07 Uh,

1:27:07 social cohesion problems and,

1:27:08 and

1:27:09 one of the,

1:27:10 um,

1:27:11 ways in which,

1:27:12 um,

1:27:13 these programs have accommodated that is targeting

1:27:17 either outcomes that are particularly relevant for

1:27:20 the refugee population rather than targeting individuals

1:27:23 or targeting demographics,

1:27:25 meaning

1:27:26 the refugees mostly go to some particular community.

1:27:29 and are in need of particular aspects of their lives

1:27:32 rather than than targeting individuals themselves,

1:27:36 go and target those territories

1:27:38 and provide those type of

1:27:41 aids that you expect refugees to be needing the most

1:27:46 like short-term shelter,

1:27:48 like

1:27:49 food security programs in.

1:27:51 Places that are very close to the,

1:27:53 to the border and,

1:27:54 and,

1:27:55 and,

1:27:55 and those type of characteristics.

1:27:57 I think there are strategies that are being

1:28:00 used,

1:28:00 but

1:28:01 there's not a lot of rigorous evidence on

1:28:03 whether to have those have like

1:28:05 any substantial um differential effects,

1:28:08 but I,

1:28:09 I would say that's

1:28:10 what we at least know from,

1:28:12 from these discussions.

1:28:18 Thank you so much,

1:28:19 Charlon,

1:28:19 and

1:28:20 I thank you,

1:28:21 Valeria,

1:28:21 for raising the,

1:28:22 the very interesting point.

1:28:23 Uh,

1:28:23 I think we are essentially at time.

1:28:25 I want to hand it back to Chloe and Nosheen,

1:28:28 um,

1:28:28 to wrap up.

1:28:28 So from my side,

1:28:29 thanks so much to,

1:28:30 to everyone for connecting for this session and thanks also to,

1:28:33 to all the participants and the organizers.

1:28:36 Uh,

1:28:36 over to you,

1:28:37 Chloe and Nosheen.

1:28:39 Thanks,

1:28:39 Marcus.

1:28:40 Um,

1:28:41 I think,

1:28:42 um,

1:28:42 we don't have that much

1:28:44 left.

1:28:44 So

1:28:45 just to conclude,

1:28:46 one of the main things is

1:28:48 now that you had a chance to go through the session and,

1:28:52 you know,

1:28:52 to get familiar with some of the topics,

1:28:56 I would encourage you to also

1:28:59 try and go back to the knowledge exercise.

1:29:01 I have

1:29:02 posted the link on the chat,

1:29:04 you know,

1:29:05 to see if there are some answers you have.

1:29:07 More clarity on

1:29:09 it would be great um if we could get those responses and like I mentioned,

1:29:13 these are anonymous.

1:29:15 In addition to this knowledge exercise,

1:29:18 we would love to have your feedback on the session today.

1:29:22 This link has also been posted on the chat,

1:29:25 so if you could

1:29:26 spend a few minutes and answer those questions,

1:29:29 it would be great.

1:29:30 And last but not least,

1:29:32 um,

1:29:33 we have

1:29:34 the link to the.

1:29:37 Materials and other information about the training,

1:29:40 you can go find the schedule and the presentations,

1:29:45 including

1:29:46 what um

1:29:47 it covers both the

1:29:49 academic papers,

1:29:50 some of which were presented today.

1:29:53 And

1:29:54 this is a series of trainings.

1:29:56 So next Wednesday,

1:29:57 keep your

1:29:58 eyes out for the force displacement

1:30:00 evidence in the health sector,

1:30:02 October 18th,

1:30:04 um,

1:30:04 at the same time.

1:30:06 So,

1:30:06 uh,

1:30:07 with these,

1:30:07 I will end here.

1:30:09 Thanks to everyone.

1:30:10 And like I said,

1:30:12 um,

1:30:12 the links are on the chat,

1:30:13 so if

1:30:14 people could just spend one or two minutes filling those out,

1:30:18 that would be great.

1:30:19 And back to you,

1:30:20 Paula,

1:30:20 for any last minute

1:30:22 closing.

1:30:27 Thank you.

1:30:27 Uh,

1:30:28 thank you now,

1:30:28 Xin,

1:30:29 and,

1:30:29 uh,

1:30:29 many thanks to,

1:30:30 to all the presenters,

1:30:31 uh,

1:30:32 today.

1:30:32 I think this was an excellent,

1:30:34 uh,

1:30:35 session.

1:30:35 It's great to see the engagement in the chat.

1:30:38 Uh,

1:30:38 I would just repeat,

1:30:39 it's important to receive your,

1:30:41 your feedback as we're in the middle of this training program and we

1:30:45 Uh,

1:30:45 I'd like to endeavor to improve,

1:30:47 um,

1:30:48 going forward and making sure these sessions are as useful

1:30:51 as,

1:30:52 uh,

1:30:52 possible to,

1:30:53 to all participants.

1:30:54 So,

1:30:55 um,

1:30:56 uh,

1:30:56 do we have the feedback form?

1:30:57 Great.

1:30:58 OK,

1:30:58 perfect.

1:30:58 I see in the chat,

1:30:59 the feedback form,

1:31:00 that includes the feedback form,

1:31:01 and also,

1:31:02 um,

1:31:03 For colleagues that you know would be interested in this,

1:31:06 uh,

1:31:06 course,

1:31:07 uh,

1:31:08 uh,

1:31:08 in the chat,

1:31:08 you can also find the link to our website and there we are,

1:31:12 uh,

1:31:13 posting the recordings,

1:31:14 um,

1:31:15 as well as all the background materials and

1:31:16 presentations that we're delivering during these sessions.

1:31:19 Finally,

1:31:20 I'll also say that there's going to be,

1:31:22 these materials will be taken and converted into an e-learning,

1:31:25 self-based in-learning course that's,

1:31:27 uh,

1:31:27 also going to be open to

1:31:29 Uh,

1:31:29 anyone,

1:31:29 the wider public,

1:31:31 uh,

1:31:31 to attend and that it will be ready,

1:31:33 uh,

1:31:33 in the beginning of the,

1:31:34 uh,

1:31:34 next year.

1:31:35 Um,

1:31:36 thanks again to everyone and,

1:31:38 uh,

1:31:38 have a nice rest of your day,

1:31:39 evening,

1:31:40 uh,

1:31:40 or,

1:31:41 uh,

1:31:41 morning,

1:31:41 wherever you are.

1:31:43 Bye.

showAllTimestamps

no

transcript

Good evening, depending on where you are. And welcome to the 3rd module of the Learning from the Evidence on Forest Displacement Training Program. This training is organized by the UK government, UNHCR World Bank Building the Evidence on Forced Displacement Research Program in collaboration with the World Bank UNHCR Joint Data Center on Forced Displacement. My name is Domenico Tavaso, I'm a senior economist at the Joint Data Center, and today's session is an impact evaluation, its application to forced displacement. The module will be moderated by Marcus Holland, a reserve manager in the Development Impacted department of the World Bank, to whom I will leave the floor in one minute. But before we get started, I would like to cover some housekeeping rules. Um, this meeting is recorded. And the recording will be posted online, uh, in the Building the Evidence, uh, Program website. All the material will be showing you today, the slides, and the list of reference will also be available on the website at the end of this, uh, at the end of this, this module. Uh, we encourage you to leave your camera on should you wish to do so, as it's always nicer for speakers to see some faces while they're talking, uh, but we also kindly ask you to keep your, uh, microphone mute when you are not talking. The session will last 90 minutes. There will be a Q&A at the end of it, uh, but you're also welcome to use the chat to post your questions and, uh, the, our speakers will try to address your question as we go through the module. Uh, in case of pressing question, uh, we kindly ask you to wait until the, the speaker has finished, uh, his presentation to pose, uh, to pose your question. Feel free to introduce yourself in the chat, mention your name and affiliation, where you're based from. Let's use the chat as an interactive space and in case you can do that, we'd also kindly ask you to resume, rename yourself uh in Zoom, uh, so as to show your name and affiliation to, to all of us. Again, welcome to all of you to this 3rd module, and I now leave the floor to Marcus, uh, to moderate this session and kick it off properly. Marcus, over to you. Thank you so much, Domenico, and, and welcome everybody. Um, great to, great to be connected um to you, to you all. Uh, before beginning, just want to say thank you again to the UK for financing the World Bank UNHCR program on building the evidence on, on forced displacement and also a big thanks to the Joint Data Center, um, as well as the World Bank's FCV Group for organizing this, um, specific training session. And then, you know, finally, to you all, uh, for joining the second session on doing better enforcibly displaced context, uh, improving outcomes through trial and adopt impact evaluation. And I realized that that's You know, a title that's slightly different from what was in the course materials, and in the next few minutes, I'll, I'll try to motivate uh the reason why. Um, next slide, please. No. So just, just to begin, uh, to motivate the session more broadly, you know, I want to say something which maybe will sound a little bit provocative, but uh I hope many of you will agree, it is actually not. And that is that the way that we've uh traditionally designed and managed humanitarian as well as development programs um is really not up to To, uh, to today's challenges, right? So one could even go as far as to say some of the traditional approaches to the design, to the management of programs are, are obsolete. Um, why is that, right? The traditional approach is akin to, uh, we could say planting 1000 flowers and hoping that some will, will bloom. So it's one where we often work largely in isolation. Uh, we use preset designs that are fully decided on in advance of implementation. There's, there's little real-time information or updating to guide this kind of adaptive program design, uh, and evaluation, if it's done at all, is, is kind of an afterthought in many cases. And so the end result here is really, you know, inefficient projects which perform below, below their potential. Next slide, please. Uh, now at DIME, DIM is the Development Impact Group, the Development Impact department of the World Bank. Over the past two decades, we've developed something that we call trial and adopt technology, uh, which is a way of increasing returns to humanitarian as well as development finance and, and really to achieve better outcomes on the ground. And this technology is based on programmatic approaches and partnerships. It embraces flexibility in design. It puts data and digital technologies at the center of operations, and it encompasses iterative trials to test ideas and adapt implementation so that we can get the job done. In other words, it allows us to test whether going left or right or up or down gets us to better results, and What we have seen from this is that employing this technology gets us more efficient investments and higher and increasing impacts. Next slide, please. And, and so just how high, right? Well, we've piloted this approach in about 160 projects, and we found that small investments in this impact evaluation, trial and adopt technology can increase the impact of our investments by more than 50%. So really, really big, big returns. Uh, next slide, please. We call this a technology, right, not an approach, not a model, but a technology because it applies a scientifically rigorous and well-defined process to resolving development challenges. So we begin with investing in data to know in what and where to invest. We trial ideas and adopt those which work best not only in theory, but, but actually in practice, and this is really the trial and adopt part of, of, of the process. Um, we work strategically and programmatically to mainstream learning and achieve better outcomes, and, and we do this with our partners in country and, and across the world to strengthen local capacity to manage programs and policies for impact. Um, and at the heart of this work really is this idea of impact evaluation, and we define impact evaluation very specifically as evaluation that establishes a direct cause-effect relationship between an intervention and outcomes. Next slide please. Um, so today's session will focus on the intuition behind impact evaluation and we'll showcase studies from four countries, uh, from Colombia, Afghanistan, ***, and Bangladesh that highlight some of the questions we can explore using, um, different types of methods. Each of these, of course, contributes to building the evidence on forced displacement and beyond this to understanding how we can design policies to improve the material and social well-being of highly vulnerable populations. Um, and this actually is, is the essence of the program on fragility, conflict, and violence that DM, that DIM runs, where we focus on building resilience to shocks, building human capital and sustainable livelihoods, and improving physical and social connectivity. And this program complements excellent sort of analytical and diagnostic work done using data from the JDC and also, for example, by our colleagues in the FCV Group or in the poverty and equity Global Practice where they have just released a series of reports on the welfare implications of forced displacement, which I think are are recommended reading. And our contribution within, within this broader ecosystem is really, you know, the, the what works piece. It's understanding what works and how to achieve better outcomes in each specific context. We do this by working with governments and leveraging internal and external partnerships. Um, for example, we're working with the World Food Program across multiple countries to understand how can we optimize um different types of, of WFP programming. Um And we'll soon be launching something called the Global Alliance for Development Impact, uh, which will be a platform to promote the intensive use of data and a scientific approach to learning to help us reach targets such as the sustainable Development Goals, really through more intelligent use of, of what are ultimately very scarce resources, and that's really what this is all about. It's, it's testing what works. It's understanding how to make each policy, each program, each intervention the best version of itself. Um, so with that, I want to thank you all again for, for being here. Um, we'll have a bit of time for, for Q&A at the end, as, as Domenico mentioned, um, but hopefully this can also be a start of a conversation with many, uh, with many of you, and I look forward to that discussion. Um, and now I want to hand over to my colleague, Nosheen. Uh, Nosheen is a research analyst at DIM focusing on FCV gender and education, and she'll begin by administering a short knowledge quiz and then take us through, uh, experimental and non-experimental, experimental methods. So, Nosheen, over to you. Thank you, Marcus. Um, so, hello, welcome everyone. So, I wanted to thank Marcus for the introduction of the concept of impact evaluation, trial and adopt technology. But before we dive into all this interesting material, wanted to take a quick Temperature around the room and really try and gain an understanding of what do we mean by these concepts and methodologies. So in the chat we have linked to a knowledge exercise. It shouldn't take more than 5 to 6 minutes and. It'll, it's completely anonymous, and we will also have a chance at the end of the session to revisit re revisit these questions to see how our session has influenced what we understand by these concepts. So, in the chat, you will find a link. And it's 9:10 right now, so let's come back by 9:15, and we can move ahead with the session. Thank you. So it's 9:15 right now and let's go ahead and start with the session. So let's start today thinking about a quick overview of the different types of evaluation or analysis that we can do. So globally, both governments, non-government partners, what they're interested in is trying to improve people's lives and they do this through different projects, programs that may be related to improving people's standards of living, improving. Access to health, education, water and sanitation, and other types of outcomes. So if, if we really want to know about how these programs are working, we can use different types of analysis and the types of questions we want to answer really motivates or allows us to use which methodology will work best. So the first one, the. Project M&E or process evaluation is typically used if you want to measure project outputs, implementation, if you want to see things that are being delivered as planned and such, then we have an impact evaluation which measures the causal impact of a program or intervention, which is the topic of our session today, and we will dive into this a bit later. We also have qualitative evaluation. So the thing with qualitative evaluation is it provides an in-depth understanding of the interventions process, it delves into the why and how questions, and really uses things such as FGDs or key informant interview to get a more comprehensive understanding. We also have cost benefit analysis, which looks at or measures the benefits of a decision or action subtracting or minusing the costs associated with taking that action. We really want to be able to see uh what are the benefits after offsetting the costs. But a lot of times it's not always possible to monetize the benefits. In those situations, we are able to also do a cost effective analysis, which measures the relative costs of outcomes of two or more actions or interventions, and we can compare and see which one is the most cost effective. So, moving on to impact evaluations, as Marcus mentioned, we are not just simply interested in evaluating a single interventions, but we want to look at the program as a whole. So when we're thinking of the trial adopt technology, uh, we can use this example here. So how can we increase women's employment? And there are various ways. To go about this, so we can think of this or that, for example, we can think of a cash grant, we can think of a cash grant combined with training, and then depending on what we find, we can move in the left or right direction. What we wanna do is to be able to scientifically measure the impact of alternative ideas and choose the one that works best in practice and that can be scaled up. So thinking about impact evaluations, first, let's break it down a bit and go through some of the concepts. So outcome, outcome is what we observe, measure, and want to affect. For instance, employment, earnings, revenue, these can be considered outcomes. Then we have impact. So impacts are the change in outcomes that we observe. Which are caused by the program. So if you think employment increased by 10%, that 10% changes the impact of the program. We also have the cause-effect relationship. So when we think about cost, we use this term because we want to explain that any effect we find is attributed only to the program and the program itself. So when we put all these terms and concepts together, we can think of some research question or what do we really want to answer? So what is the impact of an intervention or a set of variations of an intervention on a set of Outcomes and more specifically if we want to go deep into a subject we can look at what is the impact of a youth skills training or skills training combined with soft skills training on labor market job entry. So, thinking of the example I just highlighted, let's see what we have in the graph here. So, we are interested in looking at employment outcome before and after skills intervention. So, we are starting small. So here in the graph, we see Group one, which is receiving a technical. Skills training, and we see that before and after change. So I want you all to go to the poll EV.com/ Nashin Khan588. This has been also posted in the chat and answer the question, did the intervention have a positive effect on employment? I'll give one minute and then we can come back to the session. So in the poll, you can see there are 4 response options. Did the intervention have a positive effect on employment? Yes, no, no effect or cannot conclude. So go ahead and select the response. OK, we have started to get some answers. OK, thanks, thanks for those who have answered the poll. So I see that most of you have selected the response cannot conclude, while some people have selected the response, yes. So right now, 63% of the respondents have selected cannot conclude, and that is the correct answer. So why are we selecting this option? So when we see this before. After comparison, we see there Group One has increased employment, but we don't really know what's causing this increase in employment. Is it the program or is it because the economy is doing really well? There's a bustling economy and there are more jobs in the market, so now employment is higher. So that's why when we, we can't just look at before and after, but here we have another. graph where we look at what do we see when we have not just the before and after comparison, but we also have both a comparison group who do not receive the program, and we have group one which receives the program. So now that we have, we can see both the comparison group and group one, we can see Group One has a higher employment rate than the comparison group. This is because the program had a positive effect. So now we can take this one step further. Here we also look at employment outcomes before and after an intervention, but here we have two variations of the interventions. So for group one, we see that they are receiving only technical skills, whereas group two are receiving technical skills combined with soft skills training. So in this example, I want you to answer the following question. Between comparison group, Group 1 and group 2, which has the larger effect on employment. So, you can also go back to the same link and you will be able to answer this question, which group has a more positive effect or larger effect on employment? OK, I see that people have started responding to the question. 81 to 82% of the respondents are saying they have selected group 2, 6% group one. And a few have selected the comparison group or cannot conclude. So overwhelmingly you have selected the right answer, which is group two. So from this example we can see that between the three groups, group 2 has the largest impact on employment, and we can only see this because we are able to have a comparison group which. Looks at they all have the same levels of employment before the intervention and after the intervention we are able to see the changes and it also accounts for any other change that may be going on so we can use IES to learn and to get better and. One of the main things that we can understand from this example is that when we're thinking about impact evaluations and comparing different groups, it's really important to have this comparison group, because in our first figure, without the comparison group, we were not able to tell whether the program had an effect or whether other things were going on, which resulted in the improvement. So, when we're thinking about counterfactual or comparison group, we can't simply, it will, it'll be really tempting to compare those who received the program with those who don't receive the program, but we cannot always do so. This is because of several factors, because anyone who chooses or is motivated to apply and enroll in the program may also be more motivated to look for jobs and more motivated to end up with higher employment. So simply comparing these two groups will not be enough. So you want to be able to isolate that motivation and have a proper counterfactual group. So in our, so in our case of the examples of the skills training, you want to be able to have a group of youth with similar. Characteristics to those who are participating in the skills training program but who do not actually end up participating and this is the only way that we will not end up a situa in a situation where the estimates or the outcomes are biased. So how do we go about doing this? So in our search for a counterfactual, we can think about two main methods. So first is the experimental methods, which uses random assignment to create the same groups with the same characteristics on average. And if you think about our graphs that we just showed, all three levels like comparison group, group 1, group 2, all three started off at the same level. So we can use this randomization to Ensure that all groups have the same characteristics at the baseline. We also have, and when we're not able to do randomization, non-experimental methods, which also try and use statistics and assumptions to create groups with the same average preprogrammed characteristics. But this is a bit more difficult. So we will dive into this, but first, let's think about experimental methods and randomization. So what are some steps we can do to randomize? So first, we choose a sample for impact evaluation, and this is a group of all, all program participants. So if we go back to our youth skills program, it can be all the youth in the community. But we, when we think about these program participants, there might be different eligibility rules. So for instance, you. You may be able to uh define a program for youth who are above a certain age, who have a certain educational background, or who's come from a certain vulnerability. So, let's look at how this looks pictorially. So you start off with the sample of all possible program participants, and then you have one group who are eligible, like I mentioned, and then. They have, they fit the age criteria, they fit the education criteria, they fit the vulnerability criteria or any other criteria that the program we have. So this group on the right forms are IE or impact evaluation sample, and then you have another group who are not eligible, so those are not part of our program or our study and so on. Now, within the IE sample, you go about doing the randomization. So what is randomization? This is, this can be done in two ways. So you can have a public lottery, um, and do it physically in person, similar to a coin flip, or you can do it electronically through different software. But regardless of the method you use, you are able to create groups with the same characteristics through randomization. So in the example here, we have Village A and You are able to flip the coin and then see which group they're fitting into. So here, village A goes into group one that's receiving the intervention. Then you have village B and you randomize or flip a coin, and they follow or go part of the comparison group. So this is. This looks quite easy in practice and um, but there are several ways um why randomization, like maybe harder in practice to implement, but there are different ways to go about this. So, when we're thinking about randomization, it's oftentimes you have to convince governments or program partner or. Implementer who will be doing this, why we need to do randomization or why, why this is the best uh mechanism for evaluating the impacts of a program. So you identify the critical causal links, but this is not just you coming in and evaluating the program. This has to be in consultative or engaging process where. You work with clients, you work with partners, everyone comes together for this iterative process. So you facilitate a discussion on the challenges that you face, and you want to identify possible solutions. So you propose those possible solutions and you test high potential or feasible solutions through multiple variations of an intervention. And then finally, once you are able to randomize and implement these programs or the variations, you can adapt and scale up the best solution. So when we're thinking about randomization, we, there are many real world situations where randomization makes sense, and there are also some situations where it's not possible to randomize. So let's focus on the former. So when you're trying to justify why you may need to randomize, there can be several reasons. So first, budget constraints. So when government or an NGO is implementing a program, it's not It always possible to have full coverage at the same time. So random assignment is one way which ensures a fair and transparent process. For example, if you have 300 villages who are eligible and you only have funding for 150 villages at once, then you can use randomization to offer 150 for the program and 150 the comparison. You may also have another situation where you have. Funding for all 300 villages, but you're not able to implement this at the same time. For example, if you think of a cash transfer program, and this is FCV context where there's not mobile money transfer available and there's not enough government capacity to hand out these transfers to all 300 households at the same time, so you can also select a phase-in randomized. where both have the same chance, both groups have the same chance to go first, but you have one group out of the 300, 150 that gets the program, for example, in the first year, and the second group serves as comparisons in the first year, but then ends up receiving the program in the second year. And I won't get deep into the cases because these will be covered by our other. Participants, but there are different ways. You also have situations when there's limited pre-existing data, especially in these FCV contexts, so you have very limited evidence on which alternative works best. So you may be able to use random assignment to pilot variations and scale up the one that is the most successful. And last but not least, it's also important to highlight here that a pure. Comparison group is not always possible. So for our dimes as Sahel work, we have offered cash transfer to one group in the drought season, but and then we have the comparison group serves as the one that receives cash transfer in the lean season. So it's not always comparing those who do not receive it at all, but you can have different mechanisms. So, um, if it's not always possible to randomize, like I mentioned before, we can use non-experimental methods, but these are also tools that can be used to measure impacts in a rigorous manner. But one thing to highlight here that sometimes these are. Often more expensive or it can be very uh exhaustive, uh, requiring more data and tools, so, but they can be used to come up with rigorous impacts. So we are um a bit short on time, so I'll highlight these very quickly. But we will go into these further in the case studies. So the first is the regression discontinuity design, then we have a difference in difference model, and finally, we have propensity score matching. So what do we understand by regression discontinuity design? So, when we're thinking about situations when we cannot randomize, we can think of a government social programs like anti-poverty programs that targets households below um given poverty index or income. We can think of pensions where it's targeted to population above a certain age. Or you can have an education program where scholarships are given to students with high scores on standardized tests. So in all these cases, we can see that there is one threshold or a cutoff which identifies whether a group is eligible or not eligible for the program and what regression discontinuity design does. it allows us to compare individuals at the threshold or cut-off point because we can make the assumption that these individuals are very similar and have the similar characteristics. So for those individuals who are at the threshold, we can check their outcomes and see whether a program had positive or negative or what kind of impact we have. Then we look at a difference in difference model. So, difference in difference is usually a tool that's used for natural experiments. For example, if you have, um, like the COVID pandemic or Ebola pandemic, and you wanna see what happens to labor market outcomes before and after an event, um, one of the most famous papers here I've highlighted is a card in. Kruger paper which looked at the impact of an increase in minimum wage by comparing employment in the fast food sector in New Jersey and in Pennsylvania, but one of the key things to remember with this model is that while the two groups we are comparing can be different, they evolve similarly in the sense that they follow a parallel trend. So in this graph here, We have two time periods, T0 and T1, so the impact or the intervention happens in between T0 and T1. So for those that are in the comparison group, And those that are in the intervention group, we can see that they have a similar trend, but once we have an impact, then we are able to measure in what would have happened, whether before or after the program. So the third mechanism we can use is propensity score matching, and we don't use this just by itself, but it's usually used as a robustness check, and we can use it, um, it, which complements our current analysis. So what it does is matches individuals based on observable characteristics. Before an intervention takes place. And it, what it does is measures the difference between the match groups after the intervention is finished. For example, if you have a situation where you, you have um similar characteristics that can be observed and you try and accumulate as much information. as you can, and you have um several variables. What you can do is record all these characteristics and be able to measure the difference on these observable characteristics. So what it does is it allows you to estimate the impacts in a more robust way, and you can attribute it to a certain program. So there are challenges with this, um, but it's often expensive because you need a lot of observations, and the more observations you have, the better your match is. But these are some techniques that we can use, um, when we're not able to randomize. So I'll stop here. Um, this screen just shows some of the case studies that will be presented today, and Marcus also mentioned, but we have 4 case studies, Colombia and Afghan refugees returning from Pakistan, conflict-blown in rural areas in ***, as well as looking at idleness camps in Bangladesh. And what this does is touches upon some of The impact evaluation methodologies that I focused on to explain what we found in these programs. So, thank you for your participation. Uh, I know there was a lot of material, but we also have a Q&A session, so we can go back and touch upon this, and this will also be shared and there's a list of references we can go back to, to touch upon these methods. Thank you. Great, thank you. Hold, hold, hand it over to Marcus. Perfect. Thank you so much, Nosheen, for, for taking us through that intuition behind impact evaluation as well as, uh, providing a brief overview of, of some of the many methods that we use, fully realizing, like Nosheen said that this was a lot of material. Typically we would cover this over several sessions and several days, uh, when we run workshops for, uh, for our clients and partners, so. We appreciate that there was a lot of information to take in. I think the most important thing to retain there is that idea of, you know, having a A real valid comparison group, um, and there are a few different ways, um, through which we can create such a group so that we can really estimate the causal impact of a program. Um, with this, we're going to go over to our case studies, um, beginning with, uh, Arlen. So Arlen is an economist at DM. Where he focuses on fragility, conflict, and violence as well as health, and he's going to present a case on the impacts of reparations for victims of the conflict in, in Colombia. Um, this is an interesting case. Uh, it leverages the rollout of the program as well as multiple sources of data to understand uh impacts across, across multiple dimensions. So without further ado, Alan, over to, over to you. Thank you so much, Marcus. I'm gonna start sharing the screen. So as Marcus was, was mentioning, um, this is a, an impact evaluation of a program in Colombia. So to put it in the wording that she was mentioning, we are trying to evaluate the effect of an implementation or of a program that is reparations for victims of the conflict, and we try to look at the effects on the outcomes which in this case are going to be outcomes that are going to be related to the well-being of these recipients years after receiving these reparation. This is going to be part of what Nain was suggesting as non-experimental methods where we have to leverage several ingredients and we actually have to be very lucky in the characteristics of the population to be able to accomplish and uh the requirements of this type of um of methodologies. Um, but before getting too deep into The methodologies, let me just give you a brief motivation of why we started to study more uh reparations and, and the policy question that we had in mind is, uh, whether reparations can help victims of human rights violations rebuild their lives. More than 30 countries around the world have implemented these, uh, reparation programs to recognize and address. The harm suffered by victims during war, conflict, and under authoritarian regimes, and they are seen as a way to transfer uh money to these victims to help them, uh, to compensate the harm suffered during these, um, these events and allow them to recover and, and, and, and improve their well-being. Uh, these reparations are thought to be fundamental for justice and recovery. And uh they are thought to be really important in particular given how most victims are usually in very poor and vulnerable situations. And despite this, uh, thought, uh, potential effects of these reparations, there were, there was unfortunately not a lot of evidence on the effect that these reparations may, may have on this, um, this population, and there was a lot of controversial. Discussions about whether these reparations could have an effect and given how expensive these reparations were and how the characteristics of these populations were most, which are mostly in this case, victims of forced displacement and were not, would potentially not able to make a good investment of this money given the characteristics of this population caused by the victimizations some. Of the people that was against these reparations were claiming that people will be too traumatized. They wouldn't have the social networks, they wouldn't have like the social or family structure that would allow them to make a productive investment of these reparations. And because of these discussions, there was not still like a, um, a well widely accepted view of the potential effect that these reparations were having on the individuals that resist. And part of the reason was uh why we don't have or we didn't have like a good evidence on this is that first there was not a case in which we had like a good counterfactual group or a case where we had like a randomized, um, assignment of these reparations so we didn't have a good counterfactual and in cases where we didn't have a good counterfactual having uh information of these victims and rich administrative good data about this. Victims was scarce and we didn't, we weren't able to uh look at these um effects. So what we do is that we take a look at the case of Colombia, um, since this was a retrospective, uh, analysis in which we didn't have a, a random assignment and we and random assignment was, uh, not feasible in this particular case. What we have to do is that we have to rely a lot on administrative data. And we, we see, and we use information from Colombia which committed to compensating 7 million victims of the internal armed conflict in Colombia, victims of the guerrillas, paramilitary groups, and state forces, and the target was to compensate these victims by 2031. But before getting too deep into the methodology and what we do, Let me give you a little bit of context of what the case of Colombia. Colombia has had the most prolonged internal armed conflict in the Western Hemisphere, and there has been a battle between, um, guerrillas, parliamentary groups, and state forces for more than 70 years now. This conflict has victimized around 9 million victims during the last 3 decades. That is almost 11 in 5 Colombians is a victim of the conflict, and it has had important implications and devastating implications for most of the municipalities or most of the uh geographic coverage of the uh of the country with more than 90% of the municipalities suffering some type of victimization in, uh, in most rural and most border areas being the ones that are affected the most by this conflict. When I refer to a victim of the conflict in Colombia, I mostly refer to victims of forced displacement, which make for 90% of the victimizations in, in this case and make for the largest number of forcibly dispelled displaced population around the world. Uh, but I also refer to victims of murder and, and, uh, relatives of people that were forced disappeared and among other types of victimizations, but most of these victims are victims of displacements. About the time for these victimizations, most of these victimizations occurred in, in, in the early 2000s, and the government of Colombia has tried to deliver some type of uh compensation of humanitarian aid to these victims. And in order to provide this humanitarian aid in 1997 and they started. Collecting victims, uh, uh, information through the victim's registry, which is one of the main tools or one of the main sources of information that we used to be able to collect information about these victims. In 20, in 2008, the government started compensating some victims whose relatives have been murdered. Uh, during the conflict, but importantly, as part of the transition into peace in 2011, uh, in 2011, Colombia introduced the victims' law, a victims law that introduced reparations to the victims of the conflict, and, and this victim law targeted to compensate the 7 million victims that I was referring earlier. Um, so far, uh, this, under this victims law, uh, more than 1 million victims have been compensated. These compensations or these reparations are substantially large than make for around 3 times the annual household income for these recipients. And importantly for us, we can rely on rich administrative data to do a definitive analysis. We have information on the 7 million people that are eligible to receive the compensation. We have information on The 1 million individuals that have received the reparation, and we have panel data, which means that we can follow individuals on administrative data before and after they received the reparation. So we can use as comparison groups, both individuals that are eligible to receive the reparation and haven't received it, and individuals that have received it, and we can compare those individuals before and after receiving the reparation. And we have information about several aspects of these victims' lives. We are able to provide information about and and results about their work and living standards. We are able to uh take a look on what are the health effects of these reparations, and we are also able to take a look on what are the effects on the next generation's human capital accumulation, uh, which is important to understand what is the effect on the, uh, children of these victims. When we take a look at the effects. Importantly, in this graph, we have the differences between those that received the reparation before and after receiving the reparation. On the left-hand side of this graph, we observe the difference before receiving the reparations. And luckily for us, like in this case, what we observe is that There are no differences before they receiving the reparations which makes in this case for a good counterfactual. And what we see on the right are the differences between those that receive the reparation and those that don't receive the reparation. In this case, we're talking about the effects on formal employment. So what we observe here is that since these lines here are distant from zero and they don't intersect with zero, we we analyze this as a negative effect in this case. So victims one quarter after receiving the reparation are less likely to be working in the formal sector. And this is an effect that attenuates over time such that 3 years after receiving the reparation, this coefficient, the last coefficient is very close to 0 and it's actually Um, uh, smaller than the one that we had before, such that 3 years after receiving the reparation, victims are 1.6% less likely to be working in the formal employment. This effect is driven mainly by young workers, uh, shifting out of low paid and risk employments. So when we take a look at what is happening to these victims that are not working under the formal employment, what we observe is that they are actually taking more time using the money as a better outside, um, um. Outside resource to look for better jobs, to get better jobs, such that 3 years after receiving the reparation, they're actually working for better paid jobs that are less risky and that allow them to have larger incomes. And not only that, when we take a look at business creation and other type of results, what we find is that they are also creating more businesses. They're able to consume and buy more land. They're able to pay off their their old debts, and they're able to buy more durables over time. These effects add to a, a, a positive effect on health outcomes such that victims 3 years after receiving the reparation are less likely to have an emergency, emergency department visit, are less likely to have a hospitalization which we read as a positive effect on health. And not only that, the children of the victims are more likely to have a, a undergraduate enrollment and they are also performing better in the high, high school exit exam, which Makes for a case of victims invest in this money not only on their consumption, not only on their businesses, but also on their children's education, which makes for potential better and longer term outcomes. So as key messages with rich administrative data and a, a, a potential feasible design and could allow for differences in differences, estimations that could allow for uh uh identifying the causal effect of victims. In this case, we find that reparations improve victims. Lives in several outcomes including uh consumption and living standards, improving their health, their health, and also improving their outcomes for the next generation. In forthcoming exercises, try to look at the effects on other outcomes and also uh using an RCT to see what happens when we accompany these reparations with other variations of the programs including psychological workshops and investment advising, uh, workshops and that's all that I have for today. Thank you so much, Arlen, for sharing these uh powerful and important results. Uh, we're going to go right to the next case study. I wanted to encourage all of you, however, if you have any questions for any of the presenters at any time, um, if you want to put them in the chat, um, we can hold on to those until the Q&A. Um, so please, um, don't hold your questions, just, just go ahead and, and stick them in the chat. Uh, with that, I want to turn it over to Paula, who is an impact evaluation specialist in the World Bank's fragility, conflict and Violence Group and is one of the engines, I think, behind the building evidence on force displacement program. Paula's going to present to us a non-experimental study again, which looks at The impacts of changing the size of the cash transfer provided to Afghan returnees, um, using a regression discontinuity approach. So with that, Paula, over to you. Thank you, Marcos. I'll uh uh share my screen now. And uh yes, as you mentioned, this is uh uh part of the Building the Evidence Program. This study is part of the Building the Evidence Program. And um it, it is a result of a collaboration between the World Bank and the UNHCR with the uh goal of building evidence on the return experience of uh Afghan refugees. I am presenting um a work that was led by my colleagues, Hisham, uh, Nandini, and uh Christina. So, between 2016 and 2018, there was a, a large scale return of um Afghan refugees uh from Pakistan to Afghanistan and uh uh 458,000 registered with the UNHCR and 380,000 were uh were not registered. And this is on top of uh um uh 1.6 million returnees uh from Iran. A data sharing partnership was set up between the World Bank and the UNHCR with the objective of uh using uh administrative data that was being collected to understand the location, socio-economic, uh, characteristics, livelihoods, and needs of, uh, recent returnees, uh, to Afghanistan. In the context of this, uh, uh, data collaboration, uh, there was also a sudden, uh, programming, uh, change that took effect, uh, and became the opportunity to Uh, be able to rigorously evaluate the UNHCR cash reintegration assistance provided to returnees, uh, using existing administrative data and, uh, um, as, uh, Nao Xin explained, uh, regression discontinuity, uh, design, uh, method. Um, so as I mentioned, uh, administrative data sources were, were used, uh, and this included the population profiling verification and response data collected in Pakistan, um, uh, back in 2011, and then UNHCR's voluntary repatriation, uh, form. Data that's collected as uh returnees enter after they cross the border at collection points and uh the UNHCR post-return monitoring surveys that were done in 3 waves and covered, uh, a sample of over 4000 houses. There, there were some new data collections as well that were done, um, namely, namely, uh, phone surveys and, um, qualitative interviews, uh, interviews done with the, uh, 10 different individuals, uh, during COVID and, uh, repeatedly. Um, so there were a few analytical products that came out of the analysis of this data and the one we're focusing on in this presentation is the impact evaluation. Um, so as I mentioned, as, um, uh, the UNCR has in place an unconditional cash transfer program to, um, returnees from Pakistan that register, uh, upon, uh, uh, return. And, uh, uh, the amount that that's transferred, uh, was until Uh, July 1st, 2016, about uh $200. There was then the introduction, some of the sudden introduction of an increase uh that was announced by the UNHCR commissioner uh on June 29th and took effect was implemented on July 1st. So, um, Uh, quite suddenly and immediately after the decision was taken and similarly, this increase, uh, remained, uh, effective for nine months. Uh, after that there was a sudden decrease, uh, because of budgetary constraints and so a return to the $200 allowance. Um, so, uh, what this sudden policy change and unexpected, uh, made possible was the comparison between Uh, households that had returned just before the, uh, decrease, uh, so before April, uh, a specific date in April 2017, and, uh, those that, um, um, received the, the lower allowance amount because they, they returned after, uh, the decrease took effect. And since there was no way to tell when this would happen as it was unexpected and um quite suddenly introduced, um, these, uh, these two groups, we can think of these two, these two groups as uh randomly receiving one or the other amount and so, uh, as a comparable groups. So this analysis was done and then uh the findings were quite uh interesting. After 60 on average 60 months after return, um, the households that received um the higher uh amount were 17% um percentage points more, more likely to have invested in durable assets such as a house. Uh, those that received the higher amount were also 30% touch points more likely to have obtained, uh, legal documentation for all household members. Uh, there was also different spending pattern between, um, those that received, uh, the different amounts, uh, in that those that received the lower amount were more likely to spend it on immediate, uh, consumption needs. So there was an impact there also on, um, So the size of the, of the transfer affecting the spending patterns and choices of households and individuals. Um, and instead there were no um differential impacts on employment or enrollment of children in school. So overall, this impact evaluation and um this study uh generated new evidence on uh what a larger uh cash assistance amount can do in terms of, uh, uh, facilitating, uh, the, um, The return and post-return outcomes for refugees returning. Uh, this chart, I'll, I'll skip over, uh, but shows, uh, the strong impacts on house ownership, uh, which I just mentioned, documentation, and, um, spending on, uh, on food items, um, and as well as the non, non-impacts on, uh, employment, uh, and, um, um, enrollment of children in school. Um, I have this graph here to show how, uh, uh, yes, that nicely shows the differences. So, uh, the zero in this, uh, uh, graph on the On the right is the date in which the different so the reduction in the cash assistance amount was introduced and so you can see that households are returned earlier and had higher, higher amount, have much higher household um ownership at 43%, whereas those that received the lower amount are returned after the um uh change, uh, programming change, um, had a 17% uh um house ownership instead. And uh the similar lines in this next graph shows uh what I was explaining as the non uh non-impact, the non-differential impacts on employment outcomes that are the same regardless of the amount received. So to wrap up, um, this was, uh, this is an example of an impact evaluation, uh, employing a regression discontinuity, uh, method, um, and, uh, with the cutoff being the date the programming change was implemented. Um, this, uh, this can happen a lot and, uh, uh, as programs are implemented there is It's a sudden change and this can be exploited if the data is available uh to uh compare outcomes and uh evaluate the programs. And, uh, it was nice about this evaluation as well as ours that they used existing administrative data so there was no need for new data collection and also, uh, it delivered important findings, uh, six months, uh, post-return. Um, on, uh, um, what it does to receive a larger, uh, cash reintegration assistance amount. And, uh, uh, thanks everyone for the attention and I'll stop there and I'm happy to take any questions. Thank you so much, Paula. I think this is a, a very nice illustration of how we can use unexpected changes in, in policies to actually understand uh what the real impact is. So, so great case. I'm going to turn it over to um Chloe now. Chloe is a research officer um here at DIM uh where she uh leads our FCV program and also works on our social protection and humanitarian portfolios, uh, including leading our collaboration with the World Food Program. And Chloe's going to present, um, two cases back to back. So the first is a randomized controlled trial of a livelihood support program for host communities and displaced persons in, in ***. And this is an example of embedding an impact evaluation in a World Bank-supported project from the design stage on. So working with the government team from design to finish to understand what are the impacts of this program. And after that, Chloe will present a study in a program involving Rohingya refugees in Bangladesh, which was designed to understand the additional psychosocial value of work beyond just the, the monetary benefits. So, Chloe, over to you. Thank you. Um, good morning. Good afternoon, everyone. So we're heading over to ***, uh, after the presentations on Colombia and Afghanistan, um, where I'm going to present results from an impact evaluation on, um, entrepreneurship support component that was provided to both displaced populations, but host populations in quite fragile areas of the country. We have many people that are working on this impact evaluation, including other colleagues at DM, but also our partners at um Trinity College in Dublin. So, the context of ***, I'm sure everyone knows, unfortunately, is a very fragile one. a big part of the population lives below the national poverty line, and it consistently ranks one of last in any development index, including the famous HDI. It's also highly exposed to conflict and climate shocks, and yet, regardless of this level of poverty and these recurring shocks, it hosts first many refugees from um neighboring countries that are also themselves very much affected by conflict. But in addition, there's a lot of internal population movements as these shocks happen. So, we're in a context where we have an initially very vulnerable country, um, and this vulnerability is further exacerbated by these population flows. And as most of you here know, I suspect, um, until recently, a lot of the research that has been done are on exposed impacts of displacements, right? So how did the displacement affect potentially the host country or the populations. However, there's very little research that has been done. Um, how to design better programs to improve the lives of displaced and vulnerable host populations. And so this is what this initiative between, you know, the World Bank, UNHCR funded by the UK is really trying to think through. So, what can be done to improve economic and social outcomes of these population groups. And so the project that provided us with the opportunity to look a bit into this question is the World Bank funded uh refugees and Host Community Support Project, PARCA, and it's French acronym, that um is implemented by the Nigerian government with input actually from, from UNHCR and it aims at improving access to basic services and economic opportunities for refugees, IDPs, and hosts alike. So, a very important component of this project is that it is agnostic to the population it's serving. It's also in Niger's most fragile regions. So there are two main components of this project. The first one being a labor intensive public work component, which provides short-term cash transfer in exchange for labor, which aims at building infrastructure. It targets the ultra ultra poor of these communities, and again, regardless of your displacements or host status, anyone could have access to it. The second component that I'm going to be focusing on today, which is what the impact evaluation was studying, is on an entrepreneurship support package that was provided to the slightly less, or, again, we're still in a very vulnerable context, so that has to be taken into account, regardless of status, and this package included both um business trainings, coaching, savings, etc. And then a $200 cash grant, conditional in attending these trainings, and this cash grant was to be used to invest in these new entrepreneurship opportunities that were provided to the population. So, what will we learn by looking at the impacts of this entrepreneurship package? Well, first, evidence has shown that similar livelihood programs do provide strong and sustainable impacts in a wide range of contexts. So we know it works, um, especially in ***, um, which had its own adaptive social protection program implemented by the World Bank and the government. Which took place with similarly vulnerable populations, but in slightly more stable areas of the country, and that only targeted non-displaced populations. And so the contribution that we hope to make is, do these programs, these similar livelihood programs, work in very volatile and dynamic contexts where there's a lot of population flows. In addition, we're also going to try to see if these results differ by population groups, that being displaced, refugees, internally displaced, or hosts. And thinking ahead, um, these results and these types of studies could potentially help, um, unfortunately with the recent coup that the country faced, we, we expect maybe further repercussions on these fragile population groups, and maybe this study could further help them development and humanitarian partners alike that are connected here, prepare their upcoming response. So, the Pera project took place in the regions of Tillabiri, Tawa, Marradi and Difa on the left-hand side. In red, you see the villages that were targeted by the project. Um, and as you can see, if you corroborate that with Aled's most recent, uh, conflict map in the region for the last 5 years, you can see that, uh, the project is very much taking place in the most conflict prone areas of the country. So, how does this work in practice? Going back to the first session that you saw, um, this impact evaluation supported the Parker project right from its design. So in this case, we were able to join the project team as it was developing the different components and thinking about how to implement and deliver them. So first, we focused our study sample for the impact evaluation only on defiant liability, um, and we focused on 170 eligible villages to receive the entrepreneurship package. Then we allocated them by lottery to either receive this package in the 1st phase or in a 2nd phase, which is what enabled us to create these comparison groups that Noxin mentioned earlier. And also it helped with project implementation logistics, since not all villages could be targeted at once. Second, in the villages where we had more eligible poor individuals than the program could target, so over subscription, we did an individual public um lottery to decide who would get this entrepreneurship support package. So it was fair, it was public, and then it decided who was going to get this package versus who wouldn't. This is what enabled us to measure these causal impacts across the individuals, individuals who received the entrepreneurship package, to individuals in the same communities that did not get the package, and then to individuals in communities where the package had yet to be delivered. Moving on from uh the design phase, this impact evaluation also helped um Pcast uh implementation, right? So we supported the project throughout its life cycle. So in June 2020, uh, the implementing NGOs worked with the communities to identify who are the poor and ultra-poor households. As I mentioned earlier, this entrepreneurship package focused on the poor, not ultra-poor. Now, If you look on the left side, only 3% of the population is considered non-poor, so these are very relative uh vulnerability levels. So I'm not gonna go through each step, but basically what you need to kind of remember is that we helped with the implementation once we assigned um people um to the village lottery, or to the public or individual um lottery, the entrepreneurship support package was delivered. We collected in an end line in June, so very, very recently. Um, and then the remaining villages received the package, so everyone got support at the end of the project. So what are some um preliminary contributions of this impact evaluations? Well, first, remember that this is a very fragile part of the country with little existing data at conception. So as the project was designed, um, we could have some ideas about certain points, but we We weren't exactly sure. So, for example, where are the displaced? Are they everywhere? Um, are they more concentrated in certain communities or regions? That was a big question. Second, a common, maybe misconception is that the displaced are potentially worse off than the host population. Um, and this could impact their capacity maybe to absorb, you know, the project. Um, and finally, there could be tensions between different population groups. Again, we're in an area of the country that's Very prone to conflict, and so we could imagine that there are tensions or social cohesion issues. And however, what was interesting is that at baseline we got Very important and slightly different information. First, where are the displaced? So you don't need to focus too much on the graphs, but we saw that about 40% of our population is displaced in DFA of our impact evaluation population, versus only 22% in Tillaberi. This is still high, but this means that Tillaberi has slightly less levels of displace, and on top of that, Tillaberi only has internally displaced and no refugees. This could have impacts on This could have implications on the impacts of the program in these regions, cause we're talking about different population groups. As I mentioned earlier, we imagine also potentially vulnerability differences between the displaced and their hosts, and thus their capacity to maybe appropriate this project components. However, we found that all population groups are extremely vulnerable, regardless of whether they are hosted displaced. This is just one example with food security that you see here, but all our different groups barely meet the international thresholds. We could say the same about mental health, so this is, um, in terms of depression threshold, all of our populations are very, unfortunately, very close to to these depression indicators. And then in terms of tensions, as I mentioned earlier, there's actually quite little tension between our different groups, um, less than 10% reports some form of conflict, and by conflict we include discrimination, verbal abuse. Um, it's still there, but it's, it's not very different than elsewhere, and so this could also make us rethink a bit some of the social cohesion implications that we expect to get from the project. Or any other projects, by the way. So the key messages of this study in ***, well, first trial and adopt, as Marcus mentioned, or impact evaluation exercise can really help throughout the project life cycle. It's not just useful to measure pro project impacts, but also it can help facilitate the project design as we have to think about how to implement it, especially in fragile contexts where implementation and delivery is quite complex. But also the data that comes in, even from the beginning. Beginning in this case at baseline can help us rethink interventions, make the project a bit more adaptive, and anticipate a bit more where we want to go. Now, I'm sure what people are most interested in and what are the results of this project. So we have very, very preliminary findings. As I said, the endline data collection happened just last June. The project is closing right now and we've started our analysis. But right now, it suggests that this entrepreneurship package had positive effects. Effects on household income, employment, and economic activities. We're fine-tuning these numbers and hope to share them with you very soon. But more importantly, right now, we're gonna further this study a little bit more into other outcomes, as we said, psychological well-being, vertical and horizontal trust, social interactions, and especially going into this breakdown by population group, which is what a lot of people are interested in. Um, so that was it on ***. I'm going to, uh, acknowledge, of course, there's slides, everyone will have access to them, to, to the donors and the people that funded this project, um, and because we only have a few minutes, and we really want to give you time to ask questions, I'm gonna spend a brief minute on, on Bangladesh, um, and the psychosocial value of work in worryinga refugee camps. This work was done by um a colleague of ours, Erin Kelly, but uh I'll be representing her today. So, moving over to a different context, now we're in a context of refugee camps, so populations aren't mixed anymore, um. What motivated this study, right? So, we know that there are refugees worldwide. Everyone on this um training knows this, and that 70% of them face restrictions on their right to work. And though there are conventions protecting these rights, most countries either don't sign them or have expressed reservations, which thus contributes to high incidence of poverty amongst refugees because they aren't able to access employment. Now, in addition, we know very little about the impacts of being employed on mental health, though this is quite needed both um in terms of displacement response as displaced populations, uh we can suspect often have, you know, mental health um constraints, but it also can provide evidence on, you know, other unemployment schemes such as cash for work or the universal basic income. Now, what is the value of employment beyond its direct monetary benefits? So what we want to know here is, can employment help refugees, um, help improve refugees' mental health, basically. So the context is we're going to focus on being a camps in Bangladesh, and Bangladesh is not party to the 1951 convention, and the refugees cannot work. In addition, there were very little informal job opportunities due to mobility constraints, military checkpoints, etc. There's some jobs provided in camps by NGOs, and this is what we're going to be looking into right now. So, Going into the actual design, um, this I impact evaluation took place in the Kutupalon refugee camp that is divided in 149 blocks or clusters, and so we use these blocks to randomly allocate, um, people into different groups. The first one being a cash for work group, so 83 blocks received this opportunity to work, um, small labor for, for, uh, money. It was about $5 for 3 or 4 days of work per week. A second group where we're gonna provide just cash, not work, um, to people for the same amount, so about $5 and then our comparison group that did not receive cash for work or cash, just a very minor incentive to respond to the survey. So how do we measure the psychosocial value of work? Well, we're gonna compare people that get cash for work, so that get to work and they get money for it, that are compensated to people that receive nothing, at least from this component. And then, in addition to that, what we really want to understand, right, is what is the value of work beyond its pecuniary value? What is the value of work beyond being compensated by money. What does it do for your mental health? And so it's for this, we're going to compare people that receive cash for work to people that receive the same amount of cash, but may not be working. And so the type of work um is both accessible to men and women, it's, you know, very low skilled work, uh, it's not too good, not too bad, it provides, you know, it's outdoors, people can move, it's a bit repetitive, it doesn't require too much time and attention, but still a bit. Um, it has a clear purpose, but it's secondary to typical employment, so it's not like being a teacher, right? And it's offered by these NGOs. So for example, it can be, you know, some cleaning tasks in the community, uh, to, you know, sewing classes, etc. so they're, they're minor labor jobs. So what are the results? Um, well, we see, um, that work, relative to no work, uh, and cash, improves mental health. So it provides a 50% increase in the likelihood of not being depressed, which is quite big. It's similar to more or less to a 44 week long psychotherapy program that's currently being implemented in these camps. Now what we're really interested in is what is the added value of working compared to just receiving cash, and so if we isolate. These two groups, we see that work yields significantly larger improvements, up to 4 times in psychological well-being than just receiving cash. It also has implications on physical health, 20% decrease on feeling unwell, and improvements on cognitive function. What are the key messages? Well, first, uh, the psychosocial benefits of work exceed those in cash in these settings, particularly for men. Now, we can suspect that it's because women are a little bit less idle with, you know, household chores and childhood and children. This is being studied right now. These are just hypotheses. It's very important for policy to Discussions on promoting the right to work, um, especially in the case of refugee camps, where there's a lot of, you know, not much can happen, there's not a lot of work, there's a lot of idleness, and scope for leisure activities is limited. So this provides a, you know, something to do. And so this study has currently been extended to. This hypothesis in other contexts, other camps or non-camps, and trying to really isolate what elements of of work actually yield the greatest psychological benefits. Is it the fact that people are being trained in something new? Is it the fact that people are busy? Is it because they feel a sense of purpose, etc. And of course, trying to document these impacts on intra-household dynamics, um, bargaining power, intimate partner violence, as women are also participating in these programs. Um, thank you. I think we're gonna go into QAA Q&A, I'm sorry. So, um, I'll let maybe my colleagues and myself, if there are questions for me, answer questions. I'm gonna mute myself so that maybe the moderator could take over. Thanks so much, Chloe. Really interesting presentations, and I think these are both good cases that show why integrating this kind of Uh, analytical impact evaluation approach into the fabric of the project, the program from day one really allows us to understand, to understand and to answer a much broader range of questions. So when we think about what our questions will be before the program starts, we can actually design our data collection strategy and work with program implementation in such a way that we have very good quality answers to those at the end, which is not always true if we wait until the program is over. Um, with this, I want to turn it over to, to Q&A. Um, if anybody wants to come in, uh, preferably, you know, unmute yourself and, and, um, and go ahead and speak. Uh, if not, um, you can type your question in the chat, and we do have a, a couple of, of questions to, to get started that already came through in the chat. First of all, Um, for, for Paula. I've just lost the question. Here we go. Uh, Valeria Lentini from UNHCR was interested in How you measured mid-term or long-term effects on other indicators, uh, beyond the increase in transfers allowed for higher investment, uh, allowing for higher investment in durables. Uh, Valeria, perhaps you can repeat the question if you want, if, if that wasn't clear. Uh, Paula, if, if you're able to answer this question, that would be great. And then, um, for Chloe, there was a question on how, uh, depression is, is measured. Thank you. I already replied in the, in the chat, but I was thinking about the very long term in uh because we are always interested in, in sustainable solutions, right? So maybe in, in 16 months is is great, but also more in the future since you found no differences in, in the labor market or kids' enrollment. So I was interested on that and thank you very much, very interesting, the 3 presentations. Thanks. Uh, thank you, Valerie. This is a great question. And, uh, there was actually an attempt to look at the long-term effects. Uh, although it was really hard to reach households because in the meantime, they changed, uh, phone numbers and so it was really hard to reach them and the sample, uh, of the planned sample, I think less than 20%, uh, were reached. And so it wasn't, that wasn't enough data to be able to say, like to say with certainty, statistical, um, um. Yeah, uh, power, um, that, uh, uh, we were observing anything. So an attempt was done and, uh, um, yeah, um, it wasn't successful because of the difficulty in, uh, tracking, um, households, uh, in that situation. Those are returned and, uh, specifically because of the change in, uh, in phone numbers. Um. And you know, this was, um, as I explained, this study happened because there was this policy change and there was administrative data but it wasn't set up for the, from the start to look specifically at this. So possibly uh questions would have been asked differently or uh more of an effort would have been done to uh Uh, collect phone numbers and track them or like the number of the friend, the neighbor, uh, but, uh, since, uh, the data was there and, uh, all the analysis was done retrospectively, this was the, yeah, this is what, uh, what happened. So it's not impossible to reach people on the phone, but um, it was, uh, challenging this, um, um, situation. Uh, over back to you, uh, Marcus. Thank you. On the, the measurement of some of the psychological indicators, both in Osheen and, and Chloe have, have answered in the chat. We have, I think, just a minute or two left if there's anybody else that wants to come in with a, with a comment or a question. And then we'll close. And maybe it's fine. Uh, no, go ahead, uh, Valeria. I am, go ahead. There's enough time. Valeria Yanoti, please, then I can go because I already see. Thank you. Thank you. Uh, first, I just discovered some typos in my questions, so please forgive me. Uh, one of my biggest concerns is the Ecuadorian Colombian border. Uh, it's a very dangerous place and is where the conflict is still active. The Colombian conflict is still active. And now that specific part of Ecuador has become very dangerous as well because of organized crime. So some refugees, uh, they prefer, or, or asylum seekers, they prefer to hide instead of, of showing themselves or to get access to everything that the Ecuadorian government has to offer. Uh, the Ecuadorian Constitution says that. Regardless, the um, Migration status, they have access to public education, public health, but some people uh prefer not to access that because they, they are too close to the conflict and too close for what they have suffered. So I, my question is how can we reach that kind of person to design programs or to know how can we help as much as possible and to gain trust uh in Ecuador that we are not going to, to disclose any information to Colombia. So that, that's my, my concern. I don't know if if it is pertinent or not for the course, but. Yeah, thank you. Thank you, Valeria. I think it's very pertinent and I'm, I'm sorry uh for missing the question that you had placed in the chat. Um, I'll offer a very brief answer and then turn it over to my colleagues who may have more information. I think this is a really important and an interesting question, and I think this is exactly the kind of question that we would be thinking about at the outset of a program. Um, I am not aware of studies that look specifically at Different strategies for encouraging persons to come forward that may want to remain anonymous in other domains, but I think this is exactly the kind of thing that one could think about, um, testing to see which sort of strategy is, is the most effective in doing so. Um, so I think this is exactly the kind of question that, that we often, uh, think about. Uh, unfortunately, um, I don't have a, a A ready answer for you there, but perhaps others want to come in. I may come in and I'm probably not being like too insightful in terms of um Uh, any strict evidence or rigorous evidence from an impact evaluation, but there have been different discussions about the way to target victims of the conflict, and both because some victims want to remain anonymous and also because targeting specific victims or telling a portion of the population that you're not worthy, the aid, it can also create Uh, social cohesion problems and, and one of the, um, ways in which, um, these programs have accommodated that is targeting either outcomes that are particularly relevant for the refugee population rather than targeting individuals or targeting demographics, meaning the refugees mostly go to some particular community. and are in need of particular aspects of their lives rather than than targeting individuals themselves, go and target those territories and provide those type of aids that you expect refugees to be needing the most like short-term shelter, like food security programs in. Places that are very close to the, to the border and, and, and, and those type of characteristics. I think there are strategies that are being used, but there's not a lot of rigorous evidence on whether to have those have like any substantial um differential effects, but I, I would say that's what we at least know from, from these discussions. Thank you so much, Charlon, and I thank you, Valeria, for raising the, the very interesting point. Uh, I think we are essentially at time. I want to hand it back to Chloe and Nosheen, um, to wrap up. So from my side, thanks so much to, to everyone for connecting for this session and thanks also to, to all the participants and the organizers. Uh, over to you, Chloe and Nosheen. Thanks, Marcus. Um, I think, um, we don't have that much left. So just to conclude, one of the main things is now that you had a chance to go through the session and, you know, to get familiar with some of the topics, I would encourage you to also try and go back to the knowledge exercise. I have posted the link on the chat, you know, to see if there are some answers you have. More clarity on it would be great um if we could get those responses and like I mentioned, these are anonymous. In addition to this knowledge exercise, we would love to have your feedback on the session today. This link has also been posted on the chat, so if you could spend a few minutes and answer those questions, it would be great. And last but not least, um, we have the link to the. Materials and other information about the training, you can go find the schedule and the presentations, including what um it covers both the academic papers, some of which were presented today. And this is a series of trainings. So next Wednesday, keep your eyes out for the force displacement evidence in the health sector, October 18th, um, at the same time. So, uh, with these, I will end here. Thanks to everyone. And like I said, um, the links are on the chat, so if people could just spend one or two minutes filling those out, that would be great. And back to you, Paula, for any last minute closing. Thank you. Uh, thank you now, Xin, and, uh, many thanks to, to all the presenters, uh, today. I think this was an excellent, uh, session. It's great to see the engagement in the chat. Uh, I would just repeat, it's important to receive your, your feedback as we're in the middle of this training program and we Uh, I'd like to endeavor to improve, um, going forward and making sure these sessions are as useful as, uh, possible to, to all participants. So, um, uh, do we have the feedback form? Great. OK, perfect. I see in the chat, the feedback form, that includes the feedback form, and also, um, For colleagues that you know would be interested in this, uh, course, uh, uh, in the chat, you can also find the link to our website and there we are, uh, posting the recordings, um, as well as all the background materials and presentations that we're delivering during these sessions. Finally, I'll also say that there's going to be, these materials will be taken and converted into an e-learning, self-based in-learning course that's, uh, also going to be open to Uh, anyone, the wider public, uh, to attend and that it will be ready, uh, in the beginning of the, uh, next year. Um, thanks again to everyone and, uh, have a nice rest of your day, evening, uh, or, uh, morning, wherever you are. Bye.

showAllTranscripts

no

duration

PT1H31M44S

scene7File

worldbank/20231011-Module3-Impact-Evaluation

scene7Domain

scene7FileAvs

worldbank/20231011-Module3-Impact-Evaluation-AVS

title

20231011 Module3 Impact Evaluation

description

20231011 Module3 Impact Evaluation

showTimestampAndTranscript

yes