col-xs-12
col-sm-12
col-md-12
col-lg-12
col-xs-12
col-sm-12
col-md-12
col-lg-12
videoType
dynamic-media
videoDmUrl
https://delivery-p136806-e1377785.adobeaemcloud.com/adobe/assets/urn:aaid:aem:8451f7cb-c8d9-4078-b1d2-b469ca3200a7/play?assetname=RSMFP+Lecture+with+Berk+Ozler+Part+II.mp4
keyFrameImage
timestamp

00:06 Well,

00:06 good morning,

00:07 everyone,

00:07 and welcome to the 5th and final lecture in the Robert S.

00:11 McNamara

00:12 Fellowship Program lecture series.

00:14 So as a quick background,

00:16 the World Bank's

00:17 Robert S.

00:18 McNamara Fellowships Program,

00:19 known as RSMFP

00:21 matches aspiring development economics researchers from developing

00:24 countries with World Bank research economists.

00:27 Fellows have unique opportunities to participate in policy,

00:30 in rigorous policy relevant research in the

00:32 World Bank's Development Economics vice presidency.

00:34 The RSMFP lecture series is a flagship activity of the of the RSMFP.

00:39 Um,

00:40 over the course of 5 lectures,

00:41 fellows and other attendees will learn about

00:43 the latest developments in empirical research.

00:45 The lecture series focuses on topics that,

00:47 although not always taught in graduate courses,

00:49 are fundamental for development researchers.

00:51 Today,

00:52 we're delighted to welcome back Brooke Osler to give

00:54 the second half of his lecture on why randomize,

00:56 what to randomize,

00:57 and how,

00:58 combining basic principles of RCTs with practical aspects of field experiments.

01:03 Brooke Osler is a lead economist and research manager of the

01:05 Poverty and Inequality Research Program in the Development Research Group.

01:09 He received his BSC in mathematics from Bosphorus University in 1991

01:12 and his PhD in Economics from Cornell University in 2001.

01:16 After working on poverty and inequality measurement,

01:18 poverty mapping,

01:19 and the 2006 World Development Report on equity and Development earlier,

01:23 he decided to combine his interest in cash

01:25 transfer programs and HIV risks facing young women.

01:27 In Africa

01:28 by designing a field experiment in Malawi.

01:31 He has since been involved in a number of cluster randomized field experiments.

01:34 He's a co-founder of and regular contributor to

01:37 the Development Impact blog.

01:39 So,

01:39 without further ado,

01:40 let me hand it over to Burke.

01:41 Um,

01:42 thank you so much for joining us again and

01:44 looking forward to the rest of this lecture.

01:47 All right,

01:47 perfect.

01:48 Thank you,

01:48 Maria.

01:49 Uh,

01:49 and thank you,

01:50 Napur,

01:50 for,

01:51 uh,

01:51 again,

01:51 organizing this.

01:53 Um,

01:53 yeah,

01:54 like,

01:54 uh,

01:54 you were all here,

01:55 I think last time around,

01:57 and,

01:57 you know,

01:57 I didn't get through a lot of it,

01:59 so I volunteered,

02:00 but I wasn't sure there'd be

02:02 some demand.

02:02 But,

02:03 um,

02:03 the time folks tell me that there was some,

02:06 at least somewhat enthusiastic response.

02:09 So here we are again,

02:10 and this is where I

02:12 remembered we were.

02:14 stuck,

02:14 I think.

02:15 Um,

02:15 so I'm just gonna continue.

02:17 Um,

02:18 you know,

02:18 the first part was a bit more

02:21 kind of like conceptual and kind of like about counterfactuals and

02:25 things like that.

02:26 This one will be a little bit more

02:28 practical,

02:29 uh,

02:29 for when you're doing,

02:30 uh,

02:31 RCTs,

02:32 when you're planning RCTs,

02:33 what kinds of things you should be thinking about.

02:36 So it can be a bit more informal,

02:37 like,

02:37 I'm not going to

02:39 go through too much detail on kind of like,

02:42 There's technical things,

02:43 but I'm not going to go through the kind of,

02:45 like,

02:45 why those equations work and things like that.

02:48 But also feel free to,

02:50 you know,

02:50 interrupt.

02:51 You guys are generally,

02:52 uh,

02:52 I attended a few of these.

02:53 You guys are a quiet bunch.

02:55 But so,

02:55 this is the last one.

02:57 So,

02:57 um,

02:57 feel free to just kind of ask questions,

02:59 that there's really no stupid questions,

03:01 and then if I don't know the answer,

03:02 I'll just say,

03:03 I don't know,

03:04 um,

03:05 because some of this stuff is,

03:07 you know,

03:07 cutting edge,

03:08 and then we're just all learning.

03:10 Um,

03:10 together.

03:11 So,

03:11 I'm gonna

03:12 just jump back in,

03:14 um,

03:15 with the kind of

03:16 one of the common critiques of randomized experiments and,

03:19 you know,

03:20 something that policymakers

03:22 regularly face.

03:24 is,

03:25 you know,

03:26 what evidence to use for

03:29 the,

03:29 kind of like,

03:30 to inform the design of their programs.

03:33 You know,

03:33 I'm just coming from an event now in Mozambique.

03:37 Let's say in Mozambique,

03:38 they didn't really have an RCT on this,

03:40 like,

03:40 adolescent girl program that they're thinking about,

03:43 but there's an RCT from Malawi,

03:45 which is a neighboring country.

03:47 Um,

03:47 but then they have really good,

03:49 let's say,

03:49 demographic health survey data,

03:51 and they can run a bunch of like regressions,

03:53 maybe even do like diff diff or something like that,

03:56 etc.

03:57 The question is,

03:59 You know,

04:00 should we use our

04:01 own data,

04:03 but we're not going to have a randomized experiment,

04:05 or should we have the randomized experiment,

04:07 but it's not our own data,

04:09 you know.

04:09 And

04:10 people do disagree about this,

04:12 uh,

04:12 the answer to this question,

04:14 and probably there isn't a correct answer.

04:17 Um,

04:17 and people have been disagreeing,

04:18 as you can see from these quotes for like 60 years or more.

04:22 Um,

04:23 and some people think that's really internal validity,

04:26 which I'll define in a minute,

04:28 like,

04:28 you know,

04:28 RCTs are kind of like should be

04:31 prioritized.

04:32 Um,

04:33 and then external validity,

04:34 whether some evidence applies to some other context should be just secondary.

04:39 Um,

04:40 and then there are other people who say,

04:41 look,

04:42 if I'm trying to make policy,

04:43 like it makes no sense to value one type of validity above the other,

04:47 because there may literally be nothing

04:49 with that type of validity for me.

04:51 So,

04:51 what am I supposed to do?

04:53 Um,

04:54 so,

04:54 we're essentially thinking here that,

04:56 you know,

04:57 um,

04:58 think of an example like this in,

05:00 in the United States,

05:01 there's a famous,

05:02 uh,

05:03 preschool program called Head Start.

05:04 It's all over the country.

05:07 But in the 60s,

05:08 some people did a randomized experiment in like a small school district.

05:12 It's a famous study called Perry Preschool Project.

05:16 And,

05:16 you know,

05:16 that one showed that,

05:18 like,

05:18 the um

05:19 graduation rates for kids who are

05:22 in the um Head Start program

05:24 are significantly higher than people who,

05:27 who are not,

05:28 right?

05:29 But then there's a,

05:30 oops,

05:31 sorry,

05:31 there's another study that actually

05:34 shows that,

05:34 like,

05:35 um,

05:36 You look at all the Head Start programs around the country,

05:39 and you look at people in Head Start versus you look at people not in Head Start,

05:44 and the graduation rates of Head Start are 13% points lower,

05:48 right?

05:49 So,

05:50 Yeah,

05:51 in this case,

05:51 I think the answer is a little bit

05:53 easier in the sense that we can easily think of the selection issues.

05:57 Head Start program is a program for underprivileged kids.

06:01 So,

06:01 of course,

06:02 their graduation rates are going to be lower,

06:04 potentially,

06:05 even

06:06 with the program

06:07 than people who have better means and then maybe

06:10 going to better public schools or private schools,

06:12 whatever,

06:13 etc.

06:14 So,

06:14 here,

06:15 you know,

06:15 you have

06:16 this.

06:18 Kind of like exaggerated difference where the effects are going the other way,

06:22 and then theoretically speaking,

06:23 we have an expectation that like,

06:25 you know,

06:26 there's no reason to think that

06:28 this preschool program would have stark negative effects.

06:32 But

06:32 you can have

06:34 cases where both effects are positive,

06:36 but they're,

06:36 one of them is small and one of them is big,

06:38 and that makes a difference for cost effectiveness and things like that,

06:42 then you would be more scratching your head saying,

06:44 well,

06:44 which one of these we should trust more.

06:47 Right.

06:48 And in that,

06:49 in that sense,

06:50 I think,

06:50 um,

06:51 you know,

06:52 thinking about

06:53 Methods to just kind of extrapolate results from one place to the other.

06:58 Um,

06:59 there are interesting work that's been done in places

07:01 like the United States where there's something like,

07:03 uh,

07:04 I don't know,

07:04 it was like electricity utilities or something like that.

07:07 There's a bunch of different independent ones.

07:10 And then,

07:11 you know,

07:11 Colorado is very different than Montana,

07:13 which is very different than Texas,

07:15 which is different than New York.

07:17 And you have these evaluate randomized evaluations in those places.

07:22 And then maybe that gives you enough of a variation to say,

07:25 well,

07:25 what predicts success?

07:26 So you can look at kind of like these things,

07:29 and then you can go to a 5th or a 6th uh state that I didn't

07:33 um mention here,

07:34 and then try to just see if the convex combination of those estimates

07:39 creates like you,

07:40 uh,

07:40 you know,

07:41 Arizona or something like that,

07:42 where you can say,

07:43 Well,

07:43 based on what I know from all these other places,

07:46 I think Arizona is going to have something like this.

07:49 Or you might say,

07:50 well,

07:51 really,

07:51 like,

07:52 not only we don't think we're gonna

07:54 be able to do this,

07:55 but also

07:57 running that kind of an experiment in our own,

07:59 um,

08:00 backyard

08:01 is not that costly.

08:02 So we're going to do that ourselves and then decide.

08:05 So,

08:05 there's really,

08:06 in my mind,

08:07 no correct answer here.

08:09 I do think

08:10 that,

08:11 um,

08:12 People like Lance Pritchett and Angus Deaton and others who say,

08:15 you look,

08:16 you shouldn't really

08:17 prioritize um RCTs.

08:21 They,

08:22 they do have a point.

08:23 Uh,

08:23 I often disagree with them about whether we should prioritize our cities or not,

08:28 but they do have a point in the sense that,

08:31 yes,

08:31 ideal randomized experiments may take priority,

08:34 but hardly ever any RCT is ideal.

08:37 And,

08:38 um,

08:39 You know,

08:39 sometimes the confidence intervals may be too large and sometimes,

08:43 uh,

08:43 you know,

08:43 our,

08:44 our,

08:44 our confidence in other aspects because of things like

08:47 attrition and balance or like compliance and things

08:51 like that that we're gonna discuss later,

08:53 um,

08:53 may be,

08:55 may be low.

08:56 So,

08:57 in those cases,

08:58 it is possible that actually,

09:00 despite the fact that you may have a biased estimate

09:03 from a

09:04 Um,

09:05 from a,

09:05 um,

09:06 let's say,

09:06 a non-experimental study.

09:09 Um,

09:10 it may be closer to the truth

09:13 than

09:14 the RCT from another place with a big confidence interval.

09:18 So,

09:18 and there's really no way to know

09:20 without doing it yourself,

09:21 so you have to do just some extra legwork.

09:24 Um,

09:25 but

09:26 other issues,

09:27 of course,

09:27 there's the ethical issues that come with RCTs.

09:30 I'm going to touch on them a little bit later on,

09:32 but the biggest thing for me is the concept of equipoise,

09:36 which many people

09:38 do not hear about the word until you actually do RCTs.

09:42 I didn't know the word,

09:44 like,

09:44 before,

09:45 um,

09:45 especially as a non-native English speaker.

09:48 It essentially means that,

09:49 like,

09:50 do I have enough uncertainty about the answer to this question?

09:54 That I should be going ahead and randomizing people,

09:57 because

09:58 it is really,

09:58 if you think about it,

09:59 it is really unethical,

10:01 you know,

10:01 just to,

10:02 let's say,

10:03 just to have an RCT so that you can have ironclad results

10:07 for a study that you can publish well in like a journal or something like that.

10:12 On a question

10:13 that we're pretty sure of the answer,

10:15 and we're pretty sure of the kind of like,

10:17 let's say the cost effectiveness and things like that,

10:19 then

10:20 you should just be targeting the program to people

10:22 who need it rather than just randomly allocating it.

10:25 But

10:26 often,

10:27 you know,

10:27 economists,

10:28 um,

10:29 can find something about which

10:32 there isn't any equipoise.

10:33 So,

10:33 for example,

10:34 we do know

10:35 Um,

10:36 that this program works,

10:38 but we want to know whether it works better than just giving people cash,

10:41 let's say,

10:42 or something like that.

10:43 So now,

10:44 what I want to do is I want to have a horse race,

10:46 and I really am saying that

10:48 I have no idea about the relative cost effectiveness of this program to the,

10:53 to its alternative.

10:54 So,

10:55 that's the evidence I want to

10:57 ascertain.

10:58 I know that this program is going to work.

11:00 It's just that I don't know whether it's going to do

11:02 significantly better than something else that I could have done.

11:05 Um

11:07 Then there are things when you do

11:10 RCTs type of studies

11:12 that come up,

11:13 and two of them that I like the most,

11:16 and they are really big dangers to kind of like internal validity,

11:19 which is something we said we,

11:21 we value,

11:22 the RCT proponents value.

11:25 And

11:25 one of them is Hawthorne effects,

11:27 which is essentially the idea that,

11:29 like,

11:29 just because someone is watching me,

11:31 um,

11:32 I behave differently.

11:33 And my favorite example on this,

11:35 and I don't know if we covered this last time or not,

11:38 is the,

11:39 is this idea that,

11:40 you know,

11:41 that,

11:41 you know,

11:41 you can think about

11:43 what's the effect of soap,

11:45 using soap on kind of like,

11:46 uh,

11:47 you know,

11:47 health outcomes or diarrhea or something like that.

11:50 And,

11:51 um,

11:52 I have colleagues at Stanford who work on sanitation issues.

11:56 They were worried about the self-reporting issue.

11:58 So they essentially put,

12:00 like,

12:00 kind of like these observant people,

12:02 like,

12:02 in the community,

12:04 to just kind of,

12:05 like,

12:05 be observing things.

12:07 And then somebody said,

12:08 well,

12:09 actually,

12:09 because they're observing,

12:10 people are just going to be washing their hands when they're around

12:13 and not going to be doing it when they're not around.

12:15 So,

12:16 what can we do?

12:17 And what they did is they literally put,

12:18 like,

12:19 motion sensors inside the soaps,

12:21 and they distributed soaps to everybody.

12:23 And

12:24 you can guess,

12:25 like,

12:25 where this is going,

12:27 when you,

12:27 the,

12:27 you know,

12:28 there's 3 ways to measure.

12:29 You ask people,

12:30 do you wash your hands?

12:31 And let's say 65% say yes,

12:33 that they wash it after going to the bathroom and before meals.

12:37 And then,

12:38 when you actually look at what the people observed,

12:41 like,

12:41 the enumerators,

12:42 that goes down to,

12:43 like,

12:43 30% or something like that.

12:45 And then when you go to the motion sensor,

12:47 like,

12:48 how many times did the soap move during the day,

12:51 that goes down to like 10,

12:52 15%,

12:54 right?

12:54 So,

12:55 it's not only the,

12:56 there,

12:56 there's the issue of importance of measurement here that's separate,

13:00 but the bigger thing is actually people are behaving differently

13:04 because they're in an experiment and somebody's watching them.

13:07 A different version of that is actually

13:11 John Henry effects,

13:12 and this is not about being watched,

13:13 but this is about the fact that

13:15 there is an experiment going on,

13:18 and people have incentives

13:20 about

13:21 the outcome of that experiment.

13:22 So,

13:23 think of a,

13:23 let's say a

13:24 school management or teacher incentive interventions,

13:27 and then you

13:28 kind of like assign some

13:31 schools or some

13:32 classrooms into a certain regime and some to the,

13:35 to another.

13:37 Now,

13:37 people in the

13:39 treatment group,

13:40 if they want the program to succeed,

13:42 they might work harder.

13:44 And people in the control group,

13:45 let's say if they don't want it to succeed,

13:48 they might also work harder to just show that

13:50 the treatment and the control difference isn't that different,

13:52 and vice versa.

13:53 If people want it to fail,

13:55 they may just try to sabotage it and so on.

13:58 So,

13:58 here,

13:58 what's happening is because you can't really do a placebo

14:03 in social sciences,

14:04 you can't just,

14:05 you know,

14:06 people are going to know whether they're in the treatment or the control,

14:09 um,

14:11 Then,

14:11 then that's going to,

14:13 that's going to matter.

14:14 Um,

14:15 and there's even a version of this that's actually very interesting,

14:18 and it was very innovative work,

14:20 where

14:22 You know,

14:22 you have people who are

14:25 in the vaccine group

14:27 or in the placebo group.

14:29 But depending on how they

14:31 Um,

14:33 What they think,

14:34 what they got.

14:35 Let's say

14:36 that you expect that you're going to have some

14:37 side effects when you get the Moderna vaccine,

14:40 because you hear from a lot of people that Moderna does,

14:43 like,

14:43 side effects X,

14:44 Y,

14:45 Z.

14:45 Right?

14:46 So,

14:47 you're going to think,

14:47 OK,

14:48 I'm getting X,

14:49 I'm getting Y,

14:49 I'm getting Z,

14:50 so I must be vaccinated,

14:52 so I can take my mask off and I can go into restaurants.

14:56 And if the outcome is determined both by the intervention and your behavior,

15:02 there's an interaction between

15:03 what you think your treatment status is and what your behavior is.

15:07 So,

15:08 that's going to change things.

15:09 So,

15:10 the way to deal with that is actually,

15:11 you can randomize.

15:14 People and then tell them the likelihood that they're in a group.

15:17 So that

15:18 if people

15:20 perfectly know

15:22 that they are one group,

15:24 and then,

15:25 you know,

15:25 they behave a certain way versus they perfectly know they're another group,

15:29 you can actually tease out the components where

15:32 the effect is due to only the

15:35 Vaccine because these people think.

15:39 They don't have the vaccine,

15:41 but they're behave,

15:42 so they're behaving accordingly,

15:44 and then there's these other people.

15:45 So you can,

15:46 you can do that kind of thing.

15:47 In practice,

15:48 it becomes difficult,

15:49 but these are the kind of theoretical things that you have to think about.

15:52 But nonetheless,

15:53 for you,

15:54 if you're thinking,

15:54 well,

15:55 OK,

15:56 I'm going to do this,

15:57 like,

15:58 pay some teachers for better performance,

16:00 and then there are going to be other teachers who know I'm doing this,

16:03 but they're in the control group,

16:04 you have to really think about how these teachers are reacting.

16:08 Just to the fact of the

16:10 RCT

16:12 rather than

16:13 this is now the new world,

16:14 and this is in which they're going to have these incentive programs and etc.

16:18 So you have to think carefully about those kinds

16:20 of things and whether they're going to affect your,

16:22 uh,

16:23 affect your,

16:24 um,

16:24 findings.

16:26 Right,

16:27 so now,

16:27 let's talk a little bit about what types of randomized experiments you might do.

16:31 So,

16:32 one of the things that we can think about is like,

16:34 the first thing that we think about is access,

16:36 right?

16:37 So,

16:38 There is something that like I can do where

16:41 I can essentially control access to a program.

16:44 Like,

16:44 it's a new program,

16:45 it doesn't exist anywhere.

16:46 There's nothing really like it around,

16:48 you know,

16:49 we can think of it as like new mRNA vaccine or something like that.

16:52 And then what I'm going to do is I'm going to randomly assign

16:55 some people to it and some people to a placebo or something.

16:59 I can do that,

17:00 but I can do it.

17:02 Around a certain cutoff of,

17:04 let's say,

17:04 vulnerability or something like that.

17:06 So you can think of this as

17:08 Um,

17:09 you know,

17:09 elderly people are more vulnerable,

17:11 people with certain conditions are more vulnerable.

17:14 So,

17:14 I'm going to only do it among those people,

17:17 and then I'm going to leave everybody below 65 and no conditions out.

17:22 Right.

17:23 The second thing you can do is

17:25 what are called phase-in designs,

17:26 and there are various um

17:28 versions of these that I'm going to show you in a second.

17:31 Um,

17:32 but you could essentially provide access to the program

17:35 in phases,

17:36 right?

17:37 So,

17:37 this district comes in first,

17:39 this other district comes in later.

17:41 This helps when

17:43 You know,

17:43 programs are new,

17:44 and then there isn't enough administrative capacity to do everywhere all at once.

17:48 And there may also be some learning

17:50 involved.

17:51 So you do just start slowly and then,

17:53 uh,

17:54 scale up,

17:54 um,

17:55 faster.

17:56 So you could do those kinds of things.

17:58 And while you're doing that,

17:59 you can evaluate the people who are phased in versus not.

18:02 If you can randomize those rather than

18:04 selectively decide where you're going to start,

18:07 um,

18:07 you can do that.

18:09 And then there's the encouragement types of designs.

18:12 Um,

18:12 these are

18:13 cases where I can't actually control the access to the treatment.

18:17 So,

18:17 for example,

18:19 um,

18:20 you know,

18:20 I'm going to give you an example later on.

18:22 Like,

18:23 we have a blog,

18:24 uh,

18:24 and I want to know the effect of the blog on people's knowledge on impact evaluations

18:29 or methods or something like that.

18:32 The blog is out there,

18:33 anybody can read it.

18:34 So,

18:34 how do I,

18:35 um,

18:37 How do I,

18:37 um,

18:39 evaluate the effect of this blog?

18:41 Well,

18:41 then I can say,

18:42 well,

18:42 I know that,

18:43 like,

18:43 actually,

18:43 even though this blog is out there,

18:45 not everybody reads it.

18:47 Um,

18:48 I,

18:48 in fact,

18:48 I know that,

18:49 like,

18:49 from my baseline surveys,

18:50 only 10% of the relevant population of graduate students that I care about,

18:56 they are

18:57 reading this blog.

18:58 So,

18:58 what I'm going to do is I'm going to send an email encouraging them.

19:02 And hope that like that will increase the percentage that's reading.

19:05 So now if I randomize the encouragement.

19:08 Then I have a group that's like reading,

19:10 you know,

19:10 10% of them are reading it.

19:12 Then I have another group that 30% of them are reading it.

19:15 Now,

19:15 that 20% point difference I can use to estimate something.

19:19 It's not going to be the same thing that I could estimate

19:22 with uh just randomly

19:24 controlling access,

19:25 but it's going to be still something

19:28 of potential interest.

19:29 And if that's something of potential interest

19:31 that's relevant to the policymakers or academics,

19:34 then you can do that.

19:36 Right?

19:37 So,

19:37 we can think of some examples here.

19:40 So,

19:40 for example,

19:41 one thing is,

19:42 you know,

19:42 one reason or an opportunity to randomize is

19:45 you actually want to address a problem,

19:48 but you don't actually know exactly how to do it.

19:51 So,

19:51 you have,

19:51 let's say,

19:52 23 Options and you want to try them.

19:54 So,

19:54 I think you have your target population,

19:57 and then among them,

19:57 you assign them to these different alternatives to see which one might work best.

20:02 And

20:03 you might do that aesthetically or adaptively.

20:05 I'm going to talk about adaptive experiments a little bit later on.

20:09 Um,

20:10 it could be that you're introducing a new service.

20:12 So in this case,

20:13 there's no

20:14 kind of,

20:14 like,

20:14 uncertainty about what you want to do.

20:17 But you just want to know whether this new service,

20:20 like,

20:20 will people take it up,

20:21 will it be effective,

20:22 etc.

20:23 So you might try to just kind of like,

20:25 um,

20:25 you know,

20:26 invite some of your customers

20:28 to,

20:29 to take this up or not.

20:31 Um,

20:31 you can do new people.

20:33 So,

20:33 for example,

20:33 Oregon,

20:34 um,

20:35 had some more money

20:36 to add people to its Medicaid program.

20:38 So suppose that

20:39 The program was originally targeted to people

20:43 at the poverty line,

20:44 and then now they decided that they can actually

20:46 increase it to 150% of the poverty line.

20:50 And so,

20:50 but there isn't enough money to cover like all of those people initially.

20:55 So what you can do is you can actually randomly in that band of 100 to 150%.

21:00 You can randomly assign the program to some and not the others.

21:04 In fact,

21:04 Oregon,

21:05 literally the public lottery

21:07 among the eligible people,

21:08 and then you can

21:10 wait a few years and then just,

21:12 you know,

21:12 like if you have an RCT you can show the effects and then that can inform

21:16 the policy on like,

21:18 how high to go from the poverty line and how many people to cover if it cost effective,

21:23 etc.

21:24 Right?

21:25 You can do oversubscription,

21:27 uh,

21:27 things.

21:27 So often we have things

21:29 that we want to put out there and then,

21:31 then so many people actually want them.

21:33 So,

21:34 if that's the case,

21:35 then what,

21:35 and we can't,

21:36 we can't

21:37 cover all of that demand,

21:39 we can say,

21:39 well,

21:39 look,

21:39 the most ethical way to kind of like distribute this is by lottery.

21:44 These people are all presumably equally eligible.

21:47 So,

21:48 we are oversubscribed for this,

21:50 like,

21:50 offer.

21:51 Um,

21:52 so we're going to do a lottery and then sign up some people,

21:55 and then,

21:56 because now the selection issue is

21:58 handled,

21:59 right?

22:00 There are people who said,

22:02 I am interested in this program.

22:04 Like,

22:04 let's say this was job training or something like that.

22:07 You don't want to compare people who

22:09 applied for job training,

22:11 and then you randomly selected to some people who didn't

22:14 apply for job training because they're very different people.

22:17 But now you have a,

22:18 with the subscription,

22:20 you have a bunch of people who all self-selected into saying,

22:23 I need job training,

22:24 but you still can't cover all of them.

22:26 So when you randomize,

22:27 you have a really comparable group of people that are actually relevant

22:31 for the program because they all kind of after this program.

22:35 The flip side of it that I hadn't thought about

22:37 before is that sometimes there's things like unders subscription.

22:40 So for example,

22:41 um,

22:42 think about,

22:42 let's say,

22:43 Ukraine or something like that right now,

22:46 you know,

22:46 they actually literally sub,

22:48 you know,

22:48 said everybody

22:50 who is between the ages of 16 or 18 and 50

22:54 has to fight right now.

22:56 Um,

22:56 but when it was happening in,

22:58 you know,

22:59 During the Vietnam War,

23:01 the US didn't need that many people,

23:03 so they wouldn't have said,

23:04 like,

23:04 OK,

23:04 we're having universal,

23:06 you know,

23:06 like a sub conscription.

23:08 But then on the other hand,

23:10 they,

23:10 there wasn't still enough people putting up their hands saying,

23:13 like,

23:13 I want to go fight and put my life in danger.

23:16 So,

23:17 they still needed to do a lottery to actually conscript people into the army.

23:22 And they did that on the basis of people's birth dates,

23:25 uh,

23:25 and things like that.

23:26 And then if you were unlucky,

23:28 um,

23:29 then you had to go fight a war that you may or may not believe in.

23:32 So,

23:32 undersubscription is actually also an opportunity for a lottery.

23:36 You can have rotations,

23:37 like think of Roscas and things like that.

23:40 Um,

23:40 communities or individuals can take turns to be,

23:44 uh,

23:44 the treatment,

23:45 uh,

23:46 unit,

23:46 so to speak,

23:47 while the others are controlled.

23:49 And then you can have admission cutoffs and

23:51 admissions and phases and things like that.

23:54 So,

23:54 let's give a couple of examples here,

23:56 but before I do that,

23:57 like a word about ethics.

23:59 So just because you can randomly assign people to intervention,

24:02 it doesn't mean we should.

24:03 We talked about equipoise,

24:04 so,

24:05 you should have a question

24:07 about which you have a lot of uncertainty,

24:09 and the question is really important.

24:12 Um,

24:13 from a policy perspective,

24:14 like that's one

24:16 barrier.

24:17 The second one is beneficence,

24:18 which is,

24:18 this is about the Belmont principles of like ethics that we all kind of subscribe to.

24:24 We take courses before we can do

24:26 studies,

24:27 uh,

24:27 with human subjects

24:29 or animal subjects for that matter.

24:31 And beneficence essentially means that you're trying to maximize the

24:35 Benefits to the people in your study and minimize the harms.

24:39 And one of the biggest,

24:40 um,

24:41 risks of harm is kind of like privacy concerns.

24:44 And so,

24:45 we,

24:46 we do a lot in our,

24:48 you know,

24:48 like a study designs on both kind of like

24:51 how to do informed consent and things like that,

24:53 etc.

24:54 how to deal with vulnerable populations such as

24:56 children or prisoners or disabled people and etc.

25:00 But then also

25:01 in the data handling,

25:03 uh,

25:03 processes where

25:04 We're anonymizing the data and we're keeping them in very secure,

25:08 like,

25:09 uh,

25:09 servers and,

25:11 you know,

25:11 all of that kind of thing.

25:12 There's protocols for that.

25:14 So I list some resources here and I know you all have my study slide,

25:18 at the slide deck,

25:20 so you can just go to these and,

25:21 like,

25:21 read these,

25:22 um,

25:22 read these uh references.

25:25 So,

25:26 a couple of,

25:26 like,

25:27 study types,

25:27 and I'll give examples of these.

25:29 So,

25:30 for example,

25:31 what you can do is you can do a lottery,

25:33 like,

25:34 which I call these,

25:34 like,

25:35 you know,

25:35 in the industry,

25:36 like,

25:36 let's say Google,

25:37 Facebook,

25:37 etc.

25:38 they call these like classic AB tests.

25:40 So,

25:41 AB literally means like,

25:42 I can do A or I can do B

25:44 in

25:45 the case of something.

25:46 Like Facebook or something like that.

25:48 Should I show the,

25:49 you know,

25:50 advertisements,

25:51 like,

25:51 on the side or like right under,

25:53 or should,

25:54 what the color should be or etc.

25:56 They're constantly doing these experiments,

25:58 right?

25:58 So,

25:58 that's the AB test,

25:59 but they're all kind of static.

26:02 Um,

26:02 let's say,

26:03 meaning that,

26:04 like,

26:04 you're not changing the assignment probabilities or,

26:06 or anything like that over time.

26:09 So,

26:09 Examples of this is like we did a cash transfer experiment in Malawi.

26:14 We gave some,

26:15 we

26:15 chose some villages to be

26:18 conditional cash transfer villages and some villages

26:20 to be unconditional cash transfer villages,

26:22 and some villages to be controlled,

26:24 right?

26:25 What you can do sometimes is that you might say,

26:27 well,

26:28 look,

26:29 Actually,

26:30 it really seems

26:32 very inefficient and also a little bit unfair and potentially demoralizing

26:37 to exclude people who are very suitable for a program,

26:41 but also to include people who are very unsuitable for the program.

26:46 And then we should really randomize only among the people.

26:50 That we're unsure about whether they,

26:53 they are good for it or not,

26:54 and they'll benefit for it or not.

26:56 And this is David McKenzie's Nigeria Business

26:58 Plan competition was an example of this,

27:01 where essentially they had

27:04 A competition where people submitted business plans.

27:07 And at the end,

27:08 there was a quite a big reward.

27:09 Like,

27:09 I don't remember if it was $50,000 or $250,000 or something like that.

27:15 But,

27:15 you know,

27:16 big reward at the end.

27:17 So what they did is like,

27:18 they had an independent panel

27:21 anonymously score all of these proposals.

27:24 And then let's say if you got like,

27:26 between 90 and 100,

27:28 you were given the award,

27:30 right?

27:30 And then let's say if you got below 50,

27:33 Then you were not given any award.

27:35 But the people between 50 and 90,

27:38 they weren't quite sure,

27:39 like,

27:39 they were decent proposals,

27:41 but,

27:41 you know,

27:41 they could be successful,

27:43 they could be seemed a little risky.

27:45 So,

27:45 among those,

27:46 they actually randomized

27:48 the,

27:48 um,

27:49 the intervention,

27:50 and then that's,

27:51 that's the basis of the evaluation.

27:53 So,

27:53 now,

27:53 what are you doing?

27:54 You're evaluating

27:56 uh

27:57 the effect of this business plan competition

28:00 for

28:01 Firms or like entrepreneurs

28:04 who are kind of marginal,

28:06 and by marginal,

28:07 I mean

28:08 They could go either way,

28:09 like the an independent panel didn't decide that,

28:12 like,

28:12 no,

28:12 this business is definitely not going to succeed,

28:15 this proposal is terrible,

28:17 or

28:18 this proposal looks amazing,

28:19 they already have some like proof of concept,

28:21 whatever,

28:22 etc.

28:23 So you've,

28:23 you're excluding those,

28:24 your evaluation isn't going to say anything about those kinds of things,

28:28 like whether giving them

28:29 $50,000 or $250,000 is

28:33 Effective or not,

28:34 but it's going to say something about this

28:36 marginal group and that's still very valuable.

28:39 So,

28:39 you can do something like that.

28:41 I like those designs a lot actually,

28:43 because they,

28:43 they take into account some targeting mechanism where

28:46 you're being a bit more ethical rather than

28:49 kind of like randomizing everybody where some people may

28:52 definitely need,

28:53 I mean,

28:53 if you think about,

28:55 instead of business plan,

28:56 you're thinking about a poverty intervention.

28:58 Now you're talking about

28:59 much more

29:00 Uh,

29:01 potentially,

29:02 uh,

29:02 devastating consequences for being in the treatment versus the control,

29:06 right?

29:07 Phase-in designs,

29:08 um,

29:09 the most famous example of this is the Worms

29:11 paper of Miguel and Kramer from Econometrica in 2004.

29:16 And this was the,

29:17 the famous deworming study in Kenya in the Busia district,

29:21 where

29:22 they had 75 schools and they divided them into three groups,

29:25 Group A,

29:26 B,

29:26 C,

29:26 25 schools in each group.

29:29 And essentially what they said is that in year 1,

29:31 we're going to treat Group A,

29:33 and then in year 2,

29:34 we're going to treat

29:36 Group B,

29:37 and then in year 3 onwards,

29:38 like everybody will be treated.

29:41 Now,

29:42 for the,

29:43 at the end of the first year,

29:45 you can do an evaluation where you're comparing treatment group one to treat,

29:49 you know,

29:49 the A to B and C.

29:51 In year 2,

29:53 you can actually compare A and B

29:55 to C,

29:56 but then you have to take into account the fact that

29:59 A is in the treatment longer and maybe that has some effects,

30:03 right?

30:03 And from year 3 onwards,

30:06 you can't really do,

30:08 They kind of like the comparison anymore other than

30:11 it's not a pure comparison other than now

30:14 the only difference between these groups is that

30:17 they've been exposed for different durations.

30:20 Now,

30:20 there's one exception to that,

30:22 which is that

30:23 if you actually think that

30:25 You're doing things for cohorts inside the schools,

30:29 and there are going to be cohorts that just miss out and they never get treated.

30:33 So,

30:33 suppose that we say,

30:35 anybody above grade 8

30:37 is no longer eligible for deworming.

30:39 This is not true for deworming,

30:41 but I'm just hypothetical.

30:43 Um,

30:44 then what you can do is,

30:45 like,

30:46 the people who were in grade 8 in the comparison group

30:50 in B and C.

30:52 They will never get treated

30:54 going forward.

30:55 The,

30:55 the grades 8 the next year and the year after that will get always treated,

31:00 but they won't.

31:01 So you can take grade 8

31:03 in Group A versus B and C,

31:06 and forever,

31:07 you have a pure

31:09 Kind of like RCT where the control group has never been treated,

31:13 right?

31:14 So,

31:14 if that's possible,

31:15 and if there is,

31:16 if that's actually legitimate,

31:18 meaning that

31:19 maybe there's effects on these grade 8 people from the fact

31:22 that they're like 1 year younger people around them are getting

31:26 treated and then there's spillovers and etc.

31:28 then it's still not great,

31:30 but you can think of it that way.

31:32 But those are the phase and designs and

31:34 The reason why people don't like phases in design,

31:37 some people at least,

31:38 is because it's hard to do long term evaluations,

31:42 because for,

31:42 think of my study in Tanzania,

31:45 where we had people apply for business,

31:47 um,

31:49 Kind of like income generating activities

31:51 at the group level,

31:53 and

31:54 there are really no cohorts,

31:55 like comparable cohorts or anything.

31:57 They're like 100 villages,

31:58 in 100 villages,

31:59 there are 100

32:01 groups that applied for something.

32:03 And we said to 50 of them,

32:04 you will get your money now.

32:06 And then the other 50 of them,

32:08 we said,

32:08 you'll get your money next year,

32:10 right?

32:10 So we really have 1,

32:12 after 1 year,

32:13 they'll get a huge infusion of money.

32:15 And it's hard to argue that,

32:17 like,

32:17 at,

32:18 after that point,

32:19 there is this kind of any difference,

32:21 really,

32:21 other than the fact that one group had their money 1 more year.

32:25 Right?

32:25 And so,

32:26 you essentially have only one year

32:29 to kind of evaluate the effect of the program.

32:31 And if you don't think that you're going to see

32:34 the effects that you're hoping to see in that duration,

32:37 then a phase-in design isn't really good for you.

32:40 And then even,

32:40 even if you do,

32:42 it's,

32:42 you know,

32:43 it,

32:43 it's

32:44 People may say,

32:44 well,

32:45 actually,

32:45 I would really like to know what happens like 4 years later,

32:48 maybe they converge back or something like that.

32:50 You don't really have an opportunity to see that because you essentially have this

32:55 group,

32:55 comparison group has been treated.

32:58 The encouragement design,

33:00 I mentioned it already,

33:01 like the impacts of economics blogs,

33:03 David McKenzie and I like wrote a paper on this and published

33:06 it because we essentially did exactly what I described to you.

33:10 We wrote an email to

33:12 like 300 students and their advisors at the economics department saying,

33:16 hey,

33:16 we have this new blog,

33:17 like,

33:18 why don't you check it out?

33:19 And then 300 others that we had listed before and randomly excluded,

33:24 we didn't.

33:25 And it did really cause a difference in how many people do report reading our blog,

33:30 and then we could use that difference to just kind of um that differential exposure

33:35 to evaluate whether people

33:37 knew about development issues a little bit better by reading our,

33:41 um,

33:42 by reading our um blog.

33:45 But

33:45 note here is that like,

33:47 you're not any longer

33:49 evaluating the intention to treat effect,

33:51 but you're evaluating the,

33:53 what is called the local average treatment effect.

33:56 And how it's defined is that that's essentially you're

33:58 evaluating the effect of the program on people.

34:02 Who would not have

34:05 taken up the intervention,

34:06 the program,

34:07 in our case,

34:08 read our blog

34:09 on their own,

34:11 i.e. without the intervention,

34:13 but would have read it

34:15 if we intervene,

34:16 right?

34:17 That's a very specific group,

34:19 but that may be exactly the kind of group that you're interested in.

34:22 So for example,

34:22 if you're New York Times,

34:24 and you're saying,

34:26 should we advertise because we have like,

34:28 you know,

34:28 we have huge readership,

34:29 but it could be even bigger.

34:31 And what if we advertise,

34:33 which is kind of like our emails to people,

34:36 then

34:36 it will get them to sign up or not,

34:39 right?

34:39 And so,

34:40 that's exactly the group that you're interested in,

34:43 the people who would not have read the New York Times on their own,

34:47 but if you advertise,

34:48 they will.

34:49 And you want to know how big that group is,

34:51 and then what their characteristics are,

34:52 and like,

34:53 whether they'll pay,

34:54 how much they would like to pay to sign up and all of that.

34:57 So,

34:58 that

34:59 That might be the

35:00 estimate of interest,

35:02 but it may not be.

35:03 So depending on that,

35:04 you can do these encouragement designs.

35:06 The encouragement designs need to satisfy instrumental variables,

35:10 uh,

35:11 criteria,

35:12 which is that

35:13 They need to actually,

35:15 your encouragement

35:17 should only work through people taking

35:19 up the intervention and shouldn't directly affect

35:23 the,

35:23 um,

35:24 the outcome

35:25 itself,

35:26 right?

35:27 So you could imagine,

35:27 for example,

35:28 that if in our blog,

35:31 Email.

35:32 We had also given people some information.

35:34 We said,

35:35 like,

35:35 not only we said,

35:36 oh,

35:36 here's a new blog,

35:37 but also your other choices of reading things are like Chris Plattman

35:41 and uh Paul Krugman and this other blog and Tyler Cowen,

35:44 etc.

35:45 Um,

35:46 then all of a sudden,

35:48 We can't really use it anymore to isolate the effect of its effect on

35:53 because now we've essentially given people

35:55 like a bunch of other information that could influence their

35:59 knowledge.

36:00 So now,

36:01 we violated one of the,

36:03 um,

36:03 you know,

36:05 necessary conditions for an instrumental variable to work,

36:07 which is the exclusion criteria.

36:09 So,

36:10 you need to essentially think about

36:12 Is my encouragement just really only going to work through the intervention.

36:17 So,

36:17 for example,

36:18 think of the kinds of things like David McKenzie and

36:20 his colleagues at the finance group do with like banks.

36:24 Banks will just,

36:25 let's say,

36:26 send a mail,

36:27 like even like a postcard to their clients saying,

36:30 we have a new credit card,

36:31 like 3% interest rate or something like that,

36:34 like,

36:34 please,

36:35 you know,

36:35 it's hard to think that that's going to affect your finances,

36:40 seeing that.

36:41 You know,

36:42 other than you actually apply and like,

36:44 get the credit card.

36:46 Um,

36:46 so,

36:47 that's the kind of thing you want to be doing,

36:48 very light touch encouragement that hopefully works to

36:51 increase substantially the takeup of the program,

36:55 but it doesn't affect anything else.

36:59 So,

36:59 another version of phase-in is this kind of like step wedge design.

37:03 These are very popular in medicine.

37:05 So,

37:05 for example,

37:06 this is how the Gambia famous hep B vaccine

37:09 trial was done.

37:10 You can see here that the light blue areas are

37:13 areas where like people are waiting that many

37:16 weeks to be exposed to the intervention.

37:18 And then

37:19 other,

37:20 you know,

37:21 the dark blue is the one that you're actually now exposed to the intervention,

37:25 and you can essentially compare the people who

37:27 are in there versus who are not in there

37:31 throughout the,

37:32 the transition period.

37:34 But um,

37:35 You have to worry about two things.

37:37 One is time trends.

37:38 So,

37:38 for example,

37:39 some intervention may work better

37:42 um

37:42 now than

37:44 it,

37:44 it,

37:44 it will in like 3 weeks' time,

37:47 mainly because,

37:48 you know,

37:49 potentially,

37:50 maybe the government says like mask mandates or something like

37:52 that and all of a sudden vaccines aren't that important,

37:56 as opposed to,

37:57 it's a free for all and the government says there's no reason to worry.

38:01 Everybody can go anywhere they want.

38:03 And then maybe vaccines are much more important then.

38:06 So,

38:06 there's those kinds of things to worry about.

38:08 But then there's also anticipation effects,

38:10 and this was actually something that we

38:13 think

38:14 really,

38:15 um,

38:16 derailed are like the Tanzania work that I

38:19 talked to you about,

38:20 which is that,

38:22 you know,

38:22 you're saying to people,

38:23 well,

38:23 you'll get your money next year.

38:25 And we essentially thought that,

38:27 well,

38:27 this is an ultra poor program.

38:29 These people are very credit constrained.

38:31 So,

38:31 there isn't really much that they can do

38:34 in anticipation of getting the money next year.

38:36 But actually,

38:38 that turned out maybe that we have some suggestive

38:40 evidence that that may not have been true,

38:43 that people can actually say,

38:44 well,

38:46 You know,

38:46 I'm getting this big amount of money next year.

38:49 So,

38:49 and it's a credible promise.

38:52 So I can literally borrow against that promise.

38:55 Like,

38:55 it's almost becomes my,

38:56 like,

38:57 collateral,

38:58 and I can start my business

39:00 things,

39:01 like,

39:01 now.

39:01 Now

39:02 in anticipation,

39:03 or maybe I'm going to eat more or eat less or whatever.

39:06 There's all sorts of things that I can do in anticipation,

39:09 all of which

39:10 kind of bias the effects because if people didn't know

39:13 that they were going to be treated next year,

39:15 they would have acted differently.

39:18 And the final thing that I like this a lot,

39:20 and this is actually also used in medical trials,

39:23 and this is called a fried egg design,

39:25 and then you can look here at the bottom,

39:27 the kind of like the top left corner.

39:30 You know,

39:30 you,

39:30 this blue area here is a,

39:33 is a cluster,

39:34 it's a treatment cluster,

39:36 so everybody in that blue area is assigned to,

39:39 let's say,

39:39 the vaccine.

39:41 And then everybody in the brown area is assigned to placebo,

39:45 right?

39:46 But the brown area and the

39:48 um

39:49 blue area are adjacent to each other,

39:51 and I worry that there's going to be spillovers here.

39:54 So,

39:54 what I do is

39:56 I actually take the yolk of the egg,

39:58 so to speak,

39:59 and this is the kind of the white parts of the egg,

40:01 they are fried egg.

40:03 And then I only study the people in the yolk,

40:06 because they are only surrounded by people who are also vaccinated,

40:10 and these people are only surrounded by people

40:13 who are

40:14 unvaccinated.

40:15 So,

40:16 then,

40:16 essentially,

40:16 my assumption is that these people and these people

40:20 are not going to interact with each other at all,

40:22 so I'm going to get a really pure estimate.

40:25 Um,

40:25 rather,

40:26 or,

40:26 or their

40:28 amount of spillover on each other is going to be much,

40:30 much less than these people.

40:32 So,

40:32 if I studied them also,

40:34 um,

40:35 that would,

40:35 that would have been a problem,

40:37 right?

40:38 So,

40:39 And that brings me actually to kind of

40:41 like this idea of clustered versus individual randomization.

40:45 And then we talked about the stable unit treatment

40:48 violation uh value assumption last time or the Suva.

40:52 You know,

40:53 so remember that aspirin example that we were talking about last week

40:58 or two weeks ago,

40:59 you know,

41:00 does my headache depend on also you taking an aspirin?

41:03 Probably not.

41:05 But maybe,

41:05 like,

41:06 if actually,

41:07 you know,

41:07 we live together and you just become an awful person,

41:10 like when you have a headache,

41:12 then you start giving me a headache.

41:14 Maybe you taking an aspirin helps me get it,

41:17 you know,

41:17 like,

41:18 not get a headache or not.

41:19 So,

41:20 we have to always think about.

41:22 Is there

41:24 Is there any possibility of interference

41:27 from the treatment units to the control units?

41:30 And if so,

41:31 how do I deal with that?

41:33 And I think this is a big enough problem,

41:36 especially when you're trying to just kind of

41:38 publish papers.

41:39 Now,

41:39 referees are very attuned to this,

41:41 like,

41:41 you will always get these questions,

41:43 and it's,

41:44 um,

41:45 you know,

41:46 We do see

41:47 papers getting,

41:48 you know,

41:48 good papers getting

41:50 kind of like knocked and,

41:52 you know,

41:52 rejected because they can't really,

41:54 they've done an individualized randomization or something like that,

41:57 so they can't really rule out that there's no spillovers.

42:01 Um,

42:01 and so it

42:03 ends up in,

42:03 you know,

42:04 you have a perfectly good RCT at the kind of like the individual level,

42:08 you've done everything correctly.

42:09 But now people are saying,

42:11 hey,

42:11 how do I know that like,

42:12 that person and that person aren't friends,

42:14 and then they're like teaching each other the things they learned,

42:17 whatever.

42:18 And that's,

42:19 that,

42:19 that becomes a problem.

42:20 So I think it's important to think about these things.

42:23 In many cases,

42:24 there are where Suwa is violated,

42:26 like in a pandemic,

42:28 in agriculture where wind is going to take like your,

42:31 you know,

42:32 seeds and the sprays and things like that across.

42:35 So you have guard rows and agricultural experiments.

42:39 You can have general equilibrium effects.

42:41 So,

42:41 the more people I do job training with,

42:44 the more spillover effects they will have on

42:46 people who are in the control group because they

42:49 will get jobs and then they'll take the

42:51 jobs that those other people could have gotten.

42:53 So,

42:53 if there's actually a

42:55 Limited amount of,

42:56 like,

42:57 let's say,

42:57 winner take all kind of reward,

43:00 and I'm favoring one group over the other,

43:02 all I could be doing is causing displacement and then putting people,

43:05 some people in front of the queue rather than the back of the queue.

43:08 And so that's uh interference,

43:11 or in classrooms and in schools and things like that,

43:14 if I'm like,

43:15 let's say,

43:15 tutoring some people,

43:17 then they could maybe going home,

43:18 like,

43:19 and then,

43:19 you know,

43:19 helping their friends and etc.

43:21 who are in the control group.

43:23 So,

43:23 you can think of many cases like this.

43:26 So,

43:26 I think

43:27 what you want to do,

43:30 generally speaking,

43:30 if you're worried about this,

43:32 you want to

43:34 Think of like some clusters where that interaction is minimized.

43:38 So,

43:38 let's say villages in our Malawi case were kind

43:41 of like 5 kilometers far from each other.

43:44 And towards the end of the intervention,

43:46 we actually,

43:46 you know,

43:47 actually,

43:47 after the intervention finished 2 years later,

43:50 when we were talking with some

43:52 You know,

43:53 people in the control group in their follow-up surveys,

43:55 we were really happy to see that actually,

43:58 they had no idea

44:00 that there was a cash transfer program going on in other parts of the same district.

44:04 Um,

44:05 but then on the other,

44:06 uh,

44:06 uh,

44:06 and,

44:07 you know,

44:07 for good reason,

44:08 because if they know,

44:09 then there could be,

44:10 you know,

44:10 mental health effects,

44:12 they could be like,

44:12 oh,

44:13 you know,

44:13 compliance issues and

44:15 things like that.

44:16 They could be asking for money from treated people and so on and so forth.

44:20 So,

44:21 You want to do kind of like sufficient.

44:24 Distance between your subjects,

44:27 whether that distance is social or geographic or otherwise.

44:31 Um,

44:32 and then I think you run into a

44:35 Like a

44:36 main trade-off you have is power,

44:39 because individually randomized,

44:41 uh,

44:41 trials are very powerful because you essentially have

44:45 each person randomized,

44:46 so the statistical power is just much higher.

44:50 And the power loss comes from the fact that within clusters,

44:53 you have a lot of correlation.

44:55 So,

44:56 think about the,

44:57 Case where

44:59 think of households

45:00 as

45:01 clusters,

45:03 right?

45:03 But then think about something that's very,

45:05 very correlated within a household.

45:07 So,

45:07 like COVID actually now is the Omicron is a good example.

45:10 I know many people,

45:11 once their kid comes back from school and then has COVID,

45:14 the whole family gets COVID very soon,

45:17 right?

45:17 So,

45:18 in that case,

45:20 It's not going to help me to sample 2 or 3 people from that household.

45:24 It's going to give me zero extra power if

45:27 everybody is going to get COVID in that household.

45:29 Like,

45:30 it's perfectly correlated,

45:31 then all I need from that cluster is just one person,

45:35 right?

45:36 But that means,

45:36 what that means for my power calculations is that I need many,

45:40 many more clusters to get,

45:41 like,

45:42 good,

45:42 good inference,

45:43 right?

45:44 Whereas if things aren't that correlated,

45:47 You know,

45:47 let's say it's actually cancer that's not environmentally related,

45:50 but like otherwise.

45:52 And,

45:52 you know,

45:53 so then I think,

45:54 you know,

45:55 you can have 3 different people from the same household,

45:58 their outcomes aren't that correlated with each other,

46:01 that's fine.

46:02 So then you,

46:02 you,

46:03 you just still get benefits from,

46:04 like,

46:05 increasing the cluster size

46:07 rather than increasing the number of clusters.

46:09 And that's essentially the main

46:11 calculation that you're making when you're trying to do individual versus

46:16 Um,

46:16 individual versus clustered

46:18 design.

46:21 So we.

46:23 And so now,

46:24 I want to talk a little bit about factorial designs.

46:28 So,

46:29 What that means is like,

46:30 so some people also call these 2 by 2,

46:32 if there's 2 interventions that you're thinking about,

46:35 but if you,

46:36 there's 3,

46:36 it could be a 3 by 3,

46:37 whatever,

46:39 right?

46:39 So,

46:39 essentially,

46:40 you're thinking about,

46:42 I have,

46:42 let's say,

46:44 Um,

46:45 cognitive behavioral therapy for,

46:48 uh,

46:49 troubled youth,

46:50 right?

46:51 Um,

46:51 there's a recent paper that came out now some by some of my colleagues,

46:55 and then you're thinking,

46:56 well,

46:56 actually,

46:58 Therapy for troubled youth is great to reduce like

47:03 uh crime and,

47:04 you know,

47:05 to reduce their like behavioral issues,

47:07 but

47:09 They don't really have the bandwidth to,

47:11 like,

47:11 internalize all of that therapy and to deal with their lives because they're so poor

47:16 that

47:17 they have to just be,

47:18 you know,

47:18 like,

47:19 either stealing stuff or they have to be constantly worried about,

47:22 like,

47:23 you know,

47:23 where the next meal is going to come from and so on and so forth.

47:27 So this program would be much more effective.

47:30 If we just gave them like a little stipend also along with it,

47:33 so.

47:34 So,

47:34 that's your hypothesis.

47:35 Your hypothesis is that if I combine these two things,

47:39 it's going to work much better

47:41 than if I didn't combine them.

47:44 Now,

47:44 a lot of people,

47:45 what they will do is that they will cross them fully,

47:48 meaning that you have a group that's getting neither intervention.

47:51 You have a group that's getting the cash,

47:53 you have a group that's getting the,

47:55 the CBT and you have a group that's getting both,

47:59 right?

48:00 In those kinds of things,

48:02 you know,

48:02 I think so the way I described it,

48:05 like,

48:05 my intention really was like,

48:07 is CBT better

48:09 without cash?

48:09 So,

48:09 what I really care about is like,

48:12 CBT plus cash versus CBT.

48:15 It's just possible that some other people might say,

48:18 I'm actually also interested in CBT plus cash versus cash,

48:22 right?

48:23 Uh,

48:23 in one of the studies that I did that I'm linking here,

48:26 the,

48:27 we were working with the International Rescue

48:29 Committee in Liberia for adolescent girls.

48:32 They said,

48:32 actually,

48:33 we're not interested in that question at all because

48:35 we're never gonna give just cash to people.

48:38 Um,

48:39 you know,

48:39 it,

48:39 at least this was like 5,

48:41 10 years ago,

48:42 you know,

48:42 it's like,

48:42 that's not in our programming currently.

48:44 What we really do want to know though,

48:46 is that if we add,

48:47 let's say 50 cents per like attendance to our mentoring program,

48:51 will it work much better?

48:52 So,

48:53 we just want to compare that.

48:54 So,

48:54 for that purpose,

48:55 I'm going to discuss this in a second,

48:57 we actually did

48:59 Only 3 groups.

49:00 So,

49:00 we had a control group,

49:02 we had the mentoring program,

49:03 and then we had the mentoring program that's like,

49:06 conditional cash for attending the mentoring sessions.

49:09 So,

49:10 like,

49:10 each time you,

49:11 uh,

49:11 you know,

49:11 attend one of the 40 sessions,

49:14 you get,

49:14 like,

49:14 $2 or something like that.

49:16 So over the year,

49:17 that adds up to some money.

49:18 Yeah.

49:19 But

49:19 generally speaking,

49:20 you have these 2 by 2s.

49:23 And

49:23 what happens is that if you have these 2x2s,

49:26 you have to design and analyze them correctly.

49:28 And David McKenzie wrote a really nice,

49:31 um,

49:32 uh,

49:33 kind of,

49:33 um,

49:34 blog post on this like a while back,

49:37 and I covered it again recently.

49:40 So,

49:40 most of the time,

49:42 there's the long model up top,

49:43 which is that you have to have a dummy variable to analyze these,

49:47 like interventions

49:49 for each cell,

49:50 right?

49:50 So you have the,

49:51 the T1,

49:52 let's say your CBT T2 is the cache and then T1 and times T2

49:57 is the CBT and cache,

49:59 and then what you're comparing them to is the control group.

50:02 But often,

50:04 what people will say,

50:05 and this is kind of like where things go really wrong,

50:09 is that people will say,

50:10 well,

50:10 actually,

50:11 That

50:12 thing that we added

50:14 really didn't do anything.

50:15 That B3 is kind of like insignificant and small.

50:19 So,

50:19 what we're gonna do is,

50:21 it'll be much more power if we can take those people in that cell

50:25 and then reallocate them into their T1 and T2 bins,

50:29 and then we just run this short model,

50:31 which is the second thing over here.

50:33 And then

50:34 that'll give us like much better,

50:35 much better idea.

50:37 Essentially,

50:38 there's a couple of things that are wrong with that.

50:41 One of them is like,

50:42 actually,

50:44 What that's doing is now,

50:45 it's averaging,

50:46 let's say this T1 is CBT,

50:49 it's averaging it

50:50 in a world where

50:52 people aren't giving cash.

50:55 And then in a world where they are given,

50:57 let's say some significant amount of cash,

50:59 and

51:00 the average of that is,

51:02 may not be of interest really to the policymaker.

51:05 The policymaker is either going to be in one of those worlds or the other,

51:08 but not,

51:09 not the kind of like the convex combination of it,

51:11 whatever weights you put on that.

51:13 Right?

51:14 Um,

51:14 so,

51:15 that's,

51:16 that's one problem,

51:17 like,

51:17 it's just,

51:18 especially if this was a more eclectic thing that you were doing,

51:22 you know,

51:22 there's no way,

51:23 let's say you're going to be giving cash,

51:25 but also like some people are going to be in like government provided therapy.

51:29 and some people aren't,

51:30 you know,

51:31 that just sounds very silly.

51:32 So that's number one.

51:34 But then there's a second problem,

51:35 which is a statistical problem,

51:37 like the,

51:38 the people who wrote this paper,

51:40 like Mura Idaran and his colleagues,

51:42 uh,

51:42 that I,

51:43 that I kind of linked to here,

51:45 um,

51:46 they show that

51:48 When that B3 that you said is insignificant,

51:51 even if it's modestly positive,

51:54 like,

51:54 or modestly different than 0.

51:58 You get false rejection rates,

52:00 meaning that like you're essentially

52:02 saying that K1 is significant when you shouldn't actually be saying that,

52:07 right?

52:07 So the,

52:08 the explanation of that gets technical,

52:10 but that's essentially the,

52:11 so,

52:12 when you design a factorial design,

52:14 you really need to then evaluate it correctly like this,

52:18 right?

52:18 Or you can design it differently,

52:19 which I'm going to talk about in a second.

52:22 There's a bonus point here that's just kind of more general,

52:26 like,

52:26 it's important what the statistical power of your study is,

52:30 because suppose that you have a very low powered study,

52:34 right?

52:34 But then you're still getting

52:36 things that are very close to significant but not quite.

52:40 Like,

52:40 that actually can be evidence

52:43 against the null,

52:43 meaning that

52:45 You actually a P value of let's say 0.11 or 0.17,

52:49 which we would normally say is that,

52:51 oh,

52:51 this is not significant,

52:52 because your study is so low powered,

52:55 it's actually very unlikely that you would get that

52:58 P value if there was nothing going on.

53:00 So that can be actually evidence for rejecting the null hypothesis,

53:04 even though

53:05 traditionally you would say,

53:06 I don't reject.

53:08 Right?

53:09 Conversely,

53:10 if you have a very,

53:11 very powerful study,

53:12 suppose that you have like millions of observations or something like that,

53:15 and then you're getting a P value of 0.04,

53:19 now actually,

53:20 that can be such that like in that data set,

53:23 that's actually

53:25 More likely to be evidence for the null hypothesis that there's nothing going on,

53:30 rather than

53:31 there's something going on.

53:33 Because you should really be getting P values of 0.00,

53:36 you know,

53:37 0.004 rather than 0.04.

53:40 So,

53:40 it's important to always think about kind of like,

53:43 what's the power of your study and generally speaking,

53:46 design studies that are

53:48 Appropriately powered and not underpowered because that's when

53:51 you're going to be finding these kind of like,

53:54 create more likely to be finding these crazy results because

53:57 the only things that you'll find are very big effects.

54:00 So,

54:00 you'll just be saying,

54:01 oh,

54:02 look,

54:02 at

54:03 therapy is like the best thing in the world,

54:05 like,

54:06 and,

54:06 but you have like 7 observations and then

54:09 that,

54:09 that's what it's coming out of,

54:11 right?

54:11 Just by chance.

54:13 Um,

54:14 so,

54:15 You can

54:16 You can leave the interaction cell empty,

54:19 that's another option.

54:20 So if you're interested in 2,

54:21 you know,

54:22 in my example in Liberia,

54:24 we left the cash cell empty,

54:26 right?

54:27 But you can also leave the,

54:29 you know,

54:29 you can say,

54:30 I'm going to compare cash to the therapy,

54:33 and then I'm not gonna have cash plus therapy,

54:35 right?

54:36 More unconventional,

54:38 you can also choose not to have a control group.

54:40 You can say,

54:40 look,

54:41 I know CBT works.

54:43 I know cash works.

54:45 All I care about is like,

54:47 does cash plus CBT

54:49 have this multiplicative effect that nobody's thought about,

54:53 so I'm gonna compare those.

54:55 To,

54:56 like,

54:56 to the,

54:57 each of these single interventions.

54:59 I don't really need to show the efficacy of these things against the control group.

55:03 People don't like those,

55:04 mainly because in a lot of fields,

55:06 there's not really very well established

55:09 um evidence that these things work.

55:11 And then also,

55:12 even if there is,

55:13 people will say,

55:14 well,

55:15 how do I know that like,

55:17 it wouldn't have been like no effect in this setting.

55:20 So,

55:20 like,

55:20 it's,

55:21 you

55:21 always want to have it there.

55:22 So it's a bit risky.

55:24 Um,

55:25 but,

55:25 you know,

55:25 you could,

55:26 you could consider doing it.

55:27 You don't have to have,

55:28 always have to have

55:29 a 2x2 or a 3x3

55:32 when you're

55:33 thinking of multiple interventions.

55:36 So the third thing that I want to talk about,

55:38 which is,

55:39 this is something that you may not hear

55:40 about like elsewhere and it's interesting to hear about

55:43 is adaptive experiments,

55:45 right?

55:46 So,

55:46 so far what we've discussed are static designs and what do I mean by static design?

55:51 Like I have 10 of you guys there.

55:53 I want to know something about you,

55:55 how you will react to a certain thing.

55:57 Let's say like,

55:58 we only invite 5 of you to this lecture and then 5 of you not,

56:02 right?

56:03 So,

56:03 what

56:04 happens is that

56:05 all 10 of you

56:07 I just

56:08 randomize once.

56:10 5 of you are here,

56:11 5 of you are not here.

56:13 I run my experiment.

56:14 It's the length of this,

56:16 uh,

56:16 lecture,

56:17 and then,

56:18 like,

56:18 you know,

56:18 22 weeks later,

56:20 I do a survey,

56:21 and that's it,

56:22 right?

56:23 If it was an adaptive trial,

56:25 I could think of saying,

56:27 well,

56:27 actually,

56:28 I'm going to take two of these people.

56:30 I'm going to randomize them.

56:32 Um,

56:33 into

56:34 You know,

56:35 like,

56:35 uh,

56:36 let's say,

56:37 you know,

56:37 this lecture,

56:38 Berke's lecture,

56:39 David's lecture,

56:40 and like,

56:41 you know,

56:41 nothing or something.

56:42 I'll,

56:42 I'll take 3 of you,

56:44 right?

56:45 Then,

56:45 based on that I'm gonna survey you.

56:48 And then based on that,

56:49 I'm going to invite the next 3 people,

56:52 and then,

56:53 but I'm going to now change the assignment probabilities

56:56 because it looked like actually David's lecture

56:59 was doing much better than Berg's lecture,

57:01 so I'm gonna assign more people to David's lecture and less to Berke's lecture,

57:05 right?

57:06 And then,

57:06 I'm gonna accumulate that evidence and then assess it again,

57:10 and so on and so forth until I run out of people.

57:13 Right?

57:14 So,

57:15 you can see that like this requires a couple

57:18 of things that are different than static experiments,

57:20 and that's why they're not so common in social sciences,

57:23 although

57:24 they are more suitable definitely for medical context.

57:27 One of which is that

57:29 I need to have

57:31 Access to either the outcome

57:34 or a very good proxy of the outcome,

57:37 very quickly

57:39 before

57:39 I have to enroll the next batch of people,

57:42 because I want to evaluate this batch,

57:44 and then I want to change my assignment probably.

57:48 So,

57:49 for example,

57:49 in my

57:50 um

57:52 family planning work,

57:54 what I care about is like,

57:55 are people who say they don't want to get pregnant,

57:58 getting pregnant within a year,

58:00 so that a year is too long.

58:02 But I have something where

58:04 there's a,

58:05 you know,

58:05 they adopt,

58:06 I can see immediately,

58:07 like,

58:08 literally at the end of my counseling session,

58:10 whether they adopt a long-acting contraceptive method like

58:13 the IUD or the implant or not.

58:15 Now,

58:16 I know that those two things are 99.9% effective

58:20 in preventing pregnancies.

58:22 So,

58:22 essentially,

58:22 I have an extremely good proxy of,

58:24 like,

58:25 these people who have this implant in their arm.

58:28 They're not going to get pregnant next year.

58:29 So,

58:30 even though

58:31 I don't have the outcome

58:33 in hand,

58:33 I have a very good proxy,

58:35 so I can say,

58:35 actually,

58:36 I'm gonna use that proxy to update my

58:38 success probabilities,

58:40 um,

58:41 because I can see that.

58:42 That's #1.

58:43 Number 2,

58:45 It's more suited to contexts where

58:47 there's enroll,

58:49 enrollment is like continuous.

58:51 So think about like a hospital,

58:53 in my case,

58:54 the family planning unit and the maternity unit.

58:56 Women are constantly coming in every day with like requests or deliver a baby,

59:01 or like the,

59:02 you know,

59:02 have miscarriages and vaccinations for their children,

59:05 etc.

59:07 Every day,

59:08 I have a new opportunity to enroll like 1015,

59:11 20 more people.

59:12 So,

59:13 essentially,

59:14 I can say,

59:15 this week is a batch,

59:16 next week is another batch,

59:18 and then at the end of each week,

59:19 I'm just going to quickly analyze the data

59:21 and then change the assignment probabilities next week.

59:24 Like,

59:24 if you don't have that,

59:26 this option is not completely closed to you,

59:29 but you have to have a big enough sample.

59:33 And then also worry about these kind of like anticipation effects

59:36 and all this kind of spillovers and things like that,

59:39 where,

59:39 for example,

59:40 our colleague Anya Sottman did is that they had these schools um

59:44 in the country that they're working in,

59:46 and they had an intervention,

59:48 and they just divided that school

59:51 population into two.

59:52 And then batch A,

59:53 half of the schools,

59:55 and then they,

59:56 they essentially experimented

59:58 with half of the schools in batch A.

1:00:00 They assess that data after 3 months,

1:00:03 they learned something from it.

1:00:04 And then with what,

1:00:06 equipped with what they learned,

1:00:08 then they went to batch B,

1:00:10 and then they redid the experiment there.

1:00:13 So that's a version of this.

1:00:15 It's less continuous version of it,

1:00:16 and you're a little bit more uh limited because maybe

1:00:20 you have only like 500 schools and that's all you can do.

1:00:22 Whereas,

1:00:23 with my medical examples,

1:00:26 I can have like infinite many people

1:00:28 enrolling into my study,

1:00:30 so I can do kind of like better power stuff,

1:00:33 maybe with the exception of very rare diseases,

1:00:36 like,

1:00:36 so I have colleagues who work at the National Institutes of Health,

1:00:39 and they're working on things like

1:00:41 Very rare brain diseases,

1:00:43 like around the world,

1:00:44 there's like 47 people or something like that who have this.

1:00:47 So those are hard and they require coordination across the,

1:00:51 you know,

1:00:51 world among medical people to just have,

1:00:53 like,

1:00:54 essentially,

1:00:54 you have a worldwide study where everybody is enrolling on their

1:00:57 site and then you're sharing data and things like that.

1:01:00 But outside of that,

1:01:01 that's what you need.

1:01:02 So,

1:01:04 What,

1:01:04 what is the advantage of adaptive experiments?

1:01:07 I'm going to show you in a second.

1:01:09 They might make your experiment more efficient,

1:01:11 meaning that you can get to the answer faster with a smaller sample size,

1:01:15 but not always.

1:01:17 They can definitely minimize what the people

1:01:19 in machine learning world call regret,

1:01:22 meaning that

1:01:23 because I'm learning each day,

1:01:25 and then the next day I'm assigning

1:01:28 the treatment that's more likely to be succeeding to more people,

1:01:33 I'm actually assigning by definition,

1:01:35 less people to arms that I don't think are working very well.

1:01:39 So,

1:01:39 it minimizes regret.

1:01:40 It's a bit more ethical,

1:01:41 so to speak,

1:01:42 right?

1:01:43 I also,

1:01:44 because I'm allocating more people to the winner,

1:01:46 I'm generating more precise estimates for the winner.

1:01:50 And what I can also do is I can do this adaptation contextually,

1:01:53 meaning that

1:01:54 I can actually say,

1:01:56 not only I'm going to look at this batch at the,

1:01:59 you know,

1:01:59 every week,

1:02:01 but I'm going to look at this batch

1:02:02 based on whether people are adolescents or adults.

1:02:06 And I'm going to allow for the fact that the

1:02:08 program may be working very differently for those two groups.

1:02:11 And then I'm going to update

1:02:14 the assignment probabilities actually independently.

1:02:17 So,

1:02:18 I can have an experiment in 3 months.

1:02:21 That started with adults and adolescents and like treatment

1:02:24 A and treatment B all given equally to everybody.

1:02:27 And if you had a static AB

1:02:30 type AB test type of experiment,

1:02:32 RCT,

1:02:33 then

1:02:34 that 25% chance for each of those groups

1:02:37 would stay

1:02:39 throughout the entire,

1:02:40 like,

1:02:41 study period.

1:02:42 But now,

1:02:43 at the end of like 3 months in the adaptive experiment,

1:02:46 you could have arrived at the situation where

1:02:48 adolescence and treatment A,

1:02:51 we're only giving it to 2% of them.

1:02:54 But then like adults in treatment B,

1:02:57 70% of the allocations are going to there or something like that.

1:03:01 So you can have these kind of like very unbalanced things,

1:03:04 and that has implications,

1:03:05 of course.

1:03:06 So,

1:03:07 here's kind of like how to see it visually.

1:03:09 So this is a static RCT,

1:03:12 right?

1:03:12 So you say,

1:03:13 OK,

1:03:13 I'm going to allocate 25% to each of the four treatments,

1:03:17 and then that's kind of how many people I'm going to have in expectation.

1:03:21 And then I'm going to estimate at the end,

1:03:23 like,

1:03:24 these T1 that did this,

1:03:26 T2 did this,

1:03:27 this is the mean,

1:03:28 this is the confidence interval,

1:03:30 and then I'm going to say T1

1:03:31 is significantly different than T4 or so on and so forth.

1:03:36 So what's the problem with it?

1:03:38 Like,

1:03:38 like I said,

1:03:38 two things.

1:03:39 First,

1:03:40 throughout the intervention,

1:03:43 I assigned a lot of people to the stuff that was inferior,

1:03:47 so I regret that,

1:03:48 right?

1:03:49 The second thing is

1:03:50 the standard error on the winner and the standard error on the loser

1:03:54 is more or less the same because I allocated the same amount of

1:03:58 um people to each of those groups.

1:04:00 Whereas I would like this to be much more precise.

1:04:02 I'd like to be able to go to the minister and say,

1:04:05 this thing really worked.

1:04:06 It's very tight,

1:04:07 you know,

1:04:08 like,

1:04:08 I have a great estimate of this.

1:04:10 I don't have that.

1:04:11 So if I had an adaptive experiment that I had the same number of people,

1:04:14 you see,

1:04:15 there's like maybe 800 people or something like that,

1:04:18 what I would do

1:04:19 is I would actually have those people start like,

1:04:22 I'd start them equal,

1:04:23 but not assign all of them,

1:04:25 but just some of them.

1:04:27 Get my estimate,

1:04:28 right?

1:04:29 Now,

1:04:30 because I allocated many fewer people,

1:04:32 I have large confidence intervals here at the beginning,

1:04:35 at the end of the first batch.

1:04:37 But then what I do is like,

1:04:39 T1 looks like it's um

1:04:41 doing better,

1:04:42 so I'm going to increase the probability that people are assigned to T1

1:04:46 and decrease the probability that they're assigned to,

1:04:48 let's say,

1:04:49 T3 and T4,

1:04:51 right?

1:04:51 So,

1:04:51 I'm gonna do this using some statistical techniques

1:04:54 that are too complicated to go into here.

1:04:56 So then,

1:04:57 The

1:04:58 number of people assigned is now going to be like this.

1:05:01 Remember,

1:05:01 I'm still doing a randomized experiment.

1:05:04 These probabilities are still random.

1:05:06 Everybody has a chance to end up in either of these bins,

1:05:10 right?

1:05:10 Um,

1:05:11 nobody has a zero chance of anything like we discussed last time.

1:05:14 So this is still a proper randomized experiment.

1:05:16 It's just that I changed the probabilities,

1:05:19 right?

1:05:19 So now I have these estimates

1:05:21 and then I continue

1:05:23 doing this

1:05:24 and then by the end,

1:05:25 you know,

1:05:26 it's possible that now I have actually 400 people in T1,

1:05:29 350 and T2,

1:05:32 only 150 in like T3 and less than 100 in T4,

1:05:36 something like that.

1:05:37 And look now what's happened is that

1:05:39 T1,

1:05:39 which is the best,

1:05:41 I have really tight confidence intervals.

1:05:43 It's like I can tell that it's better than everything else,

1:05:47 whereas I have kind of like,

1:05:49 More uncertainty over here,

1:05:51 but that's the one that,

1:05:52 that I don't care so much about.

1:05:54 Now,

1:05:55 What we did,

1:05:57 we minimized regret to the best we could

1:06:00 because we assigned many more people during the experiment

1:06:03 to

1:06:04 the,

1:06:04 to the

1:06:06 best expected intervention,

1:06:08 and then we also have tighter confidence.

1:06:11 So,

1:06:11 how you're doing that essentially is that you're just saying,

1:06:14 well,

1:06:14 look,

1:06:14 you know,

1:06:15 I'm gonna start with a

1:06:16 kind of like a prior distribution that all of these things work the same.

1:06:20 So,

1:06:21 initially,

1:06:21 I'm going to assign them 1/3,

1:06:23 1/3,

1:06:23 1/3,

1:06:25 but then

1:06:26 afterwards,

1:06:27 I'm going to do something like I have this,

1:06:29 you know,

1:06:30 Thompson sampling method from like the 1930s,

1:06:33 um,

1:06:34 that's actually going to tell me that like,

1:06:37 There's 70% chance that

1:06:39 the arm two is going to win this race.

1:06:42 There's 25% chance that arm three is going to win,

1:06:45 5% chance arm one is going to win.

1:06:47 So,

1:06:48 I'm going to literally take those and those

1:06:50 are going to be my new assignment probabilities.

1:06:52 So,

1:06:52 then I'm going to assign people

1:06:54 next round,

1:06:55 70,

1:06:55 25,

1:06:56 05.

1:06:57 And then

1:06:58 at the end of that batch,

1:06:59 I'm going to combine those two rounds of data.

1:07:01 I'm going to recalculate those probabilities,

1:07:04 so on and so forth until I have a winner.

1:07:06 And you can have stopping rules about like,

1:07:08 when do you declare a winner,

1:07:10 so on and so forth.

1:07:11 Like maybe when you

1:07:12 reach a certain amount of confidence and

1:07:15 this and that.

1:07:16 Right.

1:07:16 You can personalize it,

1:07:18 like we said,

1:07:18 you can do it for like teenagers versus uh

1:07:22 older people,

1:07:23 adults,

1:07:23 etc.

1:07:24 Um,

1:07:26 But the caveat is that,

1:07:28 you know,

1:07:29 because

1:07:30 The way you assign treatments is now dependent on what happened before.

1:07:36 You lose the IID,

1:07:38 um,

1:07:38 you know,

1:07:38 the,

1:07:39 uh,

1:07:41 Independent draws,

1:07:43 uh,

1:07:43 individually independent draws,

1:07:45 um,

1:07:45 of the kind of like the,

1:07:47 a normal distribution.

1:07:48 So what you're gonna have is like,

1:07:50 you no longer have a,

1:07:51 going to have a normal distribution,

1:07:53 like of your error terms.

1:07:55 So that means that

1:07:56 you can't,

1:07:57 while you're doing the adaptive experiment,

1:07:59 you can't just take your data and treat it like you

1:08:02 have a regular RCT and then do the regular inferential analysis.

1:08:07 To do anything,

1:08:08 you have to do something more sophisticated.

1:08:10 And this is actually a very active area of research because

1:08:14 a lot of times these very smart people who are statisticians,

1:08:17 econometricians,

1:08:18 mathematicians,

1:08:19 computer scientists,

1:08:20 etc.

1:08:21 you know,

1:08:22 they have to do things by simulation because there's no closed form,

1:08:26 you know,

1:08:26 formulas or proofs and things like that for the kind of like,

1:08:29 let's say the,

1:08:31 the asymptotic,

1:08:32 um,

1:08:32 you know,

1:08:33 kind of like the

1:08:35 Validity of the kind of like the,

1:08:36 uh,

1:08:37 you know,

1:08:37 consistency and things like that,

1:08:39 unbiasedness,

1:08:40 all those things.

1:08:41 So this is very much an ongoing area of research,

1:08:44 but

1:08:45 The point here is that

1:08:47 there are actually opportunities

1:08:50 to do adaptive experiments,

1:08:51 and I'm doing one and other people have started doing them as well.

1:08:54 You know,

1:08:55 as people come to the hospital,

1:08:57 if you're doing it,

1:08:58 like a,

1:08:59 we have a tablet-based thing and then the tablet is doing all the randomization,

1:09:03 it's very easy to change.

1:09:05 Those randomized assignments,

1:09:07 even the nurses and doctors that are doing

1:09:10 the interventions,

1:09:11 they don't know because all that's happening is like

1:09:13 on the underlying distribution of like assignments is changing,

1:09:17 but we are actually

1:09:19 targeting better and better.

1:09:20 We're doing like personalized tailored.

1:09:24 Intervention assignment to people who are coming in.

1:09:27 So,

1:09:27 definitely people who are coming in

1:09:30 later on in the study are luckier than people who come in earlier in the study because

1:09:34 they're benefiting from all the things that we're learning,

1:09:37 and then they're just kind of,

1:09:38 like,

1:09:39 contributing to this knowledge generation,

1:09:41 uh,

1:09:41 while benefiting at the same time.

1:09:44 One thing to also

1:09:46 um

1:09:47 keep in mind is that

1:09:49 still,

1:09:50 even though this is more like this Beijing,

1:09:52 uh,

1:09:53 type of,

1:09:54 like work,

1:09:55 where,

1:09:56 you know,

1:09:56 you don't necessarily need T tests,

1:09:59 we're not in this,

1:10:00 in that world with

1:10:01 either academics or actually even policymakers,

1:10:04 like policymakers and,

1:10:05 you know,

1:10:05 people who make these decisions,

1:10:07 still gonna want to say,

1:10:09 hey,

1:10:09 compared to my status quo intervention at the schools,

1:10:12 like the way my schools are now.

1:10:14 Can you just show me that like your meals program

1:10:18 did significantly better,

1:10:20 right?

1:10:21 So to be able to do that,

1:10:23 you need

1:10:24 enough data on the status quo,

1:10:26 but in the adaptive experiment,

1:10:28 some algorithms can be very aggressive in eliminating,

1:10:32 if suppose that the status quo was losing very badly,

1:10:35 you can end up collecting very few data points on the status quo.

1:10:39 So,

1:10:39 you may need to actually anticipate that you're gonna need

1:10:43 A quote unquote control arm.

1:10:46 Right?

1:10:46 And you're going to

1:10:48 um

1:10:50 You're going to,

1:10:51 um,

1:10:52 collect enough data on the control arm

1:10:54 so that you can actually compare it reliably to the

1:10:57 T1 because T1 can have a very small confidence interval,

1:11:00 but if T4 has a very big one,

1:11:03 you still won't be able to confidently say T1 did better than T4.

1:11:07 So,

1:11:07 what we do is we have these kind of like,

1:11:10 floors,

1:11:10 so to speak.

1:11:11 So we say like,

1:11:12 we're going to allocate at least 10% of the

1:11:16 sample

1:11:17 to this group,

1:11:18 even if it's doing very badly because we need that.

1:11:21 Or what you can do is you can have a learning phase

1:11:24 where you do the adaptive experiment as aggressively as you want.

1:11:28 And then once you say,

1:11:29 I've learned that,

1:11:30 like,

1:11:31 these two ones are doing the best,

1:11:32 and then I have my status quo,

1:11:34 then you can go to a

1:11:36 evaluation phase where you actually now run a

1:11:40 static experiment quickly

1:11:42 to just say,

1:11:42 OK,

1:11:43 the winner versus the status quo.

1:11:45 Now I'm going to show that,

1:11:46 like,

1:11:46 with very high confidence that this one is doing better than the status quo.

1:11:50 You can do kind of either way.

1:11:53 Then the last 15 minutes,

1:11:54 I want to talk about a couple of things that are practices,

1:11:57 like important practices of analyzing

1:12:00 um randomized experiments.

1:12:03 So,

1:12:04 two things that come up over and over and over

1:12:06 again are kind of like baseline balance and attrition.

1:12:09 And I want to talk a little bit about each one of them.

1:12:12 And if we don't get to whatever I have left after that,

1:12:15 that's OK.

1:12:16 Um,

1:12:17 because these are kind of like the last really

1:12:19 most important things.

1:12:21 So,

1:12:21 on baseline balance,

1:12:23 um,

1:12:24 you know,

1:12:25 I think that

1:12:26 There are things that you can do at the design stage.

1:12:29 And one of the debates in

1:12:31 this between medical people and social science people has been that like,

1:12:35 there's a lot of medical folks who say,

1:12:37 look,

1:12:38 you shouldn't show a baseline balance table because you

1:12:40 have randomized in an RCT you have randomized already.

1:12:44 So,

1:12:45 any differences that you show me are going to be just coming out of that,

1:12:48 like,

1:12:49 that random chance.

1:12:50 So I don't really care,

1:12:52 like if there's differences or not.

1:12:54 And then there are others who say,

1:12:55 well,

1:12:56 actually,

1:12:57 hold on a minute,

1:12:58 that's,

1:12:58 that sounds,

1:12:59 that doesn't sound right because

1:13:01 if there's some really big difference on

1:13:03 a really important characteristic that's sermonistic,

1:13:07 that's still,

1:13:08 that's still troublesome.

1:13:09 Just the fact that I randomized doesn't mean

1:13:12 that like I shouldn't worry about it.

1:13:15 Um,

1:13:15 and then there's also the

1:13:18 Um

1:13:19 I'm going to say?

1:13:21 All right,

1:13:21 let me,

1:13:21 let,

1:13:22 let me continue.

1:13:23 So,

1:13:23 so I think

1:13:25 That's

1:13:26 There's ways actually you can avoid

1:13:29 having those massive differences by chance

1:13:32 by doing this stratification and block randomization.

1:13:35 So,

1:13:36 when I wrote this uh 2 weeks ago,

1:13:38 David McKenzie had just written on this,

1:13:40 like the perfect reference.

1:13:41 So I definitely encourage you to,

1:13:43 to read that.

1:13:44 It's now been 3 weeks.

1:13:46 But essentially,

1:13:46 his main takeaways are,

1:13:48 you want to do 2 stages.

1:13:51 The first stage,

1:13:52 when you're approaching your RCT.

1:13:55 is that

1:13:56 you want to have one or two key variables

1:13:59 on which you want to stratify your sample.

1:14:02 So,

1:14:02 for example,

1:14:02 the easiest things to think about could be like,

1:14:05 you have urban areas and you have rural areas,

1:14:07 they look very different.

1:14:08 So why not just do your experiment within them rather

1:14:11 than like doing unconditional experimentation across the whole country.

1:14:15 Or you can do male and female,

1:14:17 and you don't want to end up with,

1:14:18 like,

1:14:18 60% male and 40% female,

1:14:21 so you You can stratify on that and then have your like equal sizes of male and female,

1:14:25 and then you have,

1:14:26 you can have balance

1:14:27 inside those as well.

1:14:29 So,

1:14:29 you could do those kinds of things.

1:14:31 For example,

1:14:31 Guido Imbens,

1:14:32 who recently won the Nobel for the work he's done on,

1:14:35 partly on these things,

1:14:36 recommends that using cluster size

1:14:39 to define strata is important because often cluster size is

1:14:42 very deterministic of like a bunch of things,

1:14:45 right?

1:14:46 So then,

1:14:47 You have your strata now you're in urban areas

1:14:49 or rural areas or you're in large clusters,

1:14:52 mid-sized clusters,

1:14:53 and then

1:14:54 small sized clusters.

1:14:55 Within them now,

1:14:57 you have your units,

1:14:58 and what you want to do is you want to form,

1:15:00 let's say something like

1:15:01 quadruplets.

1:15:02 If,

1:15:03 if you have one treatment and one control,

1:15:05 you can form quadruplets.

1:15:07 And how do you form them?

1:15:09 The idea is essentially that

1:15:11 I want balance on things that really matter.

1:15:14 What are the things that really matter?

1:15:16 It's either my outcome.

1:15:18 So if I observe

1:15:20 the

1:15:20 baseline value of my outcome,

1:15:22 suppose that we're doing an education intervention and

1:15:24 I'm interested in test scores and learning,

1:15:27 and I have baseline test scores,

1:15:29 right?

1:15:30 Then I can use the baseline test scores.

1:15:33 To match people into,

1:15:35 like,

1:15:35 everybody who got a 4 out of 100.

1:15:38 Like,

1:15:38 as long as there's 4 of them,

1:15:40 I put them into that bin.

1:15:41 And then everybody who got a 5,

1:15:43 everybody who got a 6.

1:15:44 And then,

1:15:44 therefore,

1:15:46 now,

1:15:47 if I have my bin of people who got very bad grades,

1:15:50 and then within them,

1:15:52 I assigned 2 of them to treatment and 2 of them to control,

1:15:56 I know for a fact that I'm going to have balance on that variable.

1:16:00 It's impossible for me not to have balance on that variable.

1:16:03 Now,

1:16:04 if you have

1:16:05 not

1:16:06 one outcome indicator,

1:16:07 but a bunch of outcome indicators,

1:16:09 but they're closely correlated,

1:16:11 then you can make an index of them and then you can,

1:16:13 you can

1:16:14 do quadruplets like based on

1:16:17 being close,

1:16:17 you know,

1:16:18 nearest neighbors kind of thing

1:16:20 on the,

1:16:21 on the index.

1:16:23 If you don't observe the outcome at baseline,

1:16:27 then you should try to do it on the basis

1:16:29 of like baseline covariants that predict the outcome really,

1:16:32 really well.

1:16:33 So for example,

1:16:34 in stunting literature,

1:16:35 your age in months

1:16:37 is going to be extremely important,

1:16:39 so you really want to make sure that like,

1:16:42 The treatment group isn't 2 months older than the control group because

1:16:46 that's going to make a massive difference in their kind of like,

1:16:49 uh,

1:16:49 Z scores,

1:16:50 right?

1:16:51 So what you want to do is you want to take the people who are like aged 13 months,

1:16:56 and then,

1:16:57 you know,

1:16:57 like have quadruplets of them,

1:16:59 14 months,

1:16:59 15 months,

1:17:00 etc.

1:17:01 and then you want to then

1:17:02 randomized

1:17:04 within that.

1:17:04 So,

1:17:05 think of it as you have like

1:17:06 your strata,

1:17:07 you have two strata,

1:17:09 strata,

1:17:09 urban and rural.

1:17:11 Within your urban and rural,

1:17:12 you have a bunch of quadruplets,

1:17:15 right?

1:17:15 And then within those quadruplets,

1:17:17 now you're running a lottery and saying like 2

1:17:19 of these guys are going to be treatment,

1:17:21 2 of these guys are going to be in control.

1:17:23 So that's the kind of like the main takeaway.

1:17:26 And if you do this kind of thing,

1:17:29 And I think that I remember the thing that I was going to say in that debate,

1:17:32 people say,

1:17:33 well,

1:17:33 look,

1:17:33 you know,

1:17:35 If you do that,

1:17:36 then you don't really need to report the baseline balance,

1:17:39 like,

1:17:39 at least not as table one of a,

1:17:41 like a thing,

1:17:41 you can just toss it into the appendix.

1:17:43 So,

1:17:43 I think if you do this kind of thing and you're prepared,

1:17:47 my recommendation is that it's totally OK to

1:17:49 report the baseline balance in the appendix table,

1:17:52 and that You can also describe how you did this,

1:17:54 you know,

1:17:55 randomization,

1:17:56 essentially ensuring that on important key variables,

1:18:00 you will by definition have balance.

1:18:02 Then

1:18:03 it's not that important to show the baseline balance

1:18:05 because all you're doing is confirming you've done that.

1:18:08 You can just stick it into the appendix state.

1:18:10 Right?

1:18:11 The second thing people say is that,

1:18:12 well,

1:18:13 actually,

1:18:13 maybe you didn't do that,

1:18:15 but maybe

1:18:16 you had a plan for re-randomizing.

1:18:18 So you thought about this thing that can happen by chance,

1:18:21 and you said that you're going to minimize it.

1:18:23 So,

1:18:23 what you can do is essentially,

1:18:25 suppose that you have K variant,

1:18:27 K

1:18:28 characteristics at baseline,

1:18:29 you know,

1:18:30 age,

1:18:30 sex,

1:18:31 like test score,

1:18:32 da da da,

1:18:32 it's an urban,

1:18:33 rural,

1:18:34 the household size,

1:18:35 this and that.

1:18:36 And what you're saying is that,

1:18:39 let's say,

1:18:40 I don't want to see

1:18:42 any of the,

1:18:43 the

1:18:44 P,

1:18:45 the differences between

1:18:47 the treatment and control or any of those variables

1:18:50 to be bigger than a certain amount or the P value to

1:18:52 be like higher than a certain amount or something like that.

1:18:56 So,

1:18:56 you can pre-specify saying,

1:18:58 I'm going to randomize this like 100 times,

1:19:02 and then I'm going to look through,

1:19:04 and then I'm going to

1:19:06 Take the one that minimizes the maximum of this like T value,

1:19:10 right?

1:19:11 Like this difference,

1:19:12 like standardized by the um

1:19:14 by the standard errors,

1:19:16 right?

1:19:16 Or you can say,

1:19:18 I'm going to re-randomize until all T stats are below a certain threshold.

1:19:24 So suppose that you only cared about two variables,

1:19:26 like the test score and age,

1:19:28 right?

1:19:29 And you can say,

1:19:30 I'm going to re-randomize until the T tests for both of those are under 1.

1:19:35 Right?

1:19:36 Something like that.

1:19:37 So,

1:19:37 you have to be careful there,

1:19:38 like they,

1:19:38 they should be feasible.

1:19:40 If you have many variables and if you put a really

1:19:43 like

1:19:43 demanding threshold like T equals to 0.05 or something like that,

1:19:48 then

1:19:49 you're never going to get there.

1:19:51 So,

1:19:51 you want to be realistic,

1:19:52 but

1:19:53 re-randomizing is actually,

1:19:55 like,

1:19:55 in many cases,

1:19:56 if you pre-specify it and do it correctly,

1:19:59 it's equivalent to

1:20:01 You know,

1:20:01 like,

1:20:02 unconditionally randomizing

1:20:04 to get equal numbers of people in each group.

1:20:07 So,

1:20:07 you,

1:20:08 If you do this,

1:20:09 you should pre-specify for randomization inference,

1:20:12 which you guys covered with Florence,

1:20:14 and don't search over randomizations for best value,

1:20:17 meaning that

1:20:18 you didn't pre-specify,

1:20:19 you do it a bunch of times,

1:20:21 you're still not happy,

1:20:22 and then you say,

1:20:23 oh,

1:20:23 let me do one more time,

1:20:24 or now,

1:20:25 this one is OK.

1:20:27 Those are not OK because there's then very little randomness left,

1:20:30 like you're imposing your will on the kind of like the distribution.

1:20:35 But otherwise,

1:20:36 if you actually

1:20:37 do either of these two options,

1:20:39 you're fine.

1:20:40 And again,

1:20:41 if you do this,

1:20:42 you will have balance on the pre-specified variables according to your criteria,

1:20:47 right?

1:20:47 Then again,

1:20:48 in my mind,

1:20:49 then it's OK to put it in table,

1:20:51 your appendix table one,

1:20:53 right?

1:20:54 But if you haven't done either of these things,

1:20:56 let's say you inherited the

1:20:59 You know,

1:21:00 RCT or like something like that,

1:21:02 or,

1:21:02 you know,

1:21:03 just for whatever reason,

1:21:04 you didn't do it,

1:21:05 or you did do some of these things,

1:21:07 but like you did unconditional randomization,

1:21:10 but

1:21:11 you actually

1:21:12 don't know whether your RAs did a good job or maybe there was a public lottery,

1:21:17 and you don't know if the public lottery

1:21:19 went according to plan or something like that.

1:21:22 Then this is really a defensive thing.

1:21:24 You're saying,

1:21:24 look,

1:21:25 you know,

1:21:27 We randomized,

1:21:27 but we didn't do all these safeguards,

1:21:30 so now we're going to show you that,

1:21:32 like,

1:21:33 it's actually OK.

1:21:34 Like,

1:21:34 these,

1:21:34 the treatment and control look very similar to each other.

1:21:38 And like I said again,

1:21:39 like I was saying before,

1:21:41 if your study is well-powered,

1:21:43 then you're going to be more likely

1:21:45 to reject differences in covariant values between

1:21:48 arms,

1:21:49 and if it's underpowered,

1:21:51 then you're not going to be able to reject anything.

1:21:54 So,

1:21:55 You don't want

1:21:57 a badly powered study

1:21:59 to be helping you say,

1:22:01 oh,

1:22:02 I have balance because

1:22:03 there's these massive differences between treatment and control,

1:22:06 it's just that they're not statistically significant.

1:22:08 So you have to report

1:22:10 your minimum detectable effects and you,

1:22:12 your,

1:22:12 your

1:22:13 power,

1:22:14 so to speak.

1:22:15 And then you also have to have an

1:22:17 idea of like meaningful differences between these covariants.

1:22:20 Like if I see that like there's 10%

1:22:22 points more people with university education in Malawi,

1:22:26 but it's not significant,

1:22:27 I'm not going to sit there and say like,

1:22:29 oh,

1:22:29 treatment and control look exactly the same when like 10% points

1:22:33 difference in education,

1:22:35 the

1:22:35 university education is a huge deal in a place like Malawi.

1:22:40 The second thing you want to do,

1:22:41 which I don't see often enough,

1:22:42 is that

1:22:43 Instead of just looking pair-wise,

1:22:45 like this variable or is it different,

1:22:47 this variable,

1:22:48 is it different or counting them,

1:22:49 whatever,

1:22:50 you should be doing joint tests of orthogonality,

1:22:52 which means that

1:22:53 is this vector of variables for treatment

1:22:57 different than the vector of variables for control,

1:23:00 and then you want really high P values in these tests.

1:23:03 So like in the links,

1:23:04 like I have some examples here,

1:23:06 I'm very happy.

1:23:07 When I see that,

1:23:08 like,

1:23:08 the treatment and control,

1:23:10 the,

1:23:10 the difference between those vectors is like a F test,

1:23:13 like a P value of,

1:23:14 like,

1:23:14 0.862 or 0.698 or something like that.

1:23:18 And I do get actually worried when it's like 0.11 something,

1:23:23 because then I can see that actually,

1:23:24 some of the variables are looking different.

1:23:26 It's just that it hasn't quite reached

1:23:28 traditional significance level.

1:23:31 The last thing we do before we go is kind of attrition.

1:23:35 I think

1:23:36 the thing that I see with attrition that bothers me most is that

1:23:40 people generally say,

1:23:42 well,

1:23:42 OK.

1:23:44 Attrition is really kind of like

1:23:46 balanced in levels,

1:23:47 so I'm not too worried.

1:23:49 And that's essentially to me is a big cop-out

1:23:52 because you could have had like to be facetious,

1:23:55 let's say

1:23:56 you have 10% attrition in both the treatment and control group,

1:23:59 but in the treatment group,

1:24:01 all the right-handed people

1:24:03 left,

1:24:03 and then in the control group,

1:24:05 all of the 10% that left are left-handed people.

1:24:08 Now,

1:24:09 they're very different than Each other

1:24:11 and so that's going to affect things and so you can't just say,

1:24:14 OK,

1:24:15 I'm going to check levels,

1:24:17 you have to

1:24:17 check for differential attrition between treatment and control

1:24:20 in both levels as well as characteristics,

1:24:24 right?

1:24:24 If you can show,

1:24:25 show evidence that like you're OK in both of the,

1:24:28 I mean,

1:24:28 you're not gonna ever have a study that has zero attrition.

1:24:31 I mean,

1:24:31 I have one One,

1:24:32 but that's mainly because it's the administrative data.

1:24:35 So like,

1:24:36 people do their like counseling session,

1:24:38 I see them,

1:24:39 so nobody drops out.

1:24:40 But as soon as I have to follow them up,

1:24:42 like to ask them a question,

1:24:44 I'm gonna lose some of them,

1:24:45 right?

1:24:45 So you're always going to have attrition,

1:24:47 so you're always going to have to do these things.

1:24:50 If you're lucky,

1:24:51 your attrition will be minimal.

1:24:54 And then it will be balanced in levels and characteristics,

1:24:57 so that you can essentially proceed with like showing the data for like the 96%

1:25:02 of the people that you were able to follow up or something like that.

1:25:05 But in other cases,

1:25:06 you're gonna have to use,

1:25:08 like other,

1:25:09 other types of things.

1:25:11 And one of the

1:25:12 things that I,

1:25:13 uh,

1:25:14 tell people is that,

1:25:15 you know,

1:25:16 the bounds analysis.

1:25:17 I think it is really important and it needs to be much,

1:25:20 much more prominent in many studies

1:25:22 than it is.

1:25:23 Often what I see is like people say,

1:25:25 oh,

1:25:26 attrition,

1:25:26 yeah,

1:25:26 we have some attrition,

1:25:28 sometimes it's like

1:25:28 20% or something like that.

1:25:30 People say,

1:25:31 oh,

1:25:31 that's OK,

1:25:32 20% is pretty comparable to like other studies like ours,

1:25:36 and we see no differences between treatment and control,

1:25:39 so like we're just gonna go ahead.

1:25:41 I think in my mind,

1:25:42 that's more and more unacceptable,

1:25:44 and I think you want to be doing some sort of adjustment,

1:25:47 and that usually my preferred route is these

1:25:51 Kling and Liebman bounds,

1:25:52 and you can see these uh linked papers that I have here.

1:25:56 Um,

1:25:57 and what that does essentially is the following.

1:26:00 Suppose that I have like

1:26:01 5% attrition in both my cash transfer group and my control group,

1:26:06 right?

1:26:07 And then suppose that like the,

1:26:09 it's the,

1:26:09 you know,

1:26:10 something about,

1:26:11 let's say consumption

1:26:12 um that I'm interested in,

1:26:14 right?

1:26:15 So what I want,

1:26:16 so,

1:26:16 and suppose that when I just compared them without any adjustments,

1:26:19 I see that there's an effect,

1:26:21 like there's a large effect.

1:26:22 So now what I wanna say is like,

1:26:25 suppose that the 5% in the treatment group that dropped out

1:26:30 had 10% less

1:26:33 um

1:26:34 Consumption than the mean.

1:26:37 And then the control group

1:26:39 who dropped out,

1:26:40 had 10% more consumption than the meat.

1:26:43 Like,

1:26:43 what would have happened?

1:26:44 Usually,

1:26:45 we do this by standardizing,

1:26:46 so we say like,

1:26:47 what if they,

1:26:48 like,

1:26:48 treatment group had 0.1 standard deviation,

1:26:51 less consumption,

1:26:52 and the control group had 0.1% more.

1:26:54 I recalculate,

1:26:55 like,

1:26:56 I fill in those missing values for the treatment and control.

1:26:59 I recalculate the effect for all the 100%.

1:27:03 Now,

1:27:03 my effect is going to be smaller because what I did is I said,

1:27:06 I stacked the deck against myself,

1:27:09 right?

1:27:09 I said that treatment group attrition was selective in a

1:27:13 way that benefits treatment and control group selection was,

1:27:17 uh,

1:27:18 that,

1:27:19 that actually harmed their like estimates,

1:27:21 so now they got closer to each other.

1:27:23 But I do it,

1:27:25 and then there's still an effect and it's still significant.

1:27:27 Then I say,

1:27:28 OK,

1:27:29 uh,

1:27:29 let me do it again with 0.25 standard deviations.

1:27:32 Like,

1:27:32 I do it again and it's still,

1:27:34 it's smaller,

1:27:35 but it's still significant effect larger.

1:27:36 And then finally,

1:27:38 I do 0.5,

1:27:39 and that's where I lose like they become equal

1:27:42 and then there's the difference is no longer significant.

1:27:45 Now,

1:27:45 what I've done is I've said,

1:27:46 look,

1:27:47 my study is robust.

1:27:49 Even if the

1:27:51 differential attrition

1:27:53 was selected to such an extent that like even if

1:27:56 there was a

1:27:57 0.8 0.8 standard deviation difference between the consumption values

1:28:03 of the people that attrited from my study,

1:28:05 like 0.4 up for the treatment and 0.0.4% down for the control.

1:28:10 I still get

1:28:12 significant effect that cash transfers are working very well here,

1:28:16 right?

1:28:16 So then,

1:28:18 It's a matter of

1:28:20 debating with people,

1:28:22 like,

1:28:22 is it really realistic to think that attrition is so selected that,

1:28:27 like,

1:28:27 I would have these massive differences in the pattern of it,

1:28:30 like,

1:28:31 and then,

1:28:32 yes,

1:28:32 at this point now,

1:28:33 what you're doing is you're doing a little bit of a discussion

1:28:36 with your referees or with your audience and with the policymakers,

1:28:40 you know,

1:28:40 about something like

1:28:42 that you do with non-experimental studies,

1:28:44 like,

1:28:44 is the different,

1:28:45 difference in different

1:28:46 valid because the parallel trends assumption is there or is the IV estimates valid,

1:28:52 etc.

1:28:52 So you're in that world,

1:28:54 but at least you put some bounds on that world and you say,

1:28:57 you can actually say that I have differential attrition,

1:29:00 but I still have

1:29:02 A significant effect.

1:29:03 And only after you've gone to this crazy difference between these two groups,

1:29:07 which I don't think is plausible,

1:29:10 then I have like no effect.

1:29:11 So,

1:29:12 at least that's something.

1:29:13 So I think

1:29:14 that's what I suggest people do with attrition stuff.

1:29:17 And then I had one more thing on covariant adjustments,

1:29:20 but once again,

1:29:21 we've run out of time,

1:29:22 so I'm going to stop there,

1:29:23 and this time I'm not going to offer.

1:29:26 Um,

1:29:26 to do a 3rd lecture on this because we have actually very little.

1:29:30 This is the only thing that's left.

1:29:32 But,

1:29:32 um,

1:29:33 I've written so many blog posts on this,

1:29:35 and then actually there are much better ones that I linked to here

1:29:39 on this topic that,

1:29:40 like,

1:29:41 essentially,

1:29:42 Randomized experiments,

1:29:44 the use of covariates is,

1:29:46 is important.

1:29:47 I think it's very valuable,

1:29:49 but you wanna do it in a way that's transparent,

1:29:52 that's pre-specified,

1:29:53 and that's correctly specified,

1:29:56 and all the information that you need is here.

1:29:59 You don't need me to,

1:30:00 like,

1:30:00 talk about it.

1:30:01 Um,

1:30:02 but with that,

1:30:03 I think I'll stop here.

1:30:05 And I know that this is your last lecture,

1:30:07 and I know that you guys are

1:30:09 not going to be McNamara Fellows.

1:30:12 Uh,

1:30:12 soon.

1:30:13 I wish you the best.

1:30:14 I hope it was a good year for all of you.

1:30:16 We're sad that we couldn't have you here,

1:30:18 but,

1:30:19 uh,

1:30:19 I wish you all the best,

1:30:20 and if you have any questions,

1:30:21 feel free to email me.

1:30:24 Thank you so much,

1:30:25 Mark.

1:30:26 I really appreciate it.

showAllTimestamps
no
transcript
Well, good morning, everyone, and welcome to the 5th and final lecture in the Robert S. McNamara Fellowship Program lecture series. So as a quick background, the World Bank's Robert S. McNamara Fellowships Program, known as RSMFP matches aspiring development economics researchers from developing countries with World Bank research economists. Fellows have unique opportunities to participate in policy, in rigorous policy relevant research in the World Bank's Development Economics vice presidency. The RSMFP lecture series is a flagship activity of the of the RSMFP. Um, over the course of 5 lectures, fellows and other attendees will learn about the latest developments in empirical research. The lecture series focuses on topics that, although not always taught in graduate courses, are fundamental for development researchers. Today, we're delighted to welcome back Brooke Osler to give the second half of his lecture on why randomize, what to randomize, and how, combining basic principles of RCTs with practical aspects of field experiments. Brooke Osler is a lead economist and research manager of the Poverty and Inequality Research Program in the Development Research Group. He received his BSC in mathematics from Bosphorus University in 1991 and his PhD in Economics from Cornell University in 2001. After working on poverty and inequality measurement, poverty mapping, and the 2006 World Development Report on equity and Development earlier, he decided to combine his interest in cash transfer programs and HIV risks facing young women. In Africa by designing a field experiment in Malawi. He has since been involved in a number of cluster randomized field experiments. He's a co-founder of and regular contributor to the Development Impact blog. So, without further ado, let me hand it over to Burke. Um, thank you so much for joining us again and looking forward to the rest of this lecture. All right, perfect. Thank you, Maria. Uh, and thank you, Napur, for, uh, again, organizing this. Um, yeah, like, uh, you were all here, I think last time around, and, you know, I didn't get through a lot of it, so I volunteered, but I wasn't sure there'd be some demand. But, um, the time folks tell me that there was some, at least somewhat enthusiastic response. So here we are again, and this is where I remembered we were. stuck, I think. Um, so I'm just gonna continue. Um, you know, the first part was a bit more kind of like conceptual and kind of like about counterfactuals and things like that. This one will be a little bit more practical, uh, for when you're doing, uh, RCTs, when you're planning RCTs, what kinds of things you should be thinking about. So it can be a bit more informal, like, I'm not going to go through too much detail on kind of like, There's technical things, but I'm not going to go through the kind of, like, why those equations work and things like that. But also feel free to, you know, interrupt. You guys are generally, uh, I attended a few of these. You guys are a quiet bunch. But so, this is the last one. So, um, feel free to just kind of ask questions, that there's really no stupid questions, and then if I don't know the answer, I'll just say, I don't know, um, because some of this stuff is, you know, cutting edge, and then we're just all learning. Um, together. So, I'm gonna just jump back in, um, with the kind of one of the common critiques of randomized experiments and, you know, something that policymakers regularly face. is, you know, what evidence to use for the, kind of like, to inform the design of their programs. You know, I'm just coming from an event now in Mozambique. Let's say in Mozambique, they didn't really have an RCT on this, like, adolescent girl program that they're thinking about, but there's an RCT from Malawi, which is a neighboring country. Um, but then they have really good, let's say, demographic health survey data, and they can run a bunch of like regressions, maybe even do like diff diff or something like that, etc. The question is, You know, should we use our own data, but we're not going to have a randomized experiment, or should we have the randomized experiment, but it's not our own data, you know. And people do disagree about this, uh, the answer to this question, and probably there isn't a correct answer. Um, and people have been disagreeing, as you can see from these quotes for like 60 years or more. Um, and some people think that's really internal validity, which I'll define in a minute, like, you know, RCTs are kind of like should be prioritized. Um, and then external validity, whether some evidence applies to some other context should be just secondary. Um, and then there are other people who say, look, if I'm trying to make policy, like it makes no sense to value one type of validity above the other, because there may literally be nothing with that type of validity for me. So, what am I supposed to do? Um, so, we're essentially thinking here that, you know, um, think of an example like this in, in the United States, there's a famous, uh, preschool program called Head Start. It's all over the country. But in the 60s, some people did a randomized experiment in like a small school district. It's a famous study called Perry Preschool Project. And, you know, that one showed that, like, the um graduation rates for kids who are in the um Head Start program are significantly higher than people who, who are not, right? But then there's a, oops, sorry, there's another study that actually shows that, like, um, You look at all the Head Start programs around the country, and you look at people in Head Start versus you look at people not in Head Start, and the graduation rates of Head Start are 13% points lower, right? So, Yeah, in this case, I think the answer is a little bit easier in the sense that we can easily think of the selection issues. Head Start program is a program for underprivileged kids. So, of course, their graduation rates are going to be lower, potentially, even with the program than people who have better means and then maybe going to better public schools or private schools, whatever, etc. So, here, you know, you have this. Kind of like exaggerated difference where the effects are going the other way, and then theoretically speaking, we have an expectation that like, you know, there's no reason to think that this preschool program would have stark negative effects. But you can have cases where both effects are positive, but they're, one of them is small and one of them is big, and that makes a difference for cost effectiveness and things like that, then you would be more scratching your head saying, well, which one of these we should trust more. Right. And in that, in that sense, I think, um, you know, thinking about Methods to just kind of extrapolate results from one place to the other. Um, there are interesting work that's been done in places like the United States where there's something like, uh, I don't know, it was like electricity utilities or something like that. There's a bunch of different independent ones. And then, you know, Colorado is very different than Montana, which is very different than Texas, which is different than New York. And you have these evaluate randomized evaluations in those places. And then maybe that gives you enough of a variation to say, well, what predicts success? So you can look at kind of like these things, and then you can go to a 5th or a 6th uh state that I didn't um mention here, and then try to just see if the convex combination of those estimates creates like you, uh, you know, Arizona or something like that, where you can say, Well, based on what I know from all these other places, I think Arizona is going to have something like this. Or you might say, well, really, like, not only we don't think we're gonna be able to do this, but also running that kind of an experiment in our own, um, backyard is not that costly. So we're going to do that ourselves and then decide. So, there's really, in my mind, no correct answer here. I do think that, um, People like Lance Pritchett and Angus Deaton and others who say, you look, you shouldn't really prioritize um RCTs. They, they do have a point. Uh, I often disagree with them about whether we should prioritize our cities or not, but they do have a point in the sense that, yes, ideal randomized experiments may take priority, but hardly ever any RCT is ideal. And, um, You know, sometimes the confidence intervals may be too large and sometimes, uh, you know, our, our, our confidence in other aspects because of things like attrition and balance or like compliance and things like that that we're gonna discuss later, um, may be, may be low. So, in those cases, it is possible that actually, despite the fact that you may have a biased estimate from a Um, from a, um, let's say, a non-experimental study. Um, it may be closer to the truth than the RCT from another place with a big confidence interval. So, and there's really no way to know without doing it yourself, so you have to do just some extra legwork. Um, but other issues, of course, there's the ethical issues that come with RCTs. I'm going to touch on them a little bit later on, but the biggest thing for me is the concept of equipoise, which many people do not hear about the word until you actually do RCTs. I didn't know the word, like, before, um, especially as a non-native English speaker. It essentially means that, like, do I have enough uncertainty about the answer to this question? That I should be going ahead and randomizing people, because it is really, if you think about it, it is really unethical, you know, just to, let's say, just to have an RCT so that you can have ironclad results for a study that you can publish well in like a journal or something like that. On a question that we're pretty sure of the answer, and we're pretty sure of the kind of like, let's say the cost effectiveness and things like that, then you should just be targeting the program to people who need it rather than just randomly allocating it. But often, you know, economists, um, can find something about which there isn't any equipoise. So, for example, we do know Um, that this program works, but we want to know whether it works better than just giving people cash, let's say, or something like that. So now, what I want to do is I want to have a horse race, and I really am saying that I have no idea about the relative cost effectiveness of this program to the, to its alternative. So, that's the evidence I want to ascertain. I know that this program is going to work. It's just that I don't know whether it's going to do significantly better than something else that I could have done. Um Then there are things when you do RCTs type of studies that come up, and two of them that I like the most, and they are really big dangers to kind of like internal validity, which is something we said we, we value, the RCT proponents value. And one of them is Hawthorne effects, which is essentially the idea that, like, just because someone is watching me, um, I behave differently. And my favorite example on this, and I don't know if we covered this last time or not, is the, is this idea that, you know, that, you know, you can think about what's the effect of soap, using soap on kind of like, uh, you know, health outcomes or diarrhea or something like that. And, um, I have colleagues at Stanford who work on sanitation issues. They were worried about the self-reporting issue. So they essentially put, like, kind of like these observant people, like, in the community, to just kind of, like, be observing things. And then somebody said, well, actually, because they're observing, people are just going to be washing their hands when they're around and not going to be doing it when they're not around. So, what can we do? And what they did is they literally put, like, motion sensors inside the soaps, and they distributed soaps to everybody. And you can guess, like, where this is going, when you, the, you know, there's 3 ways to measure. You ask people, do you wash your hands? And let's say 65% say yes, that they wash it after going to the bathroom and before meals. And then, when you actually look at what the people observed, like, the enumerators, that goes down to, like, 30% or something like that. And then when you go to the motion sensor, like, how many times did the soap move during the day, that goes down to like 10, 15%, right? So, it's not only the, there, there's the issue of importance of measurement here that's separate, but the bigger thing is actually people are behaving differently because they're in an experiment and somebody's watching them. A different version of that is actually John Henry effects, and this is not about being watched, but this is about the fact that there is an experiment going on, and people have incentives about the outcome of that experiment. So, think of a, let's say a school management or teacher incentive interventions, and then you kind of like assign some schools or some classrooms into a certain regime and some to the, to another. Now, people in the treatment group, if they want the program to succeed, they might work harder. And people in the control group, let's say if they don't want it to succeed, they might also work harder to just show that the treatment and the control difference isn't that different, and vice versa. If people want it to fail, they may just try to sabotage it and so on. So, here, what's happening is because you can't really do a placebo in social sciences, you can't just, you know, people are going to know whether they're in the treatment or the control, um, Then, then that's going to, that's going to matter. Um, and there's even a version of this that's actually very interesting, and it was very innovative work, where You know, you have people who are in the vaccine group or in the placebo group. But depending on how they Um, What they think, what they got. Let's say that you expect that you're going to have some side effects when you get the Moderna vaccine, because you hear from a lot of people that Moderna does, like, side effects X, Y, Z. Right? So, you're going to think, OK, I'm getting X, I'm getting Y, I'm getting Z, so I must be vaccinated, so I can take my mask off and I can go into restaurants. And if the outcome is determined both by the intervention and your behavior, there's an interaction between what you think your treatment status is and what your behavior is. So, that's going to change things. So, the way to deal with that is actually, you can randomize. People and then tell them the likelihood that they're in a group. So that if people perfectly know that they are one group, and then, you know, they behave a certain way versus they perfectly know they're another group, you can actually tease out the components where the effect is due to only the Vaccine because these people think. They don't have the vaccine, but they're behave, so they're behaving accordingly, and then there's these other people. So you can, you can do that kind of thing. In practice, it becomes difficult, but these are the kind of theoretical things that you have to think about. But nonetheless, for you, if you're thinking, well, OK, I'm going to do this, like, pay some teachers for better performance, and then there are going to be other teachers who know I'm doing this, but they're in the control group, you have to really think about how these teachers are reacting. Just to the fact of the RCT rather than this is now the new world, and this is in which they're going to have these incentive programs and etc. So you have to think carefully about those kinds of things and whether they're going to affect your, uh, affect your, um, findings. Right, so now, let's talk a little bit about what types of randomized experiments you might do. So, one of the things that we can think about is like, the first thing that we think about is access, right? So, There is something that like I can do where I can essentially control access to a program. Like, it's a new program, it doesn't exist anywhere. There's nothing really like it around, you know, we can think of it as like new mRNA vaccine or something like that. And then what I'm going to do is I'm going to randomly assign some people to it and some people to a placebo or something. I can do that, but I can do it. Around a certain cutoff of, let's say, vulnerability or something like that. So you can think of this as Um, you know, elderly people are more vulnerable, people with certain conditions are more vulnerable. So, I'm going to only do it among those people, and then I'm going to leave everybody below 65 and no conditions out. Right. The second thing you can do is what are called phase-in designs, and there are various um versions of these that I'm going to show you in a second. Um, but you could essentially provide access to the program in phases, right? So, this district comes in first, this other district comes in later. This helps when You know, programs are new, and then there isn't enough administrative capacity to do everywhere all at once. And there may also be some learning involved. So you do just start slowly and then, uh, scale up, um, faster. So you could do those kinds of things. And while you're doing that, you can evaluate the people who are phased in versus not. If you can randomize those rather than selectively decide where you're going to start, um, you can do that. And then there's the encouragement types of designs. Um, these are cases where I can't actually control the access to the treatment. So, for example, um, you know, I'm going to give you an example later on. Like, we have a blog, uh, and I want to know the effect of the blog on people's knowledge on impact evaluations or methods or something like that. The blog is out there, anybody can read it. So, how do I, um, How do I, um, evaluate the effect of this blog? Well, then I can say, well, I know that, like, actually, even though this blog is out there, not everybody reads it. Um, I, in fact, I know that, like, from my baseline surveys, only 10% of the relevant population of graduate students that I care about, they are reading this blog. So, what I'm going to do is I'm going to send an email encouraging them. And hope that like that will increase the percentage that's reading. So now if I randomize the encouragement. Then I have a group that's like reading, you know, 10% of them are reading it. Then I have another group that 30% of them are reading it. Now, that 20% point difference I can use to estimate something. It's not going to be the same thing that I could estimate with uh just randomly controlling access, but it's going to be still something of potential interest. And if that's something of potential interest that's relevant to the policymakers or academics, then you can do that. Right? So, we can think of some examples here. So, for example, one thing is, you know, one reason or an opportunity to randomize is you actually want to address a problem, but you don't actually know exactly how to do it. So, you have, let's say, 23 Options and you want to try them. So, I think you have your target population, and then among them, you assign them to these different alternatives to see which one might work best. And you might do that aesthetically or adaptively. I'm going to talk about adaptive experiments a little bit later on. Um, it could be that you're introducing a new service. So in this case, there's no kind of, like, uncertainty about what you want to do. But you just want to know whether this new service, like, will people take it up, will it be effective, etc. So you might try to just kind of like, um, you know, invite some of your customers to, to take this up or not. Um, you can do new people. So, for example, Oregon, um, had some more money to add people to its Medicaid program. So suppose that The program was originally targeted to people at the poverty line, and then now they decided that they can actually increase it to 150% of the poverty line. And so, but there isn't enough money to cover like all of those people initially. So what you can do is you can actually randomly in that band of 100 to 150%. You can randomly assign the program to some and not the others. In fact, Oregon, literally the public lottery among the eligible people, and then you can wait a few years and then just, you know, like if you have an RCT you can show the effects and then that can inform the policy on like, how high to go from the poverty line and how many people to cover if it cost effective, etc. Right? You can do oversubscription, uh, things. So often we have things that we want to put out there and then, then so many people actually want them. So, if that's the case, then what, and we can't, we can't cover all of that demand, we can say, well, look, the most ethical way to kind of like distribute this is by lottery. These people are all presumably equally eligible. So, we are oversubscribed for this, like, offer. Um, so we're going to do a lottery and then sign up some people, and then, because now the selection issue is handled, right? There are people who said, I am interested in this program. Like, let's say this was job training or something like that. You don't want to compare people who applied for job training, and then you randomly selected to some people who didn't apply for job training because they're very different people. But now you have a, with the subscription, you have a bunch of people who all self-selected into saying, I need job training, but you still can't cover all of them. So when you randomize, you have a really comparable group of people that are actually relevant for the program because they all kind of after this program. The flip side of it that I hadn't thought about before is that sometimes there's things like unders subscription. So for example, um, think about, let's say, Ukraine or something like that right now, you know, they actually literally sub, you know, said everybody who is between the ages of 16 or 18 and 50 has to fight right now. Um, but when it was happening in, you know, During the Vietnam War, the US didn't need that many people, so they wouldn't have said, like, OK, we're having universal, you know, like a sub conscription. But then on the other hand, they, there wasn't still enough people putting up their hands saying, like, I want to go fight and put my life in danger. So, they still needed to do a lottery to actually conscript people into the army. And they did that on the basis of people's birth dates, uh, and things like that. And then if you were unlucky, um, then you had to go fight a war that you may or may not believe in. So, undersubscription is actually also an opportunity for a lottery. You can have rotations, like think of Roscas and things like that. Um, communities or individuals can take turns to be, uh, the treatment, uh, unit, so to speak, while the others are controlled. And then you can have admission cutoffs and admissions and phases and things like that. So, let's give a couple of examples here, but before I do that, like a word about ethics. So just because you can randomly assign people to intervention, it doesn't mean we should. We talked about equipoise, so, you should have a question about which you have a lot of uncertainty, and the question is really important. Um, from a policy perspective, like that's one barrier. The second one is beneficence, which is, this is about the Belmont principles of like ethics that we all kind of subscribe to. We take courses before we can do studies, uh, with human subjects or animal subjects for that matter. And beneficence essentially means that you're trying to maximize the Benefits to the people in your study and minimize the harms. And one of the biggest, um, risks of harm is kind of like privacy concerns. And so, we, we do a lot in our, you know, like a study designs on both kind of like how to do informed consent and things like that, etc. how to deal with vulnerable populations such as children or prisoners or disabled people and etc. But then also in the data handling, uh, processes where We're anonymizing the data and we're keeping them in very secure, like, uh, servers and, you know, all of that kind of thing. There's protocols for that. So I list some resources here and I know you all have my study slide, at the slide deck, so you can just go to these and, like, read these, um, read these uh references. So, a couple of, like, study types, and I'll give examples of these. So, for example, what you can do is you can do a lottery, like, which I call these, like, you know, in the industry, like, let's say Google, Facebook, etc. they call these like classic AB tests. So, AB literally means like, I can do A or I can do B in the case of something. Like Facebook or something like that. Should I show the, you know, advertisements, like, on the side or like right under, or should, what the color should be or etc. They're constantly doing these experiments, right? So, that's the AB test, but they're all kind of static. Um, let's say, meaning that, like, you're not changing the assignment probabilities or, or anything like that over time. So, Examples of this is like we did a cash transfer experiment in Malawi. We gave some, we chose some villages to be conditional cash transfer villages and some villages to be unconditional cash transfer villages, and some villages to be controlled, right? What you can do sometimes is that you might say, well, look, Actually, it really seems very inefficient and also a little bit unfair and potentially demoralizing to exclude people who are very suitable for a program, but also to include people who are very unsuitable for the program. And then we should really randomize only among the people. That we're unsure about whether they, they are good for it or not, and they'll benefit for it or not. And this is David McKenzie's Nigeria Business Plan competition was an example of this, where essentially they had A competition where people submitted business plans. And at the end, there was a quite a big reward. Like, I don't remember if it was $50,000 or $250,000 or something like that. But, you know, big reward at the end. So what they did is like, they had an independent panel anonymously score all of these proposals. And then let's say if you got like, between 90 and 100, you were given the award, right? And then let's say if you got below 50, Then you were not given any award. But the people between 50 and 90, they weren't quite sure, like, they were decent proposals, but, you know, they could be successful, they could be seemed a little risky. So, among those, they actually randomized the, um, the intervention, and then that's, that's the basis of the evaluation. So, now, what are you doing? You're evaluating uh the effect of this business plan competition for Firms or like entrepreneurs who are kind of marginal, and by marginal, I mean They could go either way, like the an independent panel didn't decide that, like, no, this business is definitely not going to succeed, this proposal is terrible, or this proposal looks amazing, they already have some like proof of concept, whatever, etc. So you've, you're excluding those, your evaluation isn't going to say anything about those kinds of things, like whether giving them $50,000 or $250,000 is Effective or not, but it's going to say something about this marginal group and that's still very valuable. So, you can do something like that. I like those designs a lot actually, because they, they take into account some targeting mechanism where you're being a bit more ethical rather than kind of like randomizing everybody where some people may definitely need, I mean, if you think about, instead of business plan, you're thinking about a poverty intervention. Now you're talking about much more Uh, potentially, uh, devastating consequences for being in the treatment versus the control, right? Phase-in designs, um, the most famous example of this is the Worms paper of Miguel and Kramer from Econometrica in 2004. And this was the, the famous deworming study in Kenya in the Busia district, where they had 75 schools and they divided them into three groups, Group A, B, C, 25 schools in each group. And essentially what they said is that in year 1, we're going to treat Group A, and then in year 2, we're going to treat Group B, and then in year 3 onwards, like everybody will be treated. Now, for the, at the end of the first year, you can do an evaluation where you're comparing treatment group one to treat, you know, the A to B and C. In year 2, you can actually compare A and B to C, but then you have to take into account the fact that A is in the treatment longer and maybe that has some effects, right? And from year 3 onwards, you can't really do, They kind of like the comparison anymore other than it's not a pure comparison other than now the only difference between these groups is that they've been exposed for different durations. Now, there's one exception to that, which is that if you actually think that You're doing things for cohorts inside the schools, and there are going to be cohorts that just miss out and they never get treated. So, suppose that we say, anybody above grade 8 is no longer eligible for deworming. This is not true for deworming, but I'm just hypothetical. Um, then what you can do is, like, the people who were in grade 8 in the comparison group in B and C. They will never get treated going forward. The, the grades 8 the next year and the year after that will get always treated, but they won't. So you can take grade 8 in Group A versus B and C, and forever, you have a pure Kind of like RCT where the control group has never been treated, right? So, if that's possible, and if there is, if that's actually legitimate, meaning that maybe there's effects on these grade 8 people from the fact that they're like 1 year younger people around them are getting treated and then there's spillovers and etc. then it's still not great, but you can think of it that way. But those are the phase and designs and The reason why people don't like phases in design, some people at least, is because it's hard to do long term evaluations, because for, think of my study in Tanzania, where we had people apply for business, um, Kind of like income generating activities at the group level, and there are really no cohorts, like comparable cohorts or anything. They're like 100 villages, in 100 villages, there are 100 groups that applied for something. And we said to 50 of them, you will get your money now. And then the other 50 of them, we said, you'll get your money next year, right? So we really have 1, after 1 year, they'll get a huge infusion of money. And it's hard to argue that, like, at, after that point, there is this kind of any difference, really, other than the fact that one group had their money 1 more year. Right? And so, you essentially have only one year to kind of evaluate the effect of the program. And if you don't think that you're going to see the effects that you're hoping to see in that duration, then a phase-in design isn't really good for you. And then even, even if you do, it's, you know, it, it's People may say, well, actually, I would really like to know what happens like 4 years later, maybe they converge back or something like that. You don't really have an opportunity to see that because you essentially have this group, comparison group has been treated. The encouragement design, I mentioned it already, like the impacts of economics blogs, David McKenzie and I like wrote a paper on this and published it because we essentially did exactly what I described to you. We wrote an email to like 300 students and their advisors at the economics department saying, hey, we have this new blog, like, why don't you check it out? And then 300 others that we had listed before and randomly excluded, we didn't. And it did really cause a difference in how many people do report reading our blog, and then we could use that difference to just kind of um that differential exposure to evaluate whether people knew about development issues a little bit better by reading our, um, by reading our um blog. But note here is that like, you're not any longer evaluating the intention to treat effect, but you're evaluating the, what is called the local average treatment effect. And how it's defined is that that's essentially you're evaluating the effect of the program on people. Who would not have taken up the intervention, the program, in our case, read our blog on their own, i.e. without the intervention, but would have read it if we intervene, right? That's a very specific group, but that may be exactly the kind of group that you're interested in. So for example, if you're New York Times, and you're saying, should we advertise because we have like, you know, we have huge readership, but it could be even bigger. And what if we advertise, which is kind of like our emails to people, then it will get them to sign up or not, right? And so, that's exactly the group that you're interested in, the people who would not have read the New York Times on their own, but if you advertise, they will. And you want to know how big that group is, and then what their characteristics are, and like, whether they'll pay, how much they would like to pay to sign up and all of that. So, that That might be the estimate of interest, but it may not be. So depending on that, you can do these encouragement designs. The encouragement designs need to satisfy instrumental variables, uh, criteria, which is that They need to actually, your encouragement should only work through people taking up the intervention and shouldn't directly affect the, um, the outcome itself, right? So you could imagine, for example, that if in our blog, Email. We had also given people some information. We said, like, not only we said, oh, here's a new blog, but also your other choices of reading things are like Chris Plattman and uh Paul Krugman and this other blog and Tyler Cowen, etc. Um, then all of a sudden, We can't really use it anymore to isolate the effect of its effect on because now we've essentially given people like a bunch of other information that could influence their knowledge. So now, we violated one of the, um, you know, necessary conditions for an instrumental variable to work, which is the exclusion criteria. So, you need to essentially think about Is my encouragement just really only going to work through the intervention. So, for example, think of the kinds of things like David McKenzie and his colleagues at the finance group do with like banks. Banks will just, let's say, send a mail, like even like a postcard to their clients saying, we have a new credit card, like 3% interest rate or something like that, like, please, you know, it's hard to think that that's going to affect your finances, seeing that. You know, other than you actually apply and like, get the credit card. Um, so, that's the kind of thing you want to be doing, very light touch encouragement that hopefully works to increase substantially the takeup of the program, but it doesn't affect anything else. So, another version of phase-in is this kind of like step wedge design. These are very popular in medicine. So, for example, this is how the Gambia famous hep B vaccine trial was done. You can see here that the light blue areas are areas where like people are waiting that many weeks to be exposed to the intervention. And then other, you know, the dark blue is the one that you're actually now exposed to the intervention, and you can essentially compare the people who are in there versus who are not in there throughout the, the transition period. But um, You have to worry about two things. One is time trends. So, for example, some intervention may work better um now than it, it, it will in like 3 weeks' time, mainly because, you know, potentially, maybe the government says like mask mandates or something like that and all of a sudden vaccines aren't that important, as opposed to, it's a free for all and the government says there's no reason to worry. Everybody can go anywhere they want. And then maybe vaccines are much more important then. So, there's those kinds of things to worry about. But then there's also anticipation effects, and this was actually something that we think really, um, derailed are like the Tanzania work that I talked to you about, which is that, you know, you're saying to people, well, you'll get your money next year. And we essentially thought that, well, this is an ultra poor program. These people are very credit constrained. So, there isn't really much that they can do in anticipation of getting the money next year. But actually, that turned out maybe that we have some suggestive evidence that that may not have been true, that people can actually say, well, You know, I'm getting this big amount of money next year. So, and it's a credible promise. So I can literally borrow against that promise. Like, it's almost becomes my, like, collateral, and I can start my business things, like, now. Now in anticipation, or maybe I'm going to eat more or eat less or whatever. There's all sorts of things that I can do in anticipation, all of which kind of bias the effects because if people didn't know that they were going to be treated next year, they would have acted differently. And the final thing that I like this a lot, and this is actually also used in medical trials, and this is called a fried egg design, and then you can look here at the bottom, the kind of like the top left corner. You know, you, this blue area here is a, is a cluster, it's a treatment cluster, so everybody in that blue area is assigned to, let's say, the vaccine. And then everybody in the brown area is assigned to placebo, right? But the brown area and the um blue area are adjacent to each other, and I worry that there's going to be spillovers here. So, what I do is I actually take the yolk of the egg, so to speak, and this is the kind of the white parts of the egg, they are fried egg. And then I only study the people in the yolk, because they are only surrounded by people who are also vaccinated, and these people are only surrounded by people who are unvaccinated. So, then, essentially, my assumption is that these people and these people are not going to interact with each other at all, so I'm going to get a really pure estimate. Um, rather, or, or their amount of spillover on each other is going to be much, much less than these people. So, if I studied them also, um, that would, that would have been a problem, right? So, And that brings me actually to kind of like this idea of clustered versus individual randomization. And then we talked about the stable unit treatment violation uh value assumption last time or the Suva. You know, so remember that aspirin example that we were talking about last week or two weeks ago, you know, does my headache depend on also you taking an aspirin? Probably not. But maybe, like, if actually, you know, we live together and you just become an awful person, like when you have a headache, then you start giving me a headache. Maybe you taking an aspirin helps me get it, you know, like, not get a headache or not. So, we have to always think about. Is there Is there any possibility of interference from the treatment units to the control units? And if so, how do I deal with that? And I think this is a big enough problem, especially when you're trying to just kind of publish papers. Now, referees are very attuned to this, like, you will always get these questions, and it's, um, you know, We do see papers getting, you know, good papers getting kind of like knocked and, you know, rejected because they can't really, they've done an individualized randomization or something like that, so they can't really rule out that there's no spillovers. Um, and so it ends up in, you know, you have a perfectly good RCT at the kind of like the individual level, you've done everything correctly. But now people are saying, hey, how do I know that like, that person and that person aren't friends, and then they're like teaching each other the things they learned, whatever. And that's, that, that becomes a problem. So I think it's important to think about these things. In many cases, there are where Suwa is violated, like in a pandemic, in agriculture where wind is going to take like your, you know, seeds and the sprays and things like that across. So you have guard rows and agricultural experiments. You can have general equilibrium effects. So, the more people I do job training with, the more spillover effects they will have on people who are in the control group because they will get jobs and then they'll take the jobs that those other people could have gotten. So, if there's actually a Limited amount of, like, let's say, winner take all kind of reward, and I'm favoring one group over the other, all I could be doing is causing displacement and then putting people, some people in front of the queue rather than the back of the queue. And so that's uh interference, or in classrooms and in schools and things like that, if I'm like, let's say, tutoring some people, then they could maybe going home, like, and then, you know, helping their friends and etc. who are in the control group. So, you can think of many cases like this. So, I think what you want to do, generally speaking, if you're worried about this, you want to Think of like some clusters where that interaction is minimized. So, let's say villages in our Malawi case were kind of like 5 kilometers far from each other. And towards the end of the intervention, we actually, you know, actually, after the intervention finished 2 years later, when we were talking with some You know, people in the control group in their follow-up surveys, we were really happy to see that actually, they had no idea that there was a cash transfer program going on in other parts of the same district. Um, but then on the other, uh, uh, and, you know, for good reason, because if they know, then there could be, you know, mental health effects, they could be like, oh, you know, compliance issues and things like that. They could be asking for money from treated people and so on and so forth. So, You want to do kind of like sufficient. Distance between your subjects, whether that distance is social or geographic or otherwise. Um, and then I think you run into a Like a main trade-off you have is power, because individually randomized, uh, trials are very powerful because you essentially have each person randomized, so the statistical power is just much higher. And the power loss comes from the fact that within clusters, you have a lot of correlation. So, think about the, Case where think of households as clusters, right? But then think about something that's very, very correlated within a household. So, like COVID actually now is the Omicron is a good example. I know many people, once their kid comes back from school and then has COVID, the whole family gets COVID very soon, right? So, in that case, It's not going to help me to sample 2 or 3 people from that household. It's going to give me zero extra power if everybody is going to get COVID in that household. Like, it's perfectly correlated, then all I need from that cluster is just one person, right? But that means, what that means for my power calculations is that I need many, many more clusters to get, like, good, good inference, right? Whereas if things aren't that correlated, You know, let's say it's actually cancer that's not environmentally related, but like otherwise. And, you know, so then I think, you know, you can have 3 different people from the same household, their outcomes aren't that correlated with each other, that's fine. So then you, you, you just still get benefits from, like, increasing the cluster size rather than increasing the number of clusters. And that's essentially the main calculation that you're making when you're trying to do individual versus Um, individual versus clustered design. So we. And so now, I want to talk a little bit about factorial designs. So, What that means is like, so some people also call these 2 by 2, if there's 2 interventions that you're thinking about, but if you, there's 3, it could be a 3 by 3, whatever, right? So, essentially, you're thinking about, I have, let's say, Um, cognitive behavioral therapy for, uh, troubled youth, right? Um, there's a recent paper that came out now some by some of my colleagues, and then you're thinking, well, actually, Therapy for troubled youth is great to reduce like uh crime and, you know, to reduce their like behavioral issues, but They don't really have the bandwidth to, like, internalize all of that therapy and to deal with their lives because they're so poor that they have to just be, you know, like, either stealing stuff or they have to be constantly worried about, like, you know, where the next meal is going to come from and so on and so forth. So this program would be much more effective. If we just gave them like a little stipend also along with it, so. So, that's your hypothesis. Your hypothesis is that if I combine these two things, it's going to work much better than if I didn't combine them. Now, a lot of people, what they will do is that they will cross them fully, meaning that you have a group that's getting neither intervention. You have a group that's getting the cash, you have a group that's getting the, the CBT and you have a group that's getting both, right? In those kinds of things, you know, I think so the way I described it, like, my intention really was like, is CBT better without cash? So, what I really care about is like, CBT plus cash versus CBT. It's just possible that some other people might say, I'm actually also interested in CBT plus cash versus cash, right? Uh, in one of the studies that I did that I'm linking here, the, we were working with the International Rescue Committee in Liberia for adolescent girls. They said, actually, we're not interested in that question at all because we're never gonna give just cash to people. Um, you know, it, at least this was like 5, 10 years ago, you know, it's like, that's not in our programming currently. What we really do want to know though, is that if we add, let's say 50 cents per like attendance to our mentoring program, will it work much better? So, we just want to compare that. So, for that purpose, I'm going to discuss this in a second, we actually did Only 3 groups. So, we had a control group, we had the mentoring program, and then we had the mentoring program that's like, conditional cash for attending the mentoring sessions. So, like, each time you, uh, you know, attend one of the 40 sessions, you get, like, $2 or something like that. So over the year, that adds up to some money. Yeah. But generally speaking, you have these 2 by 2s. And what happens is that if you have these 2x2s, you have to design and analyze them correctly. And David McKenzie wrote a really nice, um, uh, kind of, um, blog post on this like a while back, and I covered it again recently. So, most of the time, there's the long model up top, which is that you have to have a dummy variable to analyze these, like interventions for each cell, right? So you have the, the T1, let's say your CBT T2 is the cache and then T1 and times T2 is the CBT and cache, and then what you're comparing them to is the control group. But often, what people will say, and this is kind of like where things go really wrong, is that people will say, well, actually, That thing that we added really didn't do anything. That B3 is kind of like insignificant and small. So, what we're gonna do is, it'll be much more power if we can take those people in that cell and then reallocate them into their T1 and T2 bins, and then we just run this short model, which is the second thing over here. And then that'll give us like much better, much better idea. Essentially, there's a couple of things that are wrong with that. One of them is like, actually, What that's doing is now, it's averaging, let's say this T1 is CBT, it's averaging it in a world where people aren't giving cash. And then in a world where they are given, let's say some significant amount of cash, and the average of that is, may not be of interest really to the policymaker. The policymaker is either going to be in one of those worlds or the other, but not, not the kind of like the convex combination of it, whatever weights you put on that. Right? Um, so, that's, that's one problem, like, it's just, especially if this was a more eclectic thing that you were doing, you know, there's no way, let's say you're going to be giving cash, but also like some people are going to be in like government provided therapy. and some people aren't, you know, that just sounds very silly. So that's number one. But then there's a second problem, which is a statistical problem, like the, the people who wrote this paper, like Mura Idaran and his colleagues, uh, that I, that I kind of linked to here, um, they show that When that B3 that you said is insignificant, even if it's modestly positive, like, or modestly different than 0. You get false rejection rates, meaning that like you're essentially saying that K1 is significant when you shouldn't actually be saying that, right? So the, the explanation of that gets technical, but that's essentially the, so, when you design a factorial design, you really need to then evaluate it correctly like this, right? Or you can design it differently, which I'm going to talk about in a second. There's a bonus point here that's just kind of more general, like, it's important what the statistical power of your study is, because suppose that you have a very low powered study, right? But then you're still getting things that are very close to significant but not quite. Like, that actually can be evidence against the null, meaning that You actually a P value of let's say 0.11 or 0.17, which we would normally say is that, oh, this is not significant, because your study is so low powered, it's actually very unlikely that you would get that P value if there was nothing going on. So that can be actually evidence for rejecting the null hypothesis, even though traditionally you would say, I don't reject. Right? Conversely, if you have a very, very powerful study, suppose that you have like millions of observations or something like that, and then you're getting a P value of 0.04, now actually, that can be such that like in that data set, that's actually More likely to be evidence for the null hypothesis that there's nothing going on, rather than there's something going on. Because you should really be getting P values of 0.00, you know, 0.004 rather than 0.04. So, it's important to always think about kind of like, what's the power of your study and generally speaking, design studies that are Appropriately powered and not underpowered because that's when you're going to be finding these kind of like, create more likely to be finding these crazy results because the only things that you'll find are very big effects. So, you'll just be saying, oh, look, at therapy is like the best thing in the world, like, and, but you have like 7 observations and then that, that's what it's coming out of, right? Just by chance. Um, so, You can You can leave the interaction cell empty, that's another option. So if you're interested in 2, you know, in my example in Liberia, we left the cash cell empty, right? But you can also leave the, you know, you can say, I'm going to compare cash to the therapy, and then I'm not gonna have cash plus therapy, right? More unconventional, you can also choose not to have a control group. You can say, look, I know CBT works. I know cash works. All I care about is like, does cash plus CBT have this multiplicative effect that nobody's thought about, so I'm gonna compare those. To, like, to the, each of these single interventions. I don't really need to show the efficacy of these things against the control group. People don't like those, mainly because in a lot of fields, there's not really very well established um evidence that these things work. And then also, even if there is, people will say, well, how do I know that like, it wouldn't have been like no effect in this setting. So, like, it's, you always want to have it there. So it's a bit risky. Um, but, you know, you could, you could consider doing it. You don't have to have, always have to have a 2x2 or a 3x3 when you're thinking of multiple interventions. So the third thing that I want to talk about, which is, this is something that you may not hear about like elsewhere and it's interesting to hear about is adaptive experiments, right? So, so far what we've discussed are static designs and what do I mean by static design? Like I have 10 of you guys there. I want to know something about you, how you will react to a certain thing. Let's say like, we only invite 5 of you to this lecture and then 5 of you not, right? So, what happens is that all 10 of you I just randomize once. 5 of you are here, 5 of you are not here. I run my experiment. It's the length of this, uh, lecture, and then, like, you know, 22 weeks later, I do a survey, and that's it, right? If it was an adaptive trial, I could think of saying, well, actually, I'm going to take two of these people. I'm going to randomize them. Um, into You know, like, uh, let's say, you know, this lecture, Berke's lecture, David's lecture, and like, you know, nothing or something. I'll, I'll take 3 of you, right? Then, based on that I'm gonna survey you. And then based on that, I'm going to invite the next 3 people, and then, but I'm going to now change the assignment probabilities because it looked like actually David's lecture was doing much better than Berg's lecture, so I'm gonna assign more people to David's lecture and less to Berke's lecture, right? And then, I'm gonna accumulate that evidence and then assess it again, and so on and so forth until I run out of people. Right? So, you can see that like this requires a couple of things that are different than static experiments, and that's why they're not so common in social sciences, although they are more suitable definitely for medical context. One of which is that I need to have Access to either the outcome or a very good proxy of the outcome, very quickly before I have to enroll the next batch of people, because I want to evaluate this batch, and then I want to change my assignment probably. So, for example, in my um family planning work, what I care about is like, are people who say they don't want to get pregnant, getting pregnant within a year, so that a year is too long. But I have something where there's a, you know, they adopt, I can see immediately, like, literally at the end of my counseling session, whether they adopt a long-acting contraceptive method like the IUD or the implant or not. Now, I know that those two things are 99.9% effective in preventing pregnancies. So, essentially, I have an extremely good proxy of, like, these people who have this implant in their arm. They're not going to get pregnant next year. So, even though I don't have the outcome in hand, I have a very good proxy, so I can say, actually, I'm gonna use that proxy to update my success probabilities, um, because I can see that. That's #1. Number 2, It's more suited to contexts where there's enroll, enrollment is like continuous. So think about like a hospital, in my case, the family planning unit and the maternity unit. Women are constantly coming in every day with like requests or deliver a baby, or like the, you know, have miscarriages and vaccinations for their children, etc. Every day, I have a new opportunity to enroll like 1015, 20 more people. So, essentially, I can say, this week is a batch, next week is another batch, and then at the end of each week, I'm just going to quickly analyze the data and then change the assignment probabilities next week. Like, if you don't have that, this option is not completely closed to you, but you have to have a big enough sample. And then also worry about these kind of like anticipation effects and all this kind of spillovers and things like that, where, for example, our colleague Anya Sottman did is that they had these schools um in the country that they're working in, and they had an intervention, and they just divided that school population into two. And then batch A, half of the schools, and then they, they essentially experimented with half of the schools in batch A. They assess that data after 3 months, they learned something from it. And then with what, equipped with what they learned, then they went to batch B, and then they redid the experiment there. So that's a version of this. It's less continuous version of it, and you're a little bit more uh limited because maybe you have only like 500 schools and that's all you can do. Whereas, with my medical examples, I can have like infinite many people enrolling into my study, so I can do kind of like better power stuff, maybe with the exception of very rare diseases, like, so I have colleagues who work at the National Institutes of Health, and they're working on things like Very rare brain diseases, like around the world, there's like 47 people or something like that who have this. So those are hard and they require coordination across the, you know, world among medical people to just have, like, essentially, you have a worldwide study where everybody is enrolling on their site and then you're sharing data and things like that. But outside of that, that's what you need. So, What, what is the advantage of adaptive experiments? I'm going to show you in a second. They might make your experiment more efficient, meaning that you can get to the answer faster with a smaller sample size, but not always. They can definitely minimize what the people in machine learning world call regret, meaning that because I'm learning each day, and then the next day I'm assigning the treatment that's more likely to be succeeding to more people, I'm actually assigning by definition, less people to arms that I don't think are working very well. So, it minimizes regret. It's a bit more ethical, so to speak, right? I also, because I'm allocating more people to the winner, I'm generating more precise estimates for the winner. And what I can also do is I can do this adaptation contextually, meaning that I can actually say, not only I'm going to look at this batch at the, you know, every week, but I'm going to look at this batch based on whether people are adolescents or adults. And I'm going to allow for the fact that the program may be working very differently for those two groups. And then I'm going to update the assignment probabilities actually independently. So, I can have an experiment in 3 months. That started with adults and adolescents and like treatment A and treatment B all given equally to everybody. And if you had a static AB type AB test type of experiment, RCT, then that 25% chance for each of those groups would stay throughout the entire, like, study period. But now, at the end of like 3 months in the adaptive experiment, you could have arrived at the situation where adolescence and treatment A, we're only giving it to 2% of them. But then like adults in treatment B, 70% of the allocations are going to there or something like that. So you can have these kind of like very unbalanced things, and that has implications, of course. So, here's kind of like how to see it visually. So this is a static RCT, right? So you say, OK, I'm going to allocate 25% to each of the four treatments, and then that's kind of how many people I'm going to have in expectation. And then I'm going to estimate at the end, like, these T1 that did this, T2 did this, this is the mean, this is the confidence interval, and then I'm going to say T1 is significantly different than T4 or so on and so forth. So what's the problem with it? Like, like I said, two things. First, throughout the intervention, I assigned a lot of people to the stuff that was inferior, so I regret that, right? The second thing is the standard error on the winner and the standard error on the loser is more or less the same because I allocated the same amount of um people to each of those groups. Whereas I would like this to be much more precise. I'd like to be able to go to the minister and say, this thing really worked. It's very tight, you know, like, I have a great estimate of this. I don't have that. So if I had an adaptive experiment that I had the same number of people, you see, there's like maybe 800 people or something like that, what I would do is I would actually have those people start like, I'd start them equal, but not assign all of them, but just some of them. Get my estimate, right? Now, because I allocated many fewer people, I have large confidence intervals here at the beginning, at the end of the first batch. But then what I do is like, T1 looks like it's um doing better, so I'm going to increase the probability that people are assigned to T1 and decrease the probability that they're assigned to, let's say, T3 and T4, right? So, I'm gonna do this using some statistical techniques that are too complicated to go into here. So then, The number of people assigned is now going to be like this. Remember, I'm still doing a randomized experiment. These probabilities are still random. Everybody has a chance to end up in either of these bins, right? Um, nobody has a zero chance of anything like we discussed last time. So this is still a proper randomized experiment. It's just that I changed the probabilities, right? So now I have these estimates and then I continue doing this and then by the end, you know, it's possible that now I have actually 400 people in T1, 350 and T2, only 150 in like T3 and less than 100 in T4, something like that. And look now what's happened is that T1, which is the best, I have really tight confidence intervals. It's like I can tell that it's better than everything else, whereas I have kind of like, More uncertainty over here, but that's the one that, that I don't care so much about. Now, What we did, we minimized regret to the best we could because we assigned many more people during the experiment to the, to the best expected intervention, and then we also have tighter confidence. So, how you're doing that essentially is that you're just saying, well, look, you know, I'm gonna start with a kind of like a prior distribution that all of these things work the same. So, initially, I'm going to assign them 1/3, 1/3, 1/3, but then afterwards, I'm going to do something like I have this, you know, Thompson sampling method from like the 1930s, um, that's actually going to tell me that like, There's 70% chance that the arm two is going to win this race. There's 25% chance that arm three is going to win, 5% chance arm one is going to win. So, I'm going to literally take those and those are going to be my new assignment probabilities. So, then I'm going to assign people next round, 70, 25, 05. And then at the end of that batch, I'm going to combine those two rounds of data. I'm going to recalculate those probabilities, so on and so forth until I have a winner. And you can have stopping rules about like, when do you declare a winner, so on and so forth. Like maybe when you reach a certain amount of confidence and this and that. Right. You can personalize it, like we said, you can do it for like teenagers versus uh older people, adults, etc. Um, But the caveat is that, you know, because The way you assign treatments is now dependent on what happened before. You lose the IID, um, you know, the, uh, Independent draws, uh, individually independent draws, um, of the kind of like the, a normal distribution. So what you're gonna have is like, you no longer have a, going to have a normal distribution, like of your error terms. So that means that you can't, while you're doing the adaptive experiment, you can't just take your data and treat it like you have a regular RCT and then do the regular inferential analysis. To do anything, you have to do something more sophisticated. And this is actually a very active area of research because a lot of times these very smart people who are statisticians, econometricians, mathematicians, computer scientists, etc. you know, they have to do things by simulation because there's no closed form, you know, formulas or proofs and things like that for the kind of like, let's say the, the asymptotic, um, you know, kind of like the Validity of the kind of like the, uh, you know, consistency and things like that, unbiasedness, all those things. So this is very much an ongoing area of research, but The point here is that there are actually opportunities to do adaptive experiments, and I'm doing one and other people have started doing them as well. You know, as people come to the hospital, if you're doing it, like a, we have a tablet-based thing and then the tablet is doing all the randomization, it's very easy to change. Those randomized assignments, even the nurses and doctors that are doing the interventions, they don't know because all that's happening is like on the underlying distribution of like assignments is changing, but we are actually targeting better and better. We're doing like personalized tailored. Intervention assignment to people who are coming in. So, definitely people who are coming in later on in the study are luckier than people who come in earlier in the study because they're benefiting from all the things that we're learning, and then they're just kind of, like, contributing to this knowledge generation, uh, while benefiting at the same time. One thing to also um keep in mind is that still, even though this is more like this Beijing, uh, type of, like work, where, you know, you don't necessarily need T tests, we're not in this, in that world with either academics or actually even policymakers, like policymakers and, you know, people who make these decisions, still gonna want to say, hey, compared to my status quo intervention at the schools, like the way my schools are now. Can you just show me that like your meals program did significantly better, right? So to be able to do that, you need enough data on the status quo, but in the adaptive experiment, some algorithms can be very aggressive in eliminating, if suppose that the status quo was losing very badly, you can end up collecting very few data points on the status quo. So, you may need to actually anticipate that you're gonna need A quote unquote control arm. Right? And you're going to um You're going to, um, collect enough data on the control arm so that you can actually compare it reliably to the T1 because T1 can have a very small confidence interval, but if T4 has a very big one, you still won't be able to confidently say T1 did better than T4. So, what we do is we have these kind of like, floors, so to speak. So we say like, we're going to allocate at least 10% of the sample to this group, even if it's doing very badly because we need that. Or what you can do is you can have a learning phase where you do the adaptive experiment as aggressively as you want. And then once you say, I've learned that, like, these two ones are doing the best, and then I have my status quo, then you can go to a evaluation phase where you actually now run a static experiment quickly to just say, OK, the winner versus the status quo. Now I'm going to show that, like, with very high confidence that this one is doing better than the status quo. You can do kind of either way. Then the last 15 minutes, I want to talk about a couple of things that are practices, like important practices of analyzing um randomized experiments. So, two things that come up over and over and over again are kind of like baseline balance and attrition. And I want to talk a little bit about each one of them. And if we don't get to whatever I have left after that, that's OK. Um, because these are kind of like the last really most important things. So, on baseline balance, um, you know, I think that There are things that you can do at the design stage. And one of the debates in this between medical people and social science people has been that like, there's a lot of medical folks who say, look, you shouldn't show a baseline balance table because you have randomized in an RCT you have randomized already. So, any differences that you show me are going to be just coming out of that, like, that random chance. So I don't really care, like if there's differences or not. And then there are others who say, well, actually, hold on a minute, that's, that sounds, that doesn't sound right because if there's some really big difference on a really important characteristic that's sermonistic, that's still, that's still troublesome. Just the fact that I randomized doesn't mean that like I shouldn't worry about it. Um, and then there's also the Um I'm going to say? All right, let me, let, let me continue. So, so I think That's There's ways actually you can avoid having those massive differences by chance by doing this stratification and block randomization. So, when I wrote this uh 2 weeks ago, David McKenzie had just written on this, like the perfect reference. So I definitely encourage you to, to read that. It's now been 3 weeks. But essentially, his main takeaways are, you want to do 2 stages. The first stage, when you're approaching your RCT. is that you want to have one or two key variables on which you want to stratify your sample. So, for example, the easiest things to think about could be like, you have urban areas and you have rural areas, they look very different. So why not just do your experiment within them rather than like doing unconditional experimentation across the whole country. Or you can do male and female, and you don't want to end up with, like, 60% male and 40% female, so you You can stratify on that and then have your like equal sizes of male and female, and then you have, you can have balance inside those as well. So, you could do those kinds of things. For example, Guido Imbens, who recently won the Nobel for the work he's done on, partly on these things, recommends that using cluster size to define strata is important because often cluster size is very deterministic of like a bunch of things, right? So then, You have your strata now you're in urban areas or rural areas or you're in large clusters, mid-sized clusters, and then small sized clusters. Within them now, you have your units, and what you want to do is you want to form, let's say something like quadruplets. If, if you have one treatment and one control, you can form quadruplets. And how do you form them? The idea is essentially that I want balance on things that really matter. What are the things that really matter? It's either my outcome. So if I observe the baseline value of my outcome, suppose that we're doing an education intervention and I'm interested in test scores and learning, and I have baseline test scores, right? Then I can use the baseline test scores. To match people into, like, everybody who got a 4 out of 100. Like, as long as there's 4 of them, I put them into that bin. And then everybody who got a 5, everybody who got a 6. And then, therefore, now, if I have my bin of people who got very bad grades, and then within them, I assigned 2 of them to treatment and 2 of them to control, I know for a fact that I'm going to have balance on that variable. It's impossible for me not to have balance on that variable. Now, if you have not one outcome indicator, but a bunch of outcome indicators, but they're closely correlated, then you can make an index of them and then you can, you can do quadruplets like based on being close, you know, nearest neighbors kind of thing on the, on the index. If you don't observe the outcome at baseline, then you should try to do it on the basis of like baseline covariants that predict the outcome really, really well. So for example, in stunting literature, your age in months is going to be extremely important, so you really want to make sure that like, The treatment group isn't 2 months older than the control group because that's going to make a massive difference in their kind of like, uh, Z scores, right? So what you want to do is you want to take the people who are like aged 13 months, and then, you know, like have quadruplets of them, 14 months, 15 months, etc. and then you want to then randomized within that. So, think of it as you have like your strata, you have two strata, strata, urban and rural. Within your urban and rural, you have a bunch of quadruplets, right? And then within those quadruplets, now you're running a lottery and saying like 2 of these guys are going to be treatment, 2 of these guys are going to be in control. So that's the kind of like the main takeaway. And if you do this kind of thing, And I think that I remember the thing that I was going to say in that debate, people say, well, look, you know, If you do that, then you don't really need to report the baseline balance, like, at least not as table one of a, like a thing, you can just toss it into the appendix. So, I think if you do this kind of thing and you're prepared, my recommendation is that it's totally OK to report the baseline balance in the appendix table, and that You can also describe how you did this, you know, randomization, essentially ensuring that on important key variables, you will by definition have balance. Then it's not that important to show the baseline balance because all you're doing is confirming you've done that. You can just stick it into the appendix state. Right? The second thing people say is that, well, actually, maybe you didn't do that, but maybe you had a plan for re-randomizing. So you thought about this thing that can happen by chance, and you said that you're going to minimize it. So, what you can do is essentially, suppose that you have K variant, K characteristics at baseline, you know, age, sex, like test score, da da da, it's an urban, rural, the household size, this and that. And what you're saying is that, let's say, I don't want to see any of the, the P, the differences between the treatment and control or any of those variables to be bigger than a certain amount or the P value to be like higher than a certain amount or something like that. So, you can pre-specify saying, I'm going to randomize this like 100 times, and then I'm going to look through, and then I'm going to Take the one that minimizes the maximum of this like T value, right? Like this difference, like standardized by the um by the standard errors, right? Or you can say, I'm going to re-randomize until all T stats are below a certain threshold. So suppose that you only cared about two variables, like the test score and age, right? And you can say, I'm going to re-randomize until the T tests for both of those are under 1. Right? Something like that. So, you have to be careful there, like they, they should be feasible. If you have many variables and if you put a really like demanding threshold like T equals to 0.05 or something like that, then you're never going to get there. So, you want to be realistic, but re-randomizing is actually, like, in many cases, if you pre-specify it and do it correctly, it's equivalent to You know, like, unconditionally randomizing to get equal numbers of people in each group. So, you, If you do this, you should pre-specify for randomization inference, which you guys covered with Florence, and don't search over randomizations for best value, meaning that you didn't pre-specify, you do it a bunch of times, you're still not happy, and then you say, oh, let me do one more time, or now, this one is OK. Those are not OK because there's then very little randomness left, like you're imposing your will on the kind of like the distribution. But otherwise, if you actually do either of these two options, you're fine. And again, if you do this, you will have balance on the pre-specified variables according to your criteria, right? Then again, in my mind, then it's OK to put it in table, your appendix table one, right? But if you haven't done either of these things, let's say you inherited the You know, RCT or like something like that, or, you know, just for whatever reason, you didn't do it, or you did do some of these things, but like you did unconditional randomization, but you actually don't know whether your RAs did a good job or maybe there was a public lottery, and you don't know if the public lottery went according to plan or something like that. Then this is really a defensive thing. You're saying, look, you know, We randomized, but we didn't do all these safeguards, so now we're going to show you that, like, it's actually OK. Like, these, the treatment and control look very similar to each other. And like I said again, like I was saying before, if your study is well-powered, then you're going to be more likely to reject differences in covariant values between arms, and if it's underpowered, then you're not going to be able to reject anything. So, You don't want a badly powered study to be helping you say, oh, I have balance because there's these massive differences between treatment and control, it's just that they're not statistically significant. So you have to report your minimum detectable effects and you, your, your power, so to speak. And then you also have to have an idea of like meaningful differences between these covariants. Like if I see that like there's 10% points more people with university education in Malawi, but it's not significant, I'm not going to sit there and say like, oh, treatment and control look exactly the same when like 10% points difference in education, the university education is a huge deal in a place like Malawi. The second thing you want to do, which I don't see often enough, is that Instead of just looking pair-wise, like this variable or is it different, this variable, is it different or counting them, whatever, you should be doing joint tests of orthogonality, which means that is this vector of variables for treatment different than the vector of variables for control, and then you want really high P values in these tests. So like in the links, like I have some examples here, I'm very happy. When I see that, like, the treatment and control, the, the difference between those vectors is like a F test, like a P value of, like, 0.862 or 0.698 or something like that. And I do get actually worried when it's like 0.11 something, because then I can see that actually, some of the variables are looking different. It's just that it hasn't quite reached traditional significance level. The last thing we do before we go is kind of attrition. I think the thing that I see with attrition that bothers me most is that people generally say, well, OK. Attrition is really kind of like balanced in levels, so I'm not too worried. And that's essentially to me is a big cop-out because you could have had like to be facetious, let's say you have 10% attrition in both the treatment and control group, but in the treatment group, all the right-handed people left, and then in the control group, all of the 10% that left are left-handed people. Now, they're very different than Each other and so that's going to affect things and so you can't just say, OK, I'm going to check levels, you have to check for differential attrition between treatment and control in both levels as well as characteristics, right? If you can show, show evidence that like you're OK in both of the, I mean, you're not gonna ever have a study that has zero attrition. I mean, I have one One, but that's mainly because it's the administrative data. So like, people do their like counseling session, I see them, so nobody drops out. But as soon as I have to follow them up, like to ask them a question, I'm gonna lose some of them, right? So you're always going to have attrition, so you're always going to have to do these things. If you're lucky, your attrition will be minimal. And then it will be balanced in levels and characteristics, so that you can essentially proceed with like showing the data for like the 96% of the people that you were able to follow up or something like that. But in other cases, you're gonna have to use, like other, other types of things. And one of the things that I, uh, tell people is that, you know, the bounds analysis. I think it is really important and it needs to be much, much more prominent in many studies than it is. Often what I see is like people say, oh, attrition, yeah, we have some attrition, sometimes it's like 20% or something like that. People say, oh, that's OK, 20% is pretty comparable to like other studies like ours, and we see no differences between treatment and control, so like we're just gonna go ahead. I think in my mind, that's more and more unacceptable, and I think you want to be doing some sort of adjustment, and that usually my preferred route is these Kling and Liebman bounds, and you can see these uh linked papers that I have here. Um, and what that does essentially is the following. Suppose that I have like 5% attrition in both my cash transfer group and my control group, right? And then suppose that like the, it's the, you know, something about, let's say consumption um that I'm interested in, right? So what I want, so, and suppose that when I just compared them without any adjustments, I see that there's an effect, like there's a large effect. So now what I wanna say is like, suppose that the 5% in the treatment group that dropped out had 10% less um Consumption than the mean. And then the control group who dropped out, had 10% more consumption than the meat. Like, what would have happened? Usually, we do this by standardizing, so we say like, what if they, like, treatment group had 0.1 standard deviation, less consumption, and the control group had 0.1% more. I recalculate, like, I fill in those missing values for the treatment and control. I recalculate the effect for all the 100%. Now, my effect is going to be smaller because what I did is I said, I stacked the deck against myself, right? I said that treatment group attrition was selective in a way that benefits treatment and control group selection was, uh, that, that actually harmed their like estimates, so now they got closer to each other. But I do it, and then there's still an effect and it's still significant. Then I say, OK, uh, let me do it again with 0.25 standard deviations. Like, I do it again and it's still, it's smaller, but it's still significant effect larger. And then finally, I do 0.5, and that's where I lose like they become equal and then there's the difference is no longer significant. Now, what I've done is I've said, look, my study is robust. Even if the differential attrition was selected to such an extent that like even if there was a 0.8 0.8 standard deviation difference between the consumption values of the people that attrited from my study, like 0.4 up for the treatment and 0.0.4% down for the control. I still get significant effect that cash transfers are working very well here, right? So then, It's a matter of debating with people, like, is it really realistic to think that attrition is so selected that, like, I would have these massive differences in the pattern of it, like, and then, yes, at this point now, what you're doing is you're doing a little bit of a discussion with your referees or with your audience and with the policymakers, you know, about something like that you do with non-experimental studies, like, is the different, difference in different valid because the parallel trends assumption is there or is the IV estimates valid, etc. So you're in that world, but at least you put some bounds on that world and you say, you can actually say that I have differential attrition, but I still have A significant effect. And only after you've gone to this crazy difference between these two groups, which I don't think is plausible, then I have like no effect. So, at least that's something. So I think that's what I suggest people do with attrition stuff. And then I had one more thing on covariant adjustments, but once again, we've run out of time, so I'm going to stop there, and this time I'm not going to offer. Um, to do a 3rd lecture on this because we have actually very little. This is the only thing that's left. But, um, I've written so many blog posts on this, and then actually there are much better ones that I linked to here on this topic that, like, essentially, Randomized experiments, the use of covariates is, is important. I think it's very valuable, but you wanna do it in a way that's transparent, that's pre-specified, and that's correctly specified, and all the information that you need is here. You don't need me to, like, talk about it. Um, but with that, I think I'll stop here. And I know that this is your last lecture, and I know that you guys are not going to be McNamara Fellows. Uh, soon. I wish you the best. I hope it was a good year for all of you. We're sad that we couldn't have you here, but, uh, I wish you all the best, and if you have any questions, feel free to email me. Thank you so much, Mark. I really appreciate it.
showAllTranscripts
no
duration
PT1H30M27S
scene7File
worldbank/RSMFP Lecture with Berk Ozler Part II
scene7Domain
https://worldbank.scene7.com/
scene7FileAvs
worldbank/RSMFP Lecture with Berk Ozler Part II-AVS
title
RSMFP Lecture with Berk Ozler Part II
description
RSMFP Lecture with Berk Ozler Part II
showTimestampAndTranscript
yes
col-xs-12
col-sm-12
col-md-3
col-lg-3
col-xs-12
col-sm-12
col-md-7
col-lg-7
  • add-style
  • lp-body-content
The sixth lecture of the Robert S. McNamara Fellowships Program Lecture Series is presented by Lead Economist and Research Manager, Poverty and Inequality, Development Economics.
lp-heading-top-medium
lp-heading-bottom-medium
col-xs-12
col-sm-12
col-md-2
col-lg-2