00:06 Well,
00:06 good morning,
00:07 everyone,
00:07 and welcome to the 5th and final lecture in the Robert S.
00:11 McNamara
00:12 Fellowship Program lecture series.
00:14 So as a quick background,
00:16 the World Bank's
00:17 Robert S.
00:18 McNamara Fellowships Program,
00:19 known as RSMFP
00:21 matches aspiring development economics researchers from developing
00:24 countries with World Bank research economists.
00:27 Fellows have unique opportunities to participate in policy,
00:30 in rigorous policy relevant research in the
00:32 World Bank's Development Economics vice presidency.
00:34 The RSMFP lecture series is a flagship activity of the of the RSMFP.
00:39 Um,
00:40 over the course of 5 lectures,
00:41 fellows and other attendees will learn about
00:43 the latest developments in empirical research.
00:45 The lecture series focuses on topics that,
00:47 although not always taught in graduate courses,
00:49 are fundamental for development researchers.
00:51 Today,
00:52 we're delighted to welcome back Brooke Osler to give
00:54 the second half of his lecture on why randomize,
00:56 what to randomize,
00:57 and how,
00:58 combining basic principles of RCTs with practical aspects of field experiments.
01:03 Brooke Osler is a lead economist and research manager of the
01:05 Poverty and Inequality Research Program in the Development Research Group.
01:09 He received his BSC in mathematics from Bosphorus University in 1991
01:12 and his PhD in Economics from Cornell University in 2001.
01:16 After working on poverty and inequality measurement,
01:18 poverty mapping,
01:19 and the 2006 World Development Report on equity and Development earlier,
01:23 he decided to combine his interest in cash
01:25 transfer programs and HIV risks facing young women.
01:27 In Africa
01:28 by designing a field experiment in Malawi.
01:31 He has since been involved in a number of cluster randomized field experiments.
01:34 He's a co-founder of and regular contributor to
01:37 the Development Impact blog.
01:39 So,
01:39 without further ado,
01:40 let me hand it over to Burke.
01:41 Um,
01:42 thank you so much for joining us again and
01:44 looking forward to the rest of this lecture.
01:47 All right,
01:47 perfect.
01:48 Thank you,
01:48 Maria.
01:49 Uh,
01:49 and thank you,
01:50 Napur,
01:50 for,
01:51 uh,
01:51 again,
01:51 organizing this.
01:53 Um,
01:53 yeah,
01:54 like,
01:54 uh,
01:54 you were all here,
01:55 I think last time around,
01:57 and,
01:57 you know,
01:57 I didn't get through a lot of it,
01:59 so I volunteered,
02:00 but I wasn't sure there'd be
02:02 some demand.
02:02 But,
02:03 um,
02:03 the time folks tell me that there was some,
02:06 at least somewhat enthusiastic response.
02:09 So here we are again,
02:10 and this is where I
02:12 remembered we were.
02:14 stuck,
02:14 I think.
02:15 Um,
02:15 so I'm just gonna continue.
02:17 Um,
02:18 you know,
02:18 the first part was a bit more
02:21 kind of like conceptual and kind of like about counterfactuals and
02:25 things like that.
02:26 This one will be a little bit more
02:28 practical,
02:29 uh,
02:29 for when you're doing,
02:30 uh,
02:31 RCTs,
02:32 when you're planning RCTs,
02:33 what kinds of things you should be thinking about.
02:36 So it can be a bit more informal,
02:37 like,
02:37 I'm not going to
02:39 go through too much detail on kind of like,
02:42 There's technical things,
02:43 but I'm not going to go through the kind of,
02:45 like,
02:45 why those equations work and things like that.
02:48 But also feel free to,
02:50 you know,
02:50 interrupt.
02:51 You guys are generally,
02:52 uh,
02:52 I attended a few of these.
02:53 You guys are a quiet bunch.
02:55 But so,
02:55 this is the last one.
02:57 So,
02:57 um,
02:57 feel free to just kind of ask questions,
02:59 that there's really no stupid questions,
03:01 and then if I don't know the answer,
03:02 I'll just say,
03:03 I don't know,
03:04 um,
03:05 because some of this stuff is,
03:07 you know,
03:07 cutting edge,
03:08 and then we're just all learning.
03:10 Um,
03:10 together.
03:11 So,
03:11 I'm gonna
03:12 just jump back in,
03:14 um,
03:15 with the kind of
03:16 one of the common critiques of randomized experiments and,
03:19 you know,
03:20 something that policymakers
03:22 regularly face.
03:24 is,
03:25 you know,
03:26 what evidence to use for
03:29 the,
03:29 kind of like,
03:30 to inform the design of their programs.
03:33 You know,
03:33 I'm just coming from an event now in Mozambique.
03:37 Let's say in Mozambique,
03:38 they didn't really have an RCT on this,
03:40 like,
03:40 adolescent girl program that they're thinking about,
03:43 but there's an RCT from Malawi,
03:45 which is a neighboring country.
03:47 Um,
03:47 but then they have really good,
03:49 let's say,
03:49 demographic health survey data,
03:51 and they can run a bunch of like regressions,
03:53 maybe even do like diff diff or something like that,
03:56 etc.
03:57 The question is,
03:59 You know,
04:00 should we use our
04:01 own data,
04:03 but we're not going to have a randomized experiment,
04:05 or should we have the randomized experiment,
04:07 but it's not our own data,
04:09 you know.
04:09 And
04:10 people do disagree about this,
04:12 uh,
04:12 the answer to this question,
04:14 and probably there isn't a correct answer.
04:17 Um,
04:17 and people have been disagreeing,
04:18 as you can see from these quotes for like 60 years or more.
04:22 Um,
04:23 and some people think that's really internal validity,
04:26 which I'll define in a minute,
04:28 like,
04:28 you know,
04:28 RCTs are kind of like should be
04:31 prioritized.
04:32 Um,
04:33 and then external validity,
04:34 whether some evidence applies to some other context should be just secondary.
04:39 Um,
04:40 and then there are other people who say,
04:41 look,
04:42 if I'm trying to make policy,
04:43 like it makes no sense to value one type of validity above the other,
04:47 because there may literally be nothing
04:49 with that type of validity for me.
04:51 So,
04:51 what am I supposed to do?
04:53 Um,
04:54 so,
04:54 we're essentially thinking here that,
04:56 you know,
04:57 um,
04:58 think of an example like this in,
05:00 in the United States,
05:01 there's a famous,
05:02 uh,
05:03 preschool program called Head Start.
05:04 It's all over the country.
05:07 But in the 60s,
05:08 some people did a randomized experiment in like a small school district.
05:12 It's a famous study called Perry Preschool Project.
05:16 And,
05:16 you know,
05:16 that one showed that,
05:18 like,
05:18 the um
05:19 graduation rates for kids who are
05:22 in the um Head Start program
05:24 are significantly higher than people who,
05:27 who are not,
05:28 right?
05:29 But then there's a,
05:30 oops,
05:31 sorry,
05:31 there's another study that actually
05:34 shows that,
05:34 like,
05:35 um,
05:36 You look at all the Head Start programs around the country,
05:39 and you look at people in Head Start versus you look at people not in Head Start,
05:44 and the graduation rates of Head Start are 13% points lower,
05:48 right?
05:49 So,
05:50 Yeah,
05:51 in this case,
05:51 I think the answer is a little bit
05:53 easier in the sense that we can easily think of the selection issues.
05:57 Head Start program is a program for underprivileged kids.
06:01 So,
06:01 of course,
06:02 their graduation rates are going to be lower,
06:04 potentially,
06:05 even
06:06 with the program
06:07 than people who have better means and then maybe
06:10 going to better public schools or private schools,
06:12 whatever,
06:13 etc.
06:14 So,
06:14 here,
06:15 you know,
06:15 you have
06:16 this.
06:18 Kind of like exaggerated difference where the effects are going the other way,
06:22 and then theoretically speaking,
06:23 we have an expectation that like,
06:25 you know,
06:26 there's no reason to think that
06:28 this preschool program would have stark negative effects.
06:32 But
06:32 you can have
06:34 cases where both effects are positive,
06:36 but they're,
06:36 one of them is small and one of them is big,
06:38 and that makes a difference for cost effectiveness and things like that,
06:42 then you would be more scratching your head saying,
06:44 well,
06:44 which one of these we should trust more.
06:47 Right.
06:48 And in that,
06:49 in that sense,
06:50 I think,
06:50 um,
06:51 you know,
06:52 thinking about
06:53 Methods to just kind of extrapolate results from one place to the other.
06:58 Um,
06:59 there are interesting work that's been done in places
07:01 like the United States where there's something like,
07:03 uh,
07:04 I don't know,
07:04 it was like electricity utilities or something like that.
07:07 There's a bunch of different independent ones.
07:10 And then,
07:11 you know,
07:11 Colorado is very different than Montana,
07:13 which is very different than Texas,
07:15 which is different than New York.
07:17 And you have these evaluate randomized evaluations in those places.
07:22 And then maybe that gives you enough of a variation to say,
07:25 well,
07:25 what predicts success?
07:26 So you can look at kind of like these things,
07:29 and then you can go to a 5th or a 6th uh state that I didn't
07:33 um mention here,
07:34 and then try to just see if the convex combination of those estimates
07:39 creates like you,
07:40 uh,
07:40 you know,
07:41 Arizona or something like that,
07:42 where you can say,
07:43 Well,
07:43 based on what I know from all these other places,
07:46 I think Arizona is going to have something like this.
07:49 Or you might say,
07:50 well,
07:51 really,
07:51 like,
07:52 not only we don't think we're gonna
07:54 be able to do this,
07:55 but also
07:57 running that kind of an experiment in our own,
07:59 um,
08:00 backyard
08:01 is not that costly.
08:02 So we're going to do that ourselves and then decide.
08:05 So,
08:05 there's really,
08:06 in my mind,
08:07 no correct answer here.
08:09 I do think
08:10 that,
08:11 um,
08:12 People like Lance Pritchett and Angus Deaton and others who say,
08:15 you look,
08:16 you shouldn't really
08:17 prioritize um RCTs.
08:21 They,
08:22 they do have a point.
08:23 Uh,
08:23 I often disagree with them about whether we should prioritize our cities or not,
08:28 but they do have a point in the sense that,
08:31 yes,
08:31 ideal randomized experiments may take priority,
08:34 but hardly ever any RCT is ideal.
08:37 And,
08:38 um,
08:39 You know,
08:39 sometimes the confidence intervals may be too large and sometimes,
08:43 uh,
08:43 you know,
08:43 our,
08:44 our,
08:44 our confidence in other aspects because of things like
08:47 attrition and balance or like compliance and things
08:51 like that that we're gonna discuss later,
08:53 um,
08:53 may be,
08:55 may be low.
08:56 So,
08:57 in those cases,
08:58 it is possible that actually,
09:00 despite the fact that you may have a biased estimate
09:03 from a
09:04 Um,
09:05 from a,
09:05 um,
09:06 let's say,
09:06 a non-experimental study.
09:09 Um,
09:10 it may be closer to the truth
09:13 than
09:14 the RCT from another place with a big confidence interval.
09:18 So,
09:18 and there's really no way to know
09:20 without doing it yourself,
09:21 so you have to do just some extra legwork.
09:24 Um,
09:25 but
09:26 other issues,
09:27 of course,
09:27 there's the ethical issues that come with RCTs.
09:30 I'm going to touch on them a little bit later on,
09:32 but the biggest thing for me is the concept of equipoise,
09:36 which many people
09:38 do not hear about the word until you actually do RCTs.
09:42 I didn't know the word,
09:44 like,
09:44 before,
09:45 um,
09:45 especially as a non-native English speaker.
09:48 It essentially means that,
09:49 like,
09:50 do I have enough uncertainty about the answer to this question?
09:54 That I should be going ahead and randomizing people,
09:57 because
09:58 it is really,
09:58 if you think about it,
09:59 it is really unethical,
10:01 you know,
10:01 just to,
10:02 let's say,
10:03 just to have an RCT so that you can have ironclad results
10:07 for a study that you can publish well in like a journal or something like that.
10:12 On a question
10:13 that we're pretty sure of the answer,
10:15 and we're pretty sure of the kind of like,
10:17 let's say the cost effectiveness and things like that,
10:19 then
10:20 you should just be targeting the program to people
10:22 who need it rather than just randomly allocating it.
10:25 But
10:26 often,
10:27 you know,
10:27 economists,
10:28 um,
10:29 can find something about which
10:32 there isn't any equipoise.
10:33 So,
10:33 for example,
10:34 we do know
10:35 Um,
10:36 that this program works,
10:38 but we want to know whether it works better than just giving people cash,
10:41 let's say,
10:42 or something like that.
10:43 So now,
10:44 what I want to do is I want to have a horse race,
10:46 and I really am saying that
10:48 I have no idea about the relative cost effectiveness of this program to the,
10:53 to its alternative.
10:54 So,
10:55 that's the evidence I want to
10:57 ascertain.
10:58 I know that this program is going to work.
11:00 It's just that I don't know whether it's going to do
11:02 significantly better than something else that I could have done.
11:05 Um
11:07 Then there are things when you do
11:10 RCTs type of studies
11:12 that come up,
11:13 and two of them that I like the most,
11:16 and they are really big dangers to kind of like internal validity,
11:19 which is something we said we,
11:21 we value,
11:22 the RCT proponents value.
11:25 And
11:25 one of them is Hawthorne effects,
11:27 which is essentially the idea that,
11:29 like,
11:29 just because someone is watching me,
11:31 um,
11:32 I behave differently.
11:33 And my favorite example on this,
11:35 and I don't know if we covered this last time or not,
11:38 is the,
11:39 is this idea that,
11:40 you know,
11:41 that,
11:41 you know,
11:41 you can think about
11:43 what's the effect of soap,
11:45 using soap on kind of like,
11:46 uh,
11:47 you know,
11:47 health outcomes or diarrhea or something like that.
11:50 And,
11:51 um,
11:52 I have colleagues at Stanford who work on sanitation issues.
11:56 They were worried about the self-reporting issue.
11:58 So they essentially put,
12:00 like,
12:00 kind of like these observant people,
12:02 like,
12:02 in the community,
12:04 to just kind of,
12:05 like,
12:05 be observing things.
12:07 And then somebody said,
12:08 well,
12:09 actually,
12:09 because they're observing,
12:10 people are just going to be washing their hands when they're around
12:13 and not going to be doing it when they're not around.
12:15 So,
12:16 what can we do?
12:17 And what they did is they literally put,
12:18 like,
12:19 motion sensors inside the soaps,
12:21 and they distributed soaps to everybody.
12:23 And
12:24 you can guess,
12:25 like,
12:25 where this is going,
12:27 when you,
12:27 the,
12:27 you know,
12:28 there's 3 ways to measure.
12:29 You ask people,
12:30 do you wash your hands?
12:31 And let's say 65% say yes,
12:33 that they wash it after going to the bathroom and before meals.
12:37 And then,
12:38 when you actually look at what the people observed,
12:41 like,
12:41 the enumerators,
12:42 that goes down to,
12:43 like,
12:43 30% or something like that.
12:45 And then when you go to the motion sensor,
12:47 like,
12:48 how many times did the soap move during the day,
12:51 that goes down to like 10,
12:52 15%,
12:54 right?
12:54 So,
12:55 it's not only the,
12:56 there,
12:56 there's the issue of importance of measurement here that's separate,
13:00 but the bigger thing is actually people are behaving differently
13:04 because they're in an experiment and somebody's watching them.
13:07 A different version of that is actually
13:11 John Henry effects,
13:12 and this is not about being watched,
13:13 but this is about the fact that
13:15 there is an experiment going on,
13:18 and people have incentives
13:20 about
13:21 the outcome of that experiment.
13:22 So,
13:23 think of a,
13:23 let's say a
13:24 school management or teacher incentive interventions,
13:27 and then you
13:28 kind of like assign some
13:31 schools or some
13:32 classrooms into a certain regime and some to the,
13:35 to another.
13:37 Now,
13:37 people in the
13:39 treatment group,
13:40 if they want the program to succeed,
13:42 they might work harder.
13:44 And people in the control group,
13:45 let's say if they don't want it to succeed,
13:48 they might also work harder to just show that
13:50 the treatment and the control difference isn't that different,
13:52 and vice versa.
13:53 If people want it to fail,
13:55 they may just try to sabotage it and so on.
13:58 So,
13:58 here,
13:58 what's happening is because you can't really do a placebo
14:03 in social sciences,
14:04 you can't just,
14:05 you know,
14:06 people are going to know whether they're in the treatment or the control,
14:09 um,
14:11 Then,
14:11 then that's going to,
14:13 that's going to matter.
14:14 Um,
14:15 and there's even a version of this that's actually very interesting,
14:18 and it was very innovative work,
14:20 where
14:22 You know,
14:22 you have people who are
14:25 in the vaccine group
14:27 or in the placebo group.
14:29 But depending on how they
14:31 Um,
14:33 What they think,
14:34 what they got.
14:35 Let's say
14:36 that you expect that you're going to have some
14:37 side effects when you get the Moderna vaccine,
14:40 because you hear from a lot of people that Moderna does,
14:43 like,
14:43 side effects X,
14:44 Y,
14:45 Z.
14:45 Right?
14:46 So,
14:47 you're going to think,
14:47 OK,
14:48 I'm getting X,
14:49 I'm getting Y,
14:49 I'm getting Z,
14:50 so I must be vaccinated,
14:52 so I can take my mask off and I can go into restaurants.
14:56 And if the outcome is determined both by the intervention and your behavior,
15:02 there's an interaction between
15:03 what you think your treatment status is and what your behavior is.
15:07 So,
15:08 that's going to change things.
15:09 So,
15:10 the way to deal with that is actually,
15:11 you can randomize.
15:14 People and then tell them the likelihood that they're in a group.
15:17 So that
15:18 if people
15:20 perfectly know
15:22 that they are one group,
15:24 and then,
15:25 you know,
15:25 they behave a certain way versus they perfectly know they're another group,
15:29 you can actually tease out the components where
15:32 the effect is due to only the
15:35 Vaccine because these people think.
15:39 They don't have the vaccine,
15:41 but they're behave,
15:42 so they're behaving accordingly,
15:44 and then there's these other people.
15:45 So you can,
15:46 you can do that kind of thing.
15:47 In practice,
15:48 it becomes difficult,
15:49 but these are the kind of theoretical things that you have to think about.
15:52 But nonetheless,
15:53 for you,
15:54 if you're thinking,
15:54 well,
15:55 OK,
15:56 I'm going to do this,
15:57 like,
15:58 pay some teachers for better performance,
16:00 and then there are going to be other teachers who know I'm doing this,
16:03 but they're in the control group,
16:04 you have to really think about how these teachers are reacting.
16:08 Just to the fact of the
16:10 RCT
16:12 rather than
16:13 this is now the new world,
16:14 and this is in which they're going to have these incentive programs and etc.
16:18 So you have to think carefully about those kinds
16:20 of things and whether they're going to affect your,
16:22 uh,
16:23 affect your,
16:24 um,
16:24 findings.
16:26 Right,
16:27 so now,
16:27 let's talk a little bit about what types of randomized experiments you might do.
16:31 So,
16:32 one of the things that we can think about is like,
16:34 the first thing that we think about is access,
16:36 right?
16:37 So,
16:38 There is something that like I can do where
16:41 I can essentially control access to a program.
16:44 Like,
16:44 it's a new program,
16:45 it doesn't exist anywhere.
16:46 There's nothing really like it around,
16:48 you know,
16:49 we can think of it as like new mRNA vaccine or something like that.
16:52 And then what I'm going to do is I'm going to randomly assign
16:55 some people to it and some people to a placebo or something.
16:59 I can do that,
17:00 but I can do it.
17:02 Around a certain cutoff of,
17:04 let's say,
17:04 vulnerability or something like that.
17:06 So you can think of this as
17:08 Um,
17:09 you know,
17:09 elderly people are more vulnerable,
17:11 people with certain conditions are more vulnerable.
17:14 So,
17:14 I'm going to only do it among those people,
17:17 and then I'm going to leave everybody below 65 and no conditions out.
17:22 Right.
17:23 The second thing you can do is
17:25 what are called phase-in designs,
17:26 and there are various um
17:28 versions of these that I'm going to show you in a second.
17:31 Um,
17:32 but you could essentially provide access to the program
17:35 in phases,
17:36 right?
17:37 So,
17:37 this district comes in first,
17:39 this other district comes in later.
17:41 This helps when
17:43 You know,
17:43 programs are new,
17:44 and then there isn't enough administrative capacity to do everywhere all at once.
17:48 And there may also be some learning
17:50 involved.
17:51 So you do just start slowly and then,
17:53 uh,
17:54 scale up,
17:54 um,
17:55 faster.
17:56 So you could do those kinds of things.
17:58 And while you're doing that,
17:59 you can evaluate the people who are phased in versus not.
18:02 If you can randomize those rather than
18:04 selectively decide where you're going to start,
18:07 um,
18:07 you can do that.
18:09 And then there's the encouragement types of designs.
18:12 Um,
18:12 these are
18:13 cases where I can't actually control the access to the treatment.
18:17 So,
18:17 for example,
18:19 um,
18:20 you know,
18:20 I'm going to give you an example later on.
18:22 Like,
18:23 we have a blog,
18:24 uh,
18:24 and I want to know the effect of the blog on people's knowledge on impact evaluations
18:29 or methods or something like that.
18:32 The blog is out there,
18:33 anybody can read it.
18:34 So,
18:34 how do I,
18:35 um,
18:37 How do I,
18:37 um,
18:39 evaluate the effect of this blog?
18:41 Well,
18:41 then I can say,
18:42 well,
18:42 I know that,
18:43 like,
18:43 actually,
18:43 even though this blog is out there,
18:45 not everybody reads it.
18:47 Um,
18:48 I,
18:48 in fact,
18:48 I know that,
18:49 like,
18:49 from my baseline surveys,
18:50 only 10% of the relevant population of graduate students that I care about,
18:56 they are
18:57 reading this blog.
18:58 So,
18:58 what I'm going to do is I'm going to send an email encouraging them.
19:02 And hope that like that will increase the percentage that's reading.
19:05 So now if I randomize the encouragement.
19:08 Then I have a group that's like reading,
19:10 you know,
19:10 10% of them are reading it.
19:12 Then I have another group that 30% of them are reading it.
19:15 Now,
19:15 that 20% point difference I can use to estimate something.
19:19 It's not going to be the same thing that I could estimate
19:22 with uh just randomly
19:24 controlling access,
19:25 but it's going to be still something
19:28 of potential interest.
19:29 And if that's something of potential interest
19:31 that's relevant to the policymakers or academics,
19:34 then you can do that.
19:36 Right?
19:37 So,
19:37 we can think of some examples here.
19:40 So,
19:40 for example,
19:41 one thing is,
19:42 you know,
19:42 one reason or an opportunity to randomize is
19:45 you actually want to address a problem,
19:48 but you don't actually know exactly how to do it.
19:51 So,
19:51 you have,
19:51 let's say,
19:52 23 Options and you want to try them.
19:54 So,
19:54 I think you have your target population,
19:57 and then among them,
19:57 you assign them to these different alternatives to see which one might work best.
20:02 And
20:03 you might do that aesthetically or adaptively.
20:05 I'm going to talk about adaptive experiments a little bit later on.
20:09 Um,
20:10 it could be that you're introducing a new service.
20:12 So in this case,
20:13 there's no
20:14 kind of,
20:14 like,
20:14 uncertainty about what you want to do.
20:17 But you just want to know whether this new service,
20:20 like,
20:20 will people take it up,
20:21 will it be effective,
20:22 etc.
20:23 So you might try to just kind of like,
20:25 um,
20:25 you know,
20:26 invite some of your customers
20:28 to,
20:29 to take this up or not.
20:31 Um,
20:31 you can do new people.
20:33 So,
20:33 for example,
20:33 Oregon,
20:34 um,
20:35 had some more money
20:36 to add people to its Medicaid program.
20:38 So suppose that
20:39 The program was originally targeted to people
20:43 at the poverty line,
20:44 and then now they decided that they can actually
20:46 increase it to 150% of the poverty line.
20:50 And so,
20:50 but there isn't enough money to cover like all of those people initially.
20:55 So what you can do is you can actually randomly in that band of 100 to 150%.
21:00 You can randomly assign the program to some and not the others.
21:04 In fact,
21:04 Oregon,
21:05 literally the public lottery
21:07 among the eligible people,
21:08 and then you can
21:10 wait a few years and then just,
21:12 you know,
21:12 like if you have an RCT you can show the effects and then that can inform
21:16 the policy on like,
21:18 how high to go from the poverty line and how many people to cover if it cost effective,
21:23 etc.
21:24 Right?
21:25 You can do oversubscription,
21:27 uh,
21:27 things.
21:27 So often we have things
21:29 that we want to put out there and then,
21:31 then so many people actually want them.
21:33 So,
21:34 if that's the case,
21:35 then what,
21:35 and we can't,
21:36 we can't
21:37 cover all of that demand,
21:39 we can say,
21:39 well,
21:39 look,
21:39 the most ethical way to kind of like distribute this is by lottery.
21:44 These people are all presumably equally eligible.
21:47 So,
21:48 we are oversubscribed for this,
21:50 like,
21:50 offer.
21:51 Um,
21:52 so we're going to do a lottery and then sign up some people,
21:55 and then,
21:56 because now the selection issue is
21:58 handled,
21:59 right?
22:00 There are people who said,
22:02 I am interested in this program.
22:04 Like,
22:04 let's say this was job training or something like that.
22:07 You don't want to compare people who
22:09 applied for job training,
22:11 and then you randomly selected to some people who didn't
22:14 apply for job training because they're very different people.
22:17 But now you have a,
22:18 with the subscription,
22:20 you have a bunch of people who all self-selected into saying,
22:23 I need job training,
22:24 but you still can't cover all of them.
22:26 So when you randomize,
22:27 you have a really comparable group of people that are actually relevant
22:31 for the program because they all kind of after this program.
22:35 The flip side of it that I hadn't thought about
22:37 before is that sometimes there's things like unders subscription.
22:40 So for example,
22:41 um,
22:42 think about,
22:42 let's say,
22:43 Ukraine or something like that right now,
22:46 you know,
22:46 they actually literally sub,
22:48 you know,
22:48 said everybody
22:50 who is between the ages of 16 or 18 and 50
22:54 has to fight right now.
22:56 Um,
22:56 but when it was happening in,
22:58 you know,
22:59 During the Vietnam War,
23:01 the US didn't need that many people,
23:03 so they wouldn't have said,
23:04 like,
23:04 OK,
23:04 we're having universal,
23:06 you know,
23:06 like a sub conscription.
23:08 But then on the other hand,
23:10 they,
23:10 there wasn't still enough people putting up their hands saying,
23:13 like,
23:13 I want to go fight and put my life in danger.
23:16 So,
23:17 they still needed to do a lottery to actually conscript people into the army.
23:22 And they did that on the basis of people's birth dates,
23:25 uh,
23:25 and things like that.
23:26 And then if you were unlucky,
23:28 um,
23:29 then you had to go fight a war that you may or may not believe in.
23:32 So,
23:32 undersubscription is actually also an opportunity for a lottery.
23:36 You can have rotations,
23:37 like think of Roscas and things like that.
23:40 Um,
23:40 communities or individuals can take turns to be,
23:44 uh,
23:44 the treatment,
23:45 uh,
23:46 unit,
23:46 so to speak,
23:47 while the others are controlled.
23:49 And then you can have admission cutoffs and
23:51 admissions and phases and things like that.
23:54 So,
23:54 let's give a couple of examples here,
23:56 but before I do that,
23:57 like a word about ethics.
23:59 So just because you can randomly assign people to intervention,
24:02 it doesn't mean we should.
24:03 We talked about equipoise,
24:04 so,
24:05 you should have a question
24:07 about which you have a lot of uncertainty,
24:09 and the question is really important.
24:12 Um,
24:13 from a policy perspective,
24:14 like that's one
24:16 barrier.
24:17 The second one is beneficence,
24:18 which is,
24:18 this is about the Belmont principles of like ethics that we all kind of subscribe to.
24:24 We take courses before we can do
24:26 studies,
24:27 uh,
24:27 with human subjects
24:29 or animal subjects for that matter.
24:31 And beneficence essentially means that you're trying to maximize the
24:35 Benefits to the people in your study and minimize the harms.
24:39 And one of the biggest,
24:40 um,
24:41 risks of harm is kind of like privacy concerns.
24:44 And so,
24:45 we,
24:46 we do a lot in our,
24:48 you know,
24:48 like a study designs on both kind of like
24:51 how to do informed consent and things like that,
24:53 etc.
24:54 how to deal with vulnerable populations such as
24:56 children or prisoners or disabled people and etc.
25:00 But then also
25:01 in the data handling,
25:03 uh,
25:03 processes where
25:04 We're anonymizing the data and we're keeping them in very secure,
25:08 like,
25:09 uh,
25:09 servers and,
25:11 you know,
25:11 all of that kind of thing.
25:12 There's protocols for that.
25:14 So I list some resources here and I know you all have my study slide,
25:18 at the slide deck,
25:20 so you can just go to these and,
25:21 like,
25:21 read these,
25:22 um,
25:22 read these uh references.
25:25 So,
25:26 a couple of,
25:26 like,
25:27 study types,
25:27 and I'll give examples of these.
25:29 So,
25:30 for example,
25:31 what you can do is you can do a lottery,
25:33 like,
25:34 which I call these,
25:34 like,
25:35 you know,
25:35 in the industry,
25:36 like,
25:36 let's say Google,
25:37 Facebook,
25:37 etc.
25:38 they call these like classic AB tests.
25:40 So,
25:41 AB literally means like,
25:42 I can do A or I can do B
25:44 in
25:45 the case of something.
25:46 Like Facebook or something like that.
25:48 Should I show the,
25:49 you know,
25:50 advertisements,
25:51 like,
25:51 on the side or like right under,
25:53 or should,
25:54 what the color should be or etc.
25:56 They're constantly doing these experiments,
25:58 right?
25:58 So,
25:58 that's the AB test,
25:59 but they're all kind of static.
26:02 Um,
26:02 let's say,
26:03 meaning that,
26:04 like,
26:04 you're not changing the assignment probabilities or,
26:06 or anything like that over time.
26:09 So,
26:09 Examples of this is like we did a cash transfer experiment in Malawi.
26:14 We gave some,
26:15 we
26:15 chose some villages to be
26:18 conditional cash transfer villages and some villages
26:20 to be unconditional cash transfer villages,
26:22 and some villages to be controlled,
26:24 right?
26:25 What you can do sometimes is that you might say,
26:27 well,
26:28 look,
26:29 Actually,
26:30 it really seems
26:32 very inefficient and also a little bit unfair and potentially demoralizing
26:37 to exclude people who are very suitable for a program,
26:41 but also to include people who are very unsuitable for the program.
26:46 And then we should really randomize only among the people.
26:50 That we're unsure about whether they,
26:53 they are good for it or not,
26:54 and they'll benefit for it or not.
26:56 And this is David McKenzie's Nigeria Business
26:58 Plan competition was an example of this,
27:01 where essentially they had
27:04 A competition where people submitted business plans.
27:07 And at the end,
27:08 there was a quite a big reward.
27:09 Like,
27:09 I don't remember if it was $50,000 or $250,000 or something like that.
27:15 But,
27:15 you know,
27:16 big reward at the end.
27:17 So what they did is like,
27:18 they had an independent panel
27:21 anonymously score all of these proposals.
27:24 And then let's say if you got like,
27:26 between 90 and 100,
27:28 you were given the award,
27:30 right?
27:30 And then let's say if you got below 50,
27:33 Then you were not given any award.
27:35 But the people between 50 and 90,
27:38 they weren't quite sure,
27:39 like,
27:39 they were decent proposals,
27:41 but,
27:41 you know,
27:41 they could be successful,
27:43 they could be seemed a little risky.
27:45 So,
27:45 among those,
27:46 they actually randomized
27:48 the,
27:48 um,
27:49 the intervention,
27:50 and then that's,
27:51 that's the basis of the evaluation.
27:53 So,
27:53 now,
27:53 what are you doing?
27:54 You're evaluating
27:56 uh
27:57 the effect of this business plan competition
28:00 for
28:01 Firms or like entrepreneurs
28:04 who are kind of marginal,
28:06 and by marginal,
28:07 I mean
28:08 They could go either way,
28:09 like the an independent panel didn't decide that,
28:12 like,
28:12 no,
28:12 this business is definitely not going to succeed,
28:15 this proposal is terrible,
28:17 or
28:18 this proposal looks amazing,
28:19 they already have some like proof of concept,
28:21 whatever,
28:22 etc.
28:23 So you've,
28:23 you're excluding those,
28:24 your evaluation isn't going to say anything about those kinds of things,
28:28 like whether giving them
28:29 $50,000 or $250,000 is
28:33 Effective or not,
28:34 but it's going to say something about this
28:36 marginal group and that's still very valuable.
28:39 So,
28:39 you can do something like that.
28:41 I like those designs a lot actually,
28:43 because they,
28:43 they take into account some targeting mechanism where
28:46 you're being a bit more ethical rather than
28:49 kind of like randomizing everybody where some people may
28:52 definitely need,
28:53 I mean,
28:53 if you think about,
28:55 instead of business plan,
28:56 you're thinking about a poverty intervention.
28:58 Now you're talking about
28:59 much more
29:00 Uh,
29:01 potentially,
29:02 uh,
29:02 devastating consequences for being in the treatment versus the control,
29:06 right?
29:07 Phase-in designs,
29:08 um,
29:09 the most famous example of this is the Worms
29:11 paper of Miguel and Kramer from Econometrica in 2004.
29:16 And this was the,
29:17 the famous deworming study in Kenya in the Busia district,
29:21 where
29:22 they had 75 schools and they divided them into three groups,
29:25 Group A,
29:26 B,
29:26 C,
29:26 25 schools in each group.
29:29 And essentially what they said is that in year 1,
29:31 we're going to treat Group A,
29:33 and then in year 2,
29:34 we're going to treat
29:36 Group B,
29:37 and then in year 3 onwards,
29:38 like everybody will be treated.
29:41 Now,
29:42 for the,
29:43 at the end of the first year,
29:45 you can do an evaluation where you're comparing treatment group one to treat,
29:49 you know,
29:49 the A to B and C.
29:51 In year 2,
29:53 you can actually compare A and B
29:55 to C,
29:56 but then you have to take into account the fact that
29:59 A is in the treatment longer and maybe that has some effects,
30:03 right?
30:03 And from year 3 onwards,
30:06 you can't really do,
30:08 They kind of like the comparison anymore other than
30:11 it's not a pure comparison other than now
30:14 the only difference between these groups is that
30:17 they've been exposed for different durations.
30:20 Now,
30:20 there's one exception to that,
30:22 which is that
30:23 if you actually think that
30:25 You're doing things for cohorts inside the schools,
30:29 and there are going to be cohorts that just miss out and they never get treated.
30:33 So,
30:33 suppose that we say,
30:35 anybody above grade 8
30:37 is no longer eligible for deworming.
30:39 This is not true for deworming,
30:41 but I'm just hypothetical.
30:43 Um,
30:44 then what you can do is,
30:45 like,
30:46 the people who were in grade 8 in the comparison group
30:50 in B and C.
30:52 They will never get treated
30:54 going forward.
30:55 The,
30:55 the grades 8 the next year and the year after that will get always treated,
31:00 but they won't.
31:01 So you can take grade 8
31:03 in Group A versus B and C,
31:06 and forever,
31:07 you have a pure
31:09 Kind of like RCT where the control group has never been treated,
31:13 right?
31:14 So,
31:14 if that's possible,
31:15 and if there is,
31:16 if that's actually legitimate,
31:18 meaning that
31:19 maybe there's effects on these grade 8 people from the fact
31:22 that they're like 1 year younger people around them are getting
31:26 treated and then there's spillovers and etc.
31:28 then it's still not great,
31:30 but you can think of it that way.
31:32 But those are the phase and designs and
31:34 The reason why people don't like phases in design,
31:37 some people at least,
31:38 is because it's hard to do long term evaluations,
31:42 because for,
31:42 think of my study in Tanzania,
31:45 where we had people apply for business,
31:47 um,
31:49 Kind of like income generating activities
31:51 at the group level,
31:53 and
31:54 there are really no cohorts,
31:55 like comparable cohorts or anything.
31:57 They're like 100 villages,
31:58 in 100 villages,
31:59 there are 100
32:01 groups that applied for something.
32:03 And we said to 50 of them,
32:04 you will get your money now.
32:06 And then the other 50 of them,
32:08 we said,
32:08 you'll get your money next year,
32:10 right?
32:10 So we really have 1,
32:12 after 1 year,
32:13 they'll get a huge infusion of money.
32:15 And it's hard to argue that,
32:17 like,
32:17 at,
32:18 after that point,
32:19 there is this kind of any difference,
32:21 really,
32:21 other than the fact that one group had their money 1 more year.
32:25 Right?
32:25 And so,
32:26 you essentially have only one year
32:29 to kind of evaluate the effect of the program.
32:31 And if you don't think that you're going to see
32:34 the effects that you're hoping to see in that duration,
32:37 then a phase-in design isn't really good for you.
32:40 And then even,
32:40 even if you do,
32:42 it's,
32:42 you know,
32:43 it,
32:43 it's
32:44 People may say,
32:44 well,
32:45 actually,
32:45 I would really like to know what happens like 4 years later,
32:48 maybe they converge back or something like that.
32:50 You don't really have an opportunity to see that because you essentially have this
32:55 group,
32:55 comparison group has been treated.
32:58 The encouragement design,
33:00 I mentioned it already,
33:01 like the impacts of economics blogs,
33:03 David McKenzie and I like wrote a paper on this and published
33:06 it because we essentially did exactly what I described to you.
33:10 We wrote an email to
33:12 like 300 students and their advisors at the economics department saying,
33:16 hey,
33:16 we have this new blog,
33:17 like,
33:18 why don't you check it out?
33:19 And then 300 others that we had listed before and randomly excluded,
33:24 we didn't.
33:25 And it did really cause a difference in how many people do report reading our blog,
33:30 and then we could use that difference to just kind of um that differential exposure
33:35 to evaluate whether people
33:37 knew about development issues a little bit better by reading our,
33:41 um,
33:42 by reading our um blog.
33:45 But
33:45 note here is that like,
33:47 you're not any longer
33:49 evaluating the intention to treat effect,
33:51 but you're evaluating the,
33:53 what is called the local average treatment effect.
33:56 And how it's defined is that that's essentially you're
33:58 evaluating the effect of the program on people.
34:02 Who would not have
34:05 taken up the intervention,
34:06 the program,
34:07 in our case,
34:08 read our blog
34:09 on their own,
34:11 i.e. without the intervention,
34:13 but would have read it
34:15 if we intervene,
34:16 right?
34:17 That's a very specific group,
34:19 but that may be exactly the kind of group that you're interested in.
34:22 So for example,
34:22 if you're New York Times,
34:24 and you're saying,
34:26 should we advertise because we have like,
34:28 you know,
34:28 we have huge readership,
34:29 but it could be even bigger.
34:31 And what if we advertise,
34:33 which is kind of like our emails to people,
34:36 then
34:36 it will get them to sign up or not,
34:39 right?
34:39 And so,
34:40 that's exactly the group that you're interested in,
34:43 the people who would not have read the New York Times on their own,
34:47 but if you advertise,
34:48 they will.
34:49 And you want to know how big that group is,
34:51 and then what their characteristics are,
34:52 and like,
34:53 whether they'll pay,
34:54 how much they would like to pay to sign up and all of that.
34:57 So,
34:58 that
34:59 That might be the
35:00 estimate of interest,
35:02 but it may not be.
35:03 So depending on that,
35:04 you can do these encouragement designs.
35:06 The encouragement designs need to satisfy instrumental variables,
35:10 uh,
35:11 criteria,
35:12 which is that
35:13 They need to actually,
35:15 your encouragement
35:17 should only work through people taking
35:19 up the intervention and shouldn't directly affect
35:23 the,
35:23 um,
35:24 the outcome
35:25 itself,
35:26 right?
35:27 So you could imagine,
35:27 for example,
35:28 that if in our blog,
35:31 Email.
35:32 We had also given people some information.
35:34 We said,
35:35 like,
35:35 not only we said,
35:36 oh,
35:36 here's a new blog,
35:37 but also your other choices of reading things are like Chris Plattman
35:41 and uh Paul Krugman and this other blog and Tyler Cowen,
35:44 etc.
35:45 Um,
35:46 then all of a sudden,
35:48 We can't really use it anymore to isolate the effect of its effect on
35:53 because now we've essentially given people
35:55 like a bunch of other information that could influence their
35:59 knowledge.
36:00 So now,
36:01 we violated one of the,
36:03 um,
36:03 you know,
36:05 necessary conditions for an instrumental variable to work,
36:07 which is the exclusion criteria.
36:09 So,
36:10 you need to essentially think about
36:12 Is my encouragement just really only going to work through the intervention.
36:17 So,
36:17 for example,
36:18 think of the kinds of things like David McKenzie and
36:20 his colleagues at the finance group do with like banks.
36:24 Banks will just,
36:25 let's say,
36:26 send a mail,
36:27 like even like a postcard to their clients saying,
36:30 we have a new credit card,
36:31 like 3% interest rate or something like that,
36:34 like,
36:34 please,
36:35 you know,
36:35 it's hard to think that that's going to affect your finances,
36:40 seeing that.
36:41 You know,
36:42 other than you actually apply and like,
36:44 get the credit card.
36:46 Um,
36:46 so,
36:47 that's the kind of thing you want to be doing,
36:48 very light touch encouragement that hopefully works to
36:51 increase substantially the takeup of the program,
36:55 but it doesn't affect anything else.
36:59 So,
36:59 another version of phase-in is this kind of like step wedge design.
37:03 These are very popular in medicine.
37:05 So,
37:05 for example,
37:06 this is how the Gambia famous hep B vaccine
37:09 trial was done.
37:10 You can see here that the light blue areas are
37:13 areas where like people are waiting that many
37:16 weeks to be exposed to the intervention.
37:18 And then
37:19 other,
37:20 you know,
37:21 the dark blue is the one that you're actually now exposed to the intervention,
37:25 and you can essentially compare the people who
37:27 are in there versus who are not in there
37:31 throughout the,
37:32 the transition period.
37:34 But um,
37:35 You have to worry about two things.
37:37 One is time trends.
37:38 So,
37:38 for example,
37:39 some intervention may work better
37:42 um
37:42 now than
37:44 it,
37:44 it,
37:44 it will in like 3 weeks' time,
37:47 mainly because,
37:48 you know,
37:49 potentially,
37:50 maybe the government says like mask mandates or something like
37:52 that and all of a sudden vaccines aren't that important,
37:56 as opposed to,
37:57 it's a free for all and the government says there's no reason to worry.
38:01 Everybody can go anywhere they want.
38:03 And then maybe vaccines are much more important then.
38:06 So,
38:06 there's those kinds of things to worry about.
38:08 But then there's also anticipation effects,
38:10 and this was actually something that we
38:13 think
38:14 really,
38:15 um,
38:16 derailed are like the Tanzania work that I
38:19 talked to you about,
38:20 which is that,
38:22 you know,
38:22 you're saying to people,
38:23 well,
38:23 you'll get your money next year.
38:25 And we essentially thought that,
38:27 well,
38:27 this is an ultra poor program.
38:29 These people are very credit constrained.
38:31 So,
38:31 there isn't really much that they can do
38:34 in anticipation of getting the money next year.
38:36 But actually,
38:38 that turned out maybe that we have some suggestive
38:40 evidence that that may not have been true,
38:43 that people can actually say,
38:44 well,
38:46 You know,
38:46 I'm getting this big amount of money next year.
38:49 So,
38:49 and it's a credible promise.
38:52 So I can literally borrow against that promise.
38:55 Like,
38:55 it's almost becomes my,
38:56 like,
38:57 collateral,
38:58 and I can start my business
39:00 things,
39:01 like,
39:01 now.
39:01 Now
39:02 in anticipation,
39:03 or maybe I'm going to eat more or eat less or whatever.
39:06 There's all sorts of things that I can do in anticipation,
39:09 all of which
39:10 kind of bias the effects because if people didn't know
39:13 that they were going to be treated next year,
39:15 they would have acted differently.
39:18 And the final thing that I like this a lot,
39:20 and this is actually also used in medical trials,
39:23 and this is called a fried egg design,
39:25 and then you can look here at the bottom,
39:27 the kind of like the top left corner.
39:30 You know,
39:30 you,
39:30 this blue area here is a,
39:33 is a cluster,
39:34 it's a treatment cluster,
39:36 so everybody in that blue area is assigned to,
39:39 let's say,
39:39 the vaccine.
39:41 And then everybody in the brown area is assigned to placebo,
39:45 right?
39:46 But the brown area and the
39:48 um
39:49 blue area are adjacent to each other,
39:51 and I worry that there's going to be spillovers here.
39:54 So,
39:54 what I do is
39:56 I actually take the yolk of the egg,
39:58 so to speak,
39:59 and this is the kind of the white parts of the egg,
40:01 they are fried egg.
40:03 And then I only study the people in the yolk,
40:06 because they are only surrounded by people who are also vaccinated,
40:10 and these people are only surrounded by people
40:13 who are
40:14 unvaccinated.
40:15 So,
40:16 then,
40:16 essentially,
40:16 my assumption is that these people and these people
40:20 are not going to interact with each other at all,
40:22 so I'm going to get a really pure estimate.
40:25 Um,
40:25 rather,
40:26 or,
40:26 or their
40:28 amount of spillover on each other is going to be much,
40:30 much less than these people.
40:32 So,
40:32 if I studied them also,
40:34 um,
40:35 that would,
40:35 that would have been a problem,
40:37 right?
40:38 So,
40:39 And that brings me actually to kind of
40:41 like this idea of clustered versus individual randomization.
40:45 And then we talked about the stable unit treatment
40:48 violation uh value assumption last time or the Suva.
40:52 You know,
40:53 so remember that aspirin example that we were talking about last week
40:58 or two weeks ago,
40:59 you know,
41:00 does my headache depend on also you taking an aspirin?
41:03 Probably not.
41:05 But maybe,
41:05 like,
41:06 if actually,
41:07 you know,
41:07 we live together and you just become an awful person,
41:10 like when you have a headache,
41:12 then you start giving me a headache.
41:14 Maybe you taking an aspirin helps me get it,
41:17 you know,
41:17 like,
41:18 not get a headache or not.
41:19 So,
41:20 we have to always think about.
41:22 Is there
41:24 Is there any possibility of interference
41:27 from the treatment units to the control units?
41:30 And if so,
41:31 how do I deal with that?
41:33 And I think this is a big enough problem,
41:36 especially when you're trying to just kind of
41:38 publish papers.
41:39 Now,
41:39 referees are very attuned to this,
41:41 like,
41:41 you will always get these questions,
41:43 and it's,
41:44 um,
41:45 you know,
41:46 We do see
41:47 papers getting,
41:48 you know,
41:48 good papers getting
41:50 kind of like knocked and,
41:52 you know,
41:52 rejected because they can't really,
41:54 they've done an individualized randomization or something like that,
41:57 so they can't really rule out that there's no spillovers.
42:01 Um,
42:01 and so it
42:03 ends up in,
42:03 you know,
42:04 you have a perfectly good RCT at the kind of like the individual level,
42:08 you've done everything correctly.
42:09 But now people are saying,
42:11 hey,
42:11 how do I know that like,
42:12 that person and that person aren't friends,
42:14 and then they're like teaching each other the things they learned,
42:17 whatever.
42:18 And that's,
42:19 that,
42:19 that becomes a problem.
42:20 So I think it's important to think about these things.
42:23 In many cases,
42:24 there are where Suwa is violated,
42:26 like in a pandemic,
42:28 in agriculture where wind is going to take like your,
42:31 you know,
42:32 seeds and the sprays and things like that across.
42:35 So you have guard rows and agricultural experiments.
42:39 You can have general equilibrium effects.
42:41 So,
42:41 the more people I do job training with,
42:44 the more spillover effects they will have on
42:46 people who are in the control group because they
42:49 will get jobs and then they'll take the
42:51 jobs that those other people could have gotten.
42:53 So,
42:53 if there's actually a
42:55 Limited amount of,
42:56 like,
42:57 let's say,
42:57 winner take all kind of reward,
43:00 and I'm favoring one group over the other,
43:02 all I could be doing is causing displacement and then putting people,
43:05 some people in front of the queue rather than the back of the queue.
43:08 And so that's uh interference,
43:11 or in classrooms and in schools and things like that,
43:14 if I'm like,
43:15 let's say,
43:15 tutoring some people,
43:17 then they could maybe going home,
43:18 like,
43:19 and then,
43:19 you know,
43:19 helping their friends and etc.
43:21 who are in the control group.
43:23 So,
43:23 you can think of many cases like this.
43:26 So,
43:26 I think
43:27 what you want to do,
43:30 generally speaking,
43:30 if you're worried about this,
43:32 you want to
43:34 Think of like some clusters where that interaction is minimized.
43:38 So,
43:38 let's say villages in our Malawi case were kind
43:41 of like 5 kilometers far from each other.
43:44 And towards the end of the intervention,
43:46 we actually,
43:46 you know,
43:47 actually,
43:47 after the intervention finished 2 years later,
43:50 when we were talking with some
43:52 You know,
43:53 people in the control group in their follow-up surveys,
43:55 we were really happy to see that actually,
43:58 they had no idea
44:00 that there was a cash transfer program going on in other parts of the same district.
44:04 Um,
44:05 but then on the other,
44:06 uh,
44:06 uh,
44:06 and,
44:07 you know,
44:07 for good reason,
44:08 because if they know,
44:09 then there could be,
44:10 you know,
44:10 mental health effects,
44:12 they could be like,
44:12 oh,
44:13 you know,
44:13 compliance issues and
44:15 things like that.
44:16 They could be asking for money from treated people and so on and so forth.
44:20 So,
44:21 You want to do kind of like sufficient.
44:24 Distance between your subjects,
44:27 whether that distance is social or geographic or otherwise.
44:31 Um,
44:32 and then I think you run into a
44:35 Like a
44:36 main trade-off you have is power,
44:39 because individually randomized,
44:41 uh,
44:41 trials are very powerful because you essentially have
44:45 each person randomized,
44:46 so the statistical power is just much higher.
44:50 And the power loss comes from the fact that within clusters,
44:53 you have a lot of correlation.
44:55 So,
44:56 think about the,
44:57 Case where
44:59 think of households
45:00 as
45:01 clusters,
45:03 right?
45:03 But then think about something that's very,
45:05 very correlated within a household.
45:07 So,
45:07 like COVID actually now is the Omicron is a good example.
45:10 I know many people,
45:11 once their kid comes back from school and then has COVID,
45:14 the whole family gets COVID very soon,
45:17 right?
45:17 So,
45:18 in that case,
45:20 It's not going to help me to sample 2 or 3 people from that household.
45:24 It's going to give me zero extra power if
45:27 everybody is going to get COVID in that household.
45:29 Like,
45:30 it's perfectly correlated,
45:31 then all I need from that cluster is just one person,
45:35 right?
45:36 But that means,
45:36 what that means for my power calculations is that I need many,
45:40 many more clusters to get,
45:41 like,
45:42 good,
45:42 good inference,
45:43 right?
45:44 Whereas if things aren't that correlated,
45:47 You know,
45:47 let's say it's actually cancer that's not environmentally related,
45:50 but like otherwise.
45:52 And,
45:52 you know,
45:53 so then I think,
45:54 you know,
45:55 you can have 3 different people from the same household,
45:58 their outcomes aren't that correlated with each other,
46:01 that's fine.
46:02 So then you,
46:02 you,
46:03 you just still get benefits from,
46:04 like,
46:05 increasing the cluster size
46:07 rather than increasing the number of clusters.
46:09 And that's essentially the main
46:11 calculation that you're making when you're trying to do individual versus
46:16 Um,
46:16 individual versus clustered
46:18 design.
46:21 So we.
46:23 And so now,
46:24 I want to talk a little bit about factorial designs.
46:28 So,
46:29 What that means is like,
46:30 so some people also call these 2 by 2,
46:32 if there's 2 interventions that you're thinking about,
46:35 but if you,
46:36 there's 3,
46:36 it could be a 3 by 3,
46:37 whatever,
46:39 right?
46:39 So,
46:39 essentially,
46:40 you're thinking about,
46:42 I have,
46:42 let's say,
46:44 Um,
46:45 cognitive behavioral therapy for,
46:48 uh,
46:49 troubled youth,
46:50 right?
46:51 Um,
46:51 there's a recent paper that came out now some by some of my colleagues,
46:55 and then you're thinking,
46:56 well,
46:56 actually,
46:58 Therapy for troubled youth is great to reduce like
47:03 uh crime and,
47:04 you know,
47:05 to reduce their like behavioral issues,
47:07 but
47:09 They don't really have the bandwidth to,
47:11 like,
47:11 internalize all of that therapy and to deal with their lives because they're so poor
47:16 that
47:17 they have to just be,
47:18 you know,
47:18 like,
47:19 either stealing stuff or they have to be constantly worried about,
47:22 like,
47:23 you know,
47:23 where the next meal is going to come from and so on and so forth.
47:27 So this program would be much more effective.
47:30 If we just gave them like a little stipend also along with it,
47:33 so.
47:34 So,
47:34 that's your hypothesis.
47:35 Your hypothesis is that if I combine these two things,
47:39 it's going to work much better
47:41 than if I didn't combine them.
47:44 Now,
47:44 a lot of people,
47:45 what they will do is that they will cross them fully,
47:48 meaning that you have a group that's getting neither intervention.
47:51 You have a group that's getting the cash,
47:53 you have a group that's getting the,
47:55 the CBT and you have a group that's getting both,
47:59 right?
48:00 In those kinds of things,
48:02 you know,
48:02 I think so the way I described it,
48:05 like,
48:05 my intention really was like,
48:07 is CBT better
48:09 without cash?
48:09 So,
48:09 what I really care about is like,
48:12 CBT plus cash versus CBT.
48:15 It's just possible that some other people might say,
48:18 I'm actually also interested in CBT plus cash versus cash,
48:22 right?
48:23 Uh,
48:23 in one of the studies that I did that I'm linking here,
48:26 the,
48:27 we were working with the International Rescue
48:29 Committee in Liberia for adolescent girls.
48:32 They said,
48:32 actually,
48:33 we're not interested in that question at all because
48:35 we're never gonna give just cash to people.
48:38 Um,
48:39 you know,
48:39 it,
48:39 at least this was like 5,
48:41 10 years ago,
48:42 you know,
48:42 it's like,
48:42 that's not in our programming currently.
48:44 What we really do want to know though,
48:46 is that if we add,
48:47 let's say 50 cents per like attendance to our mentoring program,
48:51 will it work much better?
48:52 So,
48:53 we just want to compare that.
48:54 So,
48:54 for that purpose,
48:55 I'm going to discuss this in a second,
48:57 we actually did
48:59 Only 3 groups.
49:00 So,
49:00 we had a control group,
49:02 we had the mentoring program,
49:03 and then we had the mentoring program that's like,
49:06 conditional cash for attending the mentoring sessions.
49:09 So,
49:10 like,
49:10 each time you,
49:11 uh,
49:11 you know,
49:11 attend one of the 40 sessions,
49:14 you get,
49:14 like,
49:14 $2 or something like that.
49:16 So over the year,
49:17 that adds up to some money.
49:18 Yeah.
49:19 But
49:19 generally speaking,
49:20 you have these 2 by 2s.
49:23 And
49:23 what happens is that if you have these 2x2s,
49:26 you have to design and analyze them correctly.
49:28 And David McKenzie wrote a really nice,
49:31 um,
49:32 uh,
49:33 kind of,
49:33 um,
49:34 blog post on this like a while back,
49:37 and I covered it again recently.
49:40 So,
49:40 most of the time,
49:42 there's the long model up top,
49:43 which is that you have to have a dummy variable to analyze these,
49:47 like interventions
49:49 for each cell,
49:50 right?
49:50 So you have the,
49:51 the T1,
49:52 let's say your CBT T2 is the cache and then T1 and times T2
49:57 is the CBT and cache,
49:59 and then what you're comparing them to is the control group.
50:02 But often,
50:04 what people will say,
50:05 and this is kind of like where things go really wrong,
50:09 is that people will say,
50:10 well,
50:10 actually,
50:11 That
50:12 thing that we added
50:14 really didn't do anything.
50:15 That B3 is kind of like insignificant and small.
50:19 So,
50:19 what we're gonna do is,
50:21 it'll be much more power if we can take those people in that cell
50:25 and then reallocate them into their T1 and T2 bins,
50:29 and then we just run this short model,
50:31 which is the second thing over here.
50:33 And then
50:34 that'll give us like much better,
50:35 much better idea.
50:37 Essentially,
50:38 there's a couple of things that are wrong with that.
50:41 One of them is like,
50:42 actually,
50:44 What that's doing is now,
50:45 it's averaging,
50:46 let's say this T1 is CBT,
50:49 it's averaging it
50:50 in a world where
50:52 people aren't giving cash.
50:55 And then in a world where they are given,
50:57 let's say some significant amount of cash,
50:59 and
51:00 the average of that is,
51:02 may not be of interest really to the policymaker.
51:05 The policymaker is either going to be in one of those worlds or the other,
51:08 but not,
51:09 not the kind of like the convex combination of it,
51:11 whatever weights you put on that.
51:13 Right?
51:14 Um,
51:14 so,
51:15 that's,
51:16 that's one problem,
51:17 like,
51:17 it's just,
51:18 especially if this was a more eclectic thing that you were doing,
51:22 you know,
51:22 there's no way,
51:23 let's say you're going to be giving cash,
51:25 but also like some people are going to be in like government provided therapy.
51:29 and some people aren't,
51:30 you know,
51:31 that just sounds very silly.
51:32 So that's number one.
51:34 But then there's a second problem,
51:35 which is a statistical problem,
51:37 like the,
51:38 the people who wrote this paper,
51:40 like Mura Idaran and his colleagues,
51:42 uh,
51:42 that I,
51:43 that I kind of linked to here,
51:45 um,
51:46 they show that
51:48 When that B3 that you said is insignificant,
51:51 even if it's modestly positive,
51:54 like,
51:54 or modestly different than 0.
51:58 You get false rejection rates,
52:00 meaning that like you're essentially
52:02 saying that K1 is significant when you shouldn't actually be saying that,
52:07 right?
52:07 So the,
52:08 the explanation of that gets technical,
52:10 but that's essentially the,
52:11 so,
52:12 when you design a factorial design,
52:14 you really need to then evaluate it correctly like this,
52:18 right?
52:18 Or you can design it differently,
52:19 which I'm going to talk about in a second.
52:22 There's a bonus point here that's just kind of more general,
52:26 like,
52:26 it's important what the statistical power of your study is,
52:30 because suppose that you have a very low powered study,
52:34 right?
52:34 But then you're still getting
52:36 things that are very close to significant but not quite.
52:40 Like,
52:40 that actually can be evidence
52:43 against the null,
52:43 meaning that
52:45 You actually a P value of let's say 0.11 or 0.17,
52:49 which we would normally say is that,
52:51 oh,
52:51 this is not significant,
52:52 because your study is so low powered,
52:55 it's actually very unlikely that you would get that
52:58 P value if there was nothing going on.
53:00 So that can be actually evidence for rejecting the null hypothesis,
53:04 even though
53:05 traditionally you would say,
53:06 I don't reject.
53:08 Right?
53:09 Conversely,
53:10 if you have a very,
53:11 very powerful study,
53:12 suppose that you have like millions of observations or something like that,
53:15 and then you're getting a P value of 0.04,
53:19 now actually,
53:20 that can be such that like in that data set,
53:23 that's actually
53:25 More likely to be evidence for the null hypothesis that there's nothing going on,
53:30 rather than
53:31 there's something going on.
53:33 Because you should really be getting P values of 0.00,
53:36 you know,
53:37 0.004 rather than 0.04.
53:40 So,
53:40 it's important to always think about kind of like,
53:43 what's the power of your study and generally speaking,
53:46 design studies that are
53:48 Appropriately powered and not underpowered because that's when
53:51 you're going to be finding these kind of like,
53:54 create more likely to be finding these crazy results because
53:57 the only things that you'll find are very big effects.
54:00 So,
54:00 you'll just be saying,
54:01 oh,
54:02 look,
54:02 at
54:03 therapy is like the best thing in the world,
54:05 like,
54:06 and,
54:06 but you have like 7 observations and then
54:09 that,
54:09 that's what it's coming out of,
54:11 right?
54:11 Just by chance.
54:13 Um,
54:14 so,
54:15 You can
54:16 You can leave the interaction cell empty,
54:19 that's another option.
54:20 So if you're interested in 2,
54:21 you know,
54:22 in my example in Liberia,
54:24 we left the cash cell empty,
54:26 right?
54:27 But you can also leave the,
54:29 you know,
54:29 you can say,
54:30 I'm going to compare cash to the therapy,
54:33 and then I'm not gonna have cash plus therapy,
54:35 right?
54:36 More unconventional,
54:38 you can also choose not to have a control group.
54:40 You can say,
54:40 look,
54:41 I know CBT works.
54:43 I know cash works.
54:45 All I care about is like,
54:47 does cash plus CBT
54:49 have this multiplicative effect that nobody's thought about,
54:53 so I'm gonna compare those.
54:55 To,
54:56 like,
54:56 to the,
54:57 each of these single interventions.
54:59 I don't really need to show the efficacy of these things against the control group.
55:03 People don't like those,
55:04 mainly because in a lot of fields,
55:06 there's not really very well established
55:09 um evidence that these things work.
55:11 And then also,
55:12 even if there is,
55:13 people will say,
55:14 well,
55:15 how do I know that like,
55:17 it wouldn't have been like no effect in this setting.
55:20 So,
55:20 like,
55:20 it's,
55:21 you
55:21 always want to have it there.
55:22 So it's a bit risky.
55:24 Um,
55:25 but,
55:25 you know,
55:25 you could,
55:26 you could consider doing it.
55:27 You don't have to have,
55:28 always have to have
55:29 a 2x2 or a 3x3
55:32 when you're
55:33 thinking of multiple interventions.
55:36 So the third thing that I want to talk about,
55:38 which is,
55:39 this is something that you may not hear
55:40 about like elsewhere and it's interesting to hear about
55:43 is adaptive experiments,
55:45 right?
55:46 So,
55:46 so far what we've discussed are static designs and what do I mean by static design?
55:51 Like I have 10 of you guys there.
55:53 I want to know something about you,
55:55 how you will react to a certain thing.
55:57 Let's say like,
55:58 we only invite 5 of you to this lecture and then 5 of you not,
56:02 right?
56:03 So,
56:03 what
56:04 happens is that
56:05 all 10 of you
56:07 I just
56:08 randomize once.
56:10 5 of you are here,
56:11 5 of you are not here.
56:13 I run my experiment.
56:14 It's the length of this,
56:16 uh,
56:16 lecture,
56:17 and then,
56:18 like,
56:18 you know,
56:18 22 weeks later,
56:20 I do a survey,
56:21 and that's it,
56:22 right?
56:23 If it was an adaptive trial,
56:25 I could think of saying,
56:27 well,
56:27 actually,
56:28 I'm going to take two of these people.
56:30 I'm going to randomize them.
56:32 Um,
56:33 into
56:34 You know,
56:35 like,
56:35 uh,
56:36 let's say,
56:37 you know,
56:37 this lecture,
56:38 Berke's lecture,
56:39 David's lecture,
56:40 and like,
56:41 you know,
56:41 nothing or something.
56:42 I'll,
56:42 I'll take 3 of you,
56:44 right?
56:45 Then,
56:45 based on that I'm gonna survey you.
56:48 And then based on that,
56:49 I'm going to invite the next 3 people,
56:52 and then,
56:53 but I'm going to now change the assignment probabilities
56:56 because it looked like actually David's lecture
56:59 was doing much better than Berg's lecture,
57:01 so I'm gonna assign more people to David's lecture and less to Berke's lecture,
57:05 right?
57:06 And then,
57:06 I'm gonna accumulate that evidence and then assess it again,
57:10 and so on and so forth until I run out of people.
57:13 Right?
57:14 So,
57:15 you can see that like this requires a couple
57:18 of things that are different than static experiments,
57:20 and that's why they're not so common in social sciences,
57:23 although
57:24 they are more suitable definitely for medical context.
57:27 One of which is that
57:29 I need to have
57:31 Access to either the outcome
57:34 or a very good proxy of the outcome,
57:37 very quickly
57:39 before
57:39 I have to enroll the next batch of people,
57:42 because I want to evaluate this batch,
57:44 and then I want to change my assignment probably.
57:48 So,
57:49 for example,
57:49 in my
57:50 um
57:52 family planning work,
57:54 what I care about is like,
57:55 are people who say they don't want to get pregnant,
57:58 getting pregnant within a year,
58:00 so that a year is too long.
58:02 But I have something where
58:04 there's a,
58:05 you know,
58:05 they adopt,
58:06 I can see immediately,
58:07 like,
58:08 literally at the end of my counseling session,
58:10 whether they adopt a long-acting contraceptive method like
58:13 the IUD or the implant or not.
58:15 Now,
58:16 I know that those two things are 99.9% effective
58:20 in preventing pregnancies.
58:22 So,
58:22 essentially,
58:22 I have an extremely good proxy of,
58:24 like,
58:25 these people who have this implant in their arm.
58:28 They're not going to get pregnant next year.
58:29 So,
58:30 even though
58:31 I don't have the outcome
58:33 in hand,
58:33 I have a very good proxy,
58:35 so I can say,
58:35 actually,
58:36 I'm gonna use that proxy to update my
58:38 success probabilities,
58:40 um,
58:41 because I can see that.
58:42 That's #1.
58:43 Number 2,
58:45 It's more suited to contexts where
58:47 there's enroll,
58:49 enrollment is like continuous.
58:51 So think about like a hospital,
58:53 in my case,
58:54 the family planning unit and the maternity unit.
58:56 Women are constantly coming in every day with like requests or deliver a baby,
59:01 or like the,
59:02 you know,
59:02 have miscarriages and vaccinations for their children,
59:05 etc.
59:07 Every day,
59:08 I have a new opportunity to enroll like 1015,
59:11 20 more people.
59:12 So,
59:13 essentially,
59:14 I can say,
59:15 this week is a batch,
59:16 next week is another batch,
59:18 and then at the end of each week,
59:19 I'm just going to quickly analyze the data
59:21 and then change the assignment probabilities next week.
59:24 Like,
59:24 if you don't have that,
59:26 this option is not completely closed to you,
59:29 but you have to have a big enough sample.
59:33 And then also worry about these kind of like anticipation effects
59:36 and all this kind of spillovers and things like that,
59:39 where,
59:39 for example,
59:40 our colleague Anya Sottman did is that they had these schools um
59:44 in the country that they're working in,
59:46 and they had an intervention,
59:48 and they just divided that school
59:51 population into two.
59:52 And then batch A,
59:53 half of the schools,
59:55 and then they,
59:56 they essentially experimented
59:58 with half of the schools in batch A.
1:00:00 They assess that data after 3 months,
1:00:03 they learned something from it.
1:00:04 And then with what,
1:00:06 equipped with what they learned,
1:00:08 then they went to batch B,
1:00:10 and then they redid the experiment there.
1:00:13 So that's a version of this.
1:00:15 It's less continuous version of it,
1:00:16 and you're a little bit more uh limited because maybe
1:00:20 you have only like 500 schools and that's all you can do.
1:00:22 Whereas,
1:00:23 with my medical examples,
1:00:26 I can have like infinite many people
1:00:28 enrolling into my study,
1:00:30 so I can do kind of like better power stuff,
1:00:33 maybe with the exception of very rare diseases,
1:00:36 like,
1:00:36 so I have colleagues who work at the National Institutes of Health,
1:00:39 and they're working on things like
1:00:41 Very rare brain diseases,
1:00:43 like around the world,
1:00:44 there's like 47 people or something like that who have this.
1:00:47 So those are hard and they require coordination across the,
1:00:51 you know,
1:00:51 world among medical people to just have,
1:00:53 like,
1:00:54 essentially,
1:00:54 you have a worldwide study where everybody is enrolling on their
1:00:57 site and then you're sharing data and things like that.
1:01:00 But outside of that,
1:01:01 that's what you need.
1:01:02 So,
1:01:04 What,
1:01:04 what is the advantage of adaptive experiments?
1:01:07 I'm going to show you in a second.
1:01:09 They might make your experiment more efficient,
1:01:11 meaning that you can get to the answer faster with a smaller sample size,
1:01:15 but not always.
1:01:17 They can definitely minimize what the people
1:01:19 in machine learning world call regret,
1:01:22 meaning that
1:01:23 because I'm learning each day,
1:01:25 and then the next day I'm assigning
1:01:28 the treatment that's more likely to be succeeding to more people,
1:01:33 I'm actually assigning by definition,
1:01:35 less people to arms that I don't think are working very well.
1:01:39 So,
1:01:39 it minimizes regret.
1:01:40 It's a bit more ethical,
1:01:41 so to speak,
1:01:42 right?
1:01:43 I also,
1:01:44 because I'm allocating more people to the winner,
1:01:46 I'm generating more precise estimates for the winner.
1:01:50 And what I can also do is I can do this adaptation contextually,
1:01:53 meaning that
1:01:54 I can actually say,
1:01:56 not only I'm going to look at this batch at the,
1:01:59 you know,
1:01:59 every week,
1:02:01 but I'm going to look at this batch
1:02:02 based on whether people are adolescents or adults.
1:02:06 And I'm going to allow for the fact that the
1:02:08 program may be working very differently for those two groups.
1:02:11 And then I'm going to update
1:02:14 the assignment probabilities actually independently.
1:02:17 So,
1:02:18 I can have an experiment in 3 months.
1:02:21 That started with adults and adolescents and like treatment
1:02:24 A and treatment B all given equally to everybody.
1:02:27 And if you had a static AB
1:02:30 type AB test type of experiment,
1:02:32 RCT,
1:02:33 then
1:02:34 that 25% chance for each of those groups
1:02:37 would stay
1:02:39 throughout the entire,
1:02:40 like,
1:02:41 study period.
1:02:42 But now,
1:02:43 at the end of like 3 months in the adaptive experiment,
1:02:46 you could have arrived at the situation where
1:02:48 adolescence and treatment A,
1:02:51 we're only giving it to 2% of them.
1:02:54 But then like adults in treatment B,
1:02:57 70% of the allocations are going to there or something like that.
1:03:01 So you can have these kind of like very unbalanced things,
1:03:04 and that has implications,
1:03:05 of course.
1:03:06 So,
1:03:07 here's kind of like how to see it visually.
1:03:09 So this is a static RCT,
1:03:12 right?
1:03:12 So you say,
1:03:13 OK,
1:03:13 I'm going to allocate 25% to each of the four treatments,
1:03:17 and then that's kind of how many people I'm going to have in expectation.
1:03:21 And then I'm going to estimate at the end,
1:03:23 like,
1:03:24 these T1 that did this,
1:03:26 T2 did this,
1:03:27 this is the mean,
1:03:28 this is the confidence interval,
1:03:30 and then I'm going to say T1
1:03:31 is significantly different than T4 or so on and so forth.
1:03:36 So what's the problem with it?
1:03:38 Like,
1:03:38 like I said,
1:03:38 two things.
1:03:39 First,
1:03:40 throughout the intervention,
1:03:43 I assigned a lot of people to the stuff that was inferior,
1:03:47 so I regret that,
1:03:48 right?
1:03:49 The second thing is
1:03:50 the standard error on the winner and the standard error on the loser
1:03:54 is more or less the same because I allocated the same amount of
1:03:58 um people to each of those groups.
1:04:00 Whereas I would like this to be much more precise.
1:04:02 I'd like to be able to go to the minister and say,
1:04:05 this thing really worked.
1:04:06 It's very tight,
1:04:07 you know,
1:04:08 like,
1:04:08 I have a great estimate of this.
1:04:10 I don't have that.
1:04:11 So if I had an adaptive experiment that I had the same number of people,
1:04:14 you see,
1:04:15 there's like maybe 800 people or something like that,
1:04:18 what I would do
1:04:19 is I would actually have those people start like,
1:04:22 I'd start them equal,
1:04:23 but not assign all of them,
1:04:25 but just some of them.
1:04:27 Get my estimate,
1:04:28 right?
1:04:29 Now,
1:04:30 because I allocated many fewer people,
1:04:32 I have large confidence intervals here at the beginning,
1:04:35 at the end of the first batch.
1:04:37 But then what I do is like,
1:04:39 T1 looks like it's um
1:04:41 doing better,
1:04:42 so I'm going to increase the probability that people are assigned to T1
1:04:46 and decrease the probability that they're assigned to,
1:04:48 let's say,
1:04:49 T3 and T4,
1:04:51 right?
1:04:51 So,
1:04:51 I'm gonna do this using some statistical techniques
1:04:54 that are too complicated to go into here.
1:04:56 So then,
1:04:57 The
1:04:58 number of people assigned is now going to be like this.
1:05:01 Remember,
1:05:01 I'm still doing a randomized experiment.
1:05:04 These probabilities are still random.
1:05:06 Everybody has a chance to end up in either of these bins,
1:05:10 right?
1:05:10 Um,
1:05:11 nobody has a zero chance of anything like we discussed last time.
1:05:14 So this is still a proper randomized experiment.
1:05:16 It's just that I changed the probabilities,
1:05:19 right?
1:05:19 So now I have these estimates
1:05:21 and then I continue
1:05:23 doing this
1:05:24 and then by the end,
1:05:25 you know,
1:05:26 it's possible that now I have actually 400 people in T1,
1:05:29 350 and T2,
1:05:32 only 150 in like T3 and less than 100 in T4,
1:05:36 something like that.
1:05:37 And look now what's happened is that
1:05:39 T1,
1:05:39 which is the best,
1:05:41 I have really tight confidence intervals.
1:05:43 It's like I can tell that it's better than everything else,
1:05:47 whereas I have kind of like,
1:05:49 More uncertainty over here,
1:05:51 but that's the one that,
1:05:52 that I don't care so much about.
1:05:54 Now,
1:05:55 What we did,
1:05:57 we minimized regret to the best we could
1:06:00 because we assigned many more people during the experiment
1:06:03 to
1:06:04 the,
1:06:04 to the
1:06:06 best expected intervention,
1:06:08 and then we also have tighter confidence.
1:06:11 So,
1:06:11 how you're doing that essentially is that you're just saying,
1:06:14 well,
1:06:14 look,
1:06:14 you know,
1:06:15 I'm gonna start with a
1:06:16 kind of like a prior distribution that all of these things work the same.
1:06:20 So,
1:06:21 initially,
1:06:21 I'm going to assign them 1/3,
1:06:23 1/3,
1:06:23 1/3,
1:06:25 but then
1:06:26 afterwards,
1:06:27 I'm going to do something like I have this,
1:06:29 you know,
1:06:30 Thompson sampling method from like the 1930s,
1:06:33 um,
1:06:34 that's actually going to tell me that like,
1:06:37 There's 70% chance that
1:06:39 the arm two is going to win this race.
1:06:42 There's 25% chance that arm three is going to win,
1:06:45 5% chance arm one is going to win.
1:06:47 So,
1:06:48 I'm going to literally take those and those
1:06:50 are going to be my new assignment probabilities.
1:06:52 So,
1:06:52 then I'm going to assign people
1:06:54 next round,
1:06:55 70,
1:06:55 25,
1:06:56 05.
1:06:57 And then
1:06:58 at the end of that batch,
1:06:59 I'm going to combine those two rounds of data.
1:07:01 I'm going to recalculate those probabilities,
1:07:04 so on and so forth until I have a winner.
1:07:06 And you can have stopping rules about like,
1:07:08 when do you declare a winner,
1:07:10 so on and so forth.
1:07:11 Like maybe when you
1:07:12 reach a certain amount of confidence and
1:07:15 this and that.
1:07:16 Right.
1:07:16 You can personalize it,
1:07:18 like we said,
1:07:18 you can do it for like teenagers versus uh
1:07:22 older people,
1:07:23 adults,
1:07:23 etc.
1:07:24 Um,
1:07:26 But the caveat is that,
1:07:28 you know,
1:07:29 because
1:07:30 The way you assign treatments is now dependent on what happened before.
1:07:36 You lose the IID,
1:07:38 um,
1:07:38 you know,
1:07:38 the,
1:07:39 uh,
1:07:41 Independent draws,
1:07:43 uh,
1:07:43 individually independent draws,
1:07:45 um,
1:07:45 of the kind of like the,
1:07:47 a normal distribution.
1:07:48 So what you're gonna have is like,
1:07:50 you no longer have a,
1:07:51 going to have a normal distribution,
1:07:53 like of your error terms.
1:07:55 So that means that
1:07:56 you can't,
1:07:57 while you're doing the adaptive experiment,
1:07:59 you can't just take your data and treat it like you
1:08:02 have a regular RCT and then do the regular inferential analysis.
1:08:07 To do anything,
1:08:08 you have to do something more sophisticated.
1:08:10 And this is actually a very active area of research because
1:08:14 a lot of times these very smart people who are statisticians,
1:08:17 econometricians,
1:08:18 mathematicians,
1:08:19 computer scientists,
1:08:20 etc.
1:08:21 you know,
1:08:22 they have to do things by simulation because there's no closed form,
1:08:26 you know,
1:08:26 formulas or proofs and things like that for the kind of like,
1:08:29 let's say the,
1:08:31 the asymptotic,
1:08:32 um,
1:08:32 you know,
1:08:33 kind of like the
1:08:35 Validity of the kind of like the,
1:08:36 uh,
1:08:37 you know,
1:08:37 consistency and things like that,
1:08:39 unbiasedness,
1:08:40 all those things.
1:08:41 So this is very much an ongoing area of research,
1:08:44 but
1:08:45 The point here is that
1:08:47 there are actually opportunities
1:08:50 to do adaptive experiments,
1:08:51 and I'm doing one and other people have started doing them as well.
1:08:54 You know,
1:08:55 as people come to the hospital,
1:08:57 if you're doing it,
1:08:58 like a,
1:08:59 we have a tablet-based thing and then the tablet is doing all the randomization,
1:09:03 it's very easy to change.
1:09:05 Those randomized assignments,
1:09:07 even the nurses and doctors that are doing
1:09:10 the interventions,
1:09:11 they don't know because all that's happening is like
1:09:13 on the underlying distribution of like assignments is changing,
1:09:17 but we are actually
1:09:19 targeting better and better.
1:09:20 We're doing like personalized tailored.
1:09:24 Intervention assignment to people who are coming in.
1:09:27 So,
1:09:27 definitely people who are coming in
1:09:30 later on in the study are luckier than people who come in earlier in the study because
1:09:34 they're benefiting from all the things that we're learning,
1:09:37 and then they're just kind of,
1:09:38 like,
1:09:39 contributing to this knowledge generation,
1:09:41 uh,
1:09:41 while benefiting at the same time.
1:09:44 One thing to also
1:09:46 um
1:09:47 keep in mind is that
1:09:49 still,
1:09:50 even though this is more like this Beijing,
1:09:52 uh,
1:09:53 type of,
1:09:54 like work,
1:09:55 where,
1:09:56 you know,
1:09:56 you don't necessarily need T tests,
1:09:59 we're not in this,
1:10:00 in that world with
1:10:01 either academics or actually even policymakers,
1:10:04 like policymakers and,
1:10:05 you know,
1:10:05 people who make these decisions,
1:10:07 still gonna want to say,
1:10:09 hey,
1:10:09 compared to my status quo intervention at the schools,
1:10:12 like the way my schools are now.
1:10:14 Can you just show me that like your meals program
1:10:18 did significantly better,
1:10:20 right?
1:10:21 So to be able to do that,
1:10:23 you need
1:10:24 enough data on the status quo,
1:10:26 but in the adaptive experiment,
1:10:28 some algorithms can be very aggressive in eliminating,
1:10:32 if suppose that the status quo was losing very badly,
1:10:35 you can end up collecting very few data points on the status quo.
1:10:39 So,
1:10:39 you may need to actually anticipate that you're gonna need
1:10:43 A quote unquote control arm.
1:10:46 Right?
1:10:46 And you're going to
1:10:48 um
1:10:50 You're going to,
1:10:51 um,
1:10:52 collect enough data on the control arm
1:10:54 so that you can actually compare it reliably to the
1:10:57 T1 because T1 can have a very small confidence interval,
1:11:00 but if T4 has a very big one,
1:11:03 you still won't be able to confidently say T1 did better than T4.
1:11:07 So,
1:11:07 what we do is we have these kind of like,
1:11:10 floors,
1:11:10 so to speak.
1:11:11 So we say like,
1:11:12 we're going to allocate at least 10% of the
1:11:16 sample
1:11:17 to this group,
1:11:18 even if it's doing very badly because we need that.
1:11:21 Or what you can do is you can have a learning phase
1:11:24 where you do the adaptive experiment as aggressively as you want.
1:11:28 And then once you say,
1:11:29 I've learned that,
1:11:30 like,
1:11:31 these two ones are doing the best,
1:11:32 and then I have my status quo,
1:11:34 then you can go to a
1:11:36 evaluation phase where you actually now run a
1:11:40 static experiment quickly
1:11:42 to just say,
1:11:42 OK,
1:11:43 the winner versus the status quo.
1:11:45 Now I'm going to show that,
1:11:46 like,
1:11:46 with very high confidence that this one is doing better than the status quo.
1:11:50 You can do kind of either way.
1:11:53 Then the last 15 minutes,
1:11:54 I want to talk about a couple of things that are practices,
1:11:57 like important practices of analyzing
1:12:00 um randomized experiments.
1:12:03 So,
1:12:04 two things that come up over and over and over
1:12:06 again are kind of like baseline balance and attrition.
1:12:09 And I want to talk a little bit about each one of them.
1:12:12 And if we don't get to whatever I have left after that,
1:12:15 that's OK.
1:12:16 Um,
1:12:17 because these are kind of like the last really
1:12:19 most important things.
1:12:21 So,
1:12:21 on baseline balance,
1:12:23 um,
1:12:24 you know,
1:12:25 I think that
1:12:26 There are things that you can do at the design stage.
1:12:29 And one of the debates in
1:12:31 this between medical people and social science people has been that like,
1:12:35 there's a lot of medical folks who say,
1:12:37 look,
1:12:38 you shouldn't show a baseline balance table because you
1:12:40 have randomized in an RCT you have randomized already.
1:12:44 So,
1:12:45 any differences that you show me are going to be just coming out of that,
1:12:48 like,
1:12:49 that random chance.
1:12:50 So I don't really care,
1:12:52 like if there's differences or not.
1:12:54 And then there are others who say,
1:12:55 well,
1:12:56 actually,
1:12:57 hold on a minute,
1:12:58 that's,
1:12:58 that sounds,
1:12:59 that doesn't sound right because
1:13:01 if there's some really big difference on
1:13:03 a really important characteristic that's sermonistic,
1:13:07 that's still,
1:13:08 that's still troublesome.
1:13:09 Just the fact that I randomized doesn't mean
1:13:12 that like I shouldn't worry about it.
1:13:15 Um,
1:13:15 and then there's also the
1:13:18 Um
1:13:19 I'm going to say?
1:13:21 All right,
1:13:21 let me,
1:13:21 let,
1:13:22 let me continue.
1:13:23 So,
1:13:23 so I think
1:13:25 That's
1:13:26 There's ways actually you can avoid
1:13:29 having those massive differences by chance
1:13:32 by doing this stratification and block randomization.
1:13:35 So,
1:13:36 when I wrote this uh 2 weeks ago,
1:13:38 David McKenzie had just written on this,
1:13:40 like the perfect reference.
1:13:41 So I definitely encourage you to,
1:13:43 to read that.
1:13:44 It's now been 3 weeks.
1:13:46 But essentially,
1:13:46 his main takeaways are,
1:13:48 you want to do 2 stages.
1:13:51 The first stage,
1:13:52 when you're approaching your RCT.
1:13:55 is that
1:13:56 you want to have one or two key variables
1:13:59 on which you want to stratify your sample.
1:14:02 So,
1:14:02 for example,
1:14:02 the easiest things to think about could be like,
1:14:05 you have urban areas and you have rural areas,
1:14:07 they look very different.
1:14:08 So why not just do your experiment within them rather
1:14:11 than like doing unconditional experimentation across the whole country.
1:14:15 Or you can do male and female,
1:14:17 and you don't want to end up with,
1:14:18 like,
1:14:18 60% male and 40% female,
1:14:21 so you You can stratify on that and then have your like equal sizes of male and female,
1:14:25 and then you have,
1:14:26 you can have balance
1:14:27 inside those as well.
1:14:29 So,
1:14:29 you could do those kinds of things.
1:14:31 For example,
1:14:31 Guido Imbens,
1:14:32 who recently won the Nobel for the work he's done on,
1:14:35 partly on these things,
1:14:36 recommends that using cluster size
1:14:39 to define strata is important because often cluster size is
1:14:42 very deterministic of like a bunch of things,
1:14:45 right?
1:14:46 So then,
1:14:47 You have your strata now you're in urban areas
1:14:49 or rural areas or you're in large clusters,
1:14:52 mid-sized clusters,
1:14:53 and then
1:14:54 small sized clusters.
1:14:55 Within them now,
1:14:57 you have your units,
1:14:58 and what you want to do is you want to form,
1:15:00 let's say something like
1:15:01 quadruplets.
1:15:02 If,
1:15:03 if you have one treatment and one control,
1:15:05 you can form quadruplets.
1:15:07 And how do you form them?
1:15:09 The idea is essentially that
1:15:11 I want balance on things that really matter.
1:15:14 What are the things that really matter?
1:15:16 It's either my outcome.
1:15:18 So if I observe
1:15:20 the
1:15:20 baseline value of my outcome,
1:15:22 suppose that we're doing an education intervention and
1:15:24 I'm interested in test scores and learning,
1:15:27 and I have baseline test scores,
1:15:29 right?
1:15:30 Then I can use the baseline test scores.
1:15:33 To match people into,
1:15:35 like,
1:15:35 everybody who got a 4 out of 100.
1:15:38 Like,
1:15:38 as long as there's 4 of them,
1:15:40 I put them into that bin.
1:15:41 And then everybody who got a 5,
1:15:43 everybody who got a 6.
1:15:44 And then,
1:15:44 therefore,
1:15:46 now,
1:15:47 if I have my bin of people who got very bad grades,
1:15:50 and then within them,
1:15:52 I assigned 2 of them to treatment and 2 of them to control,
1:15:56 I know for a fact that I'm going to have balance on that variable.
1:16:00 It's impossible for me not to have balance on that variable.
1:16:03 Now,
1:16:04 if you have
1:16:05 not
1:16:06 one outcome indicator,
1:16:07 but a bunch of outcome indicators,
1:16:09 but they're closely correlated,
1:16:11 then you can make an index of them and then you can,
1:16:13 you can
1:16:14 do quadruplets like based on
1:16:17 being close,
1:16:17 you know,
1:16:18 nearest neighbors kind of thing
1:16:20 on the,
1:16:21 on the index.
1:16:23 If you don't observe the outcome at baseline,
1:16:27 then you should try to do it on the basis
1:16:29 of like baseline covariants that predict the outcome really,
1:16:32 really well.
1:16:33 So for example,
1:16:34 in stunting literature,
1:16:35 your age in months
1:16:37 is going to be extremely important,
1:16:39 so you really want to make sure that like,
1:16:42 The treatment group isn't 2 months older than the control group because
1:16:46 that's going to make a massive difference in their kind of like,
1:16:49 uh,
1:16:49 Z scores,
1:16:50 right?
1:16:51 So what you want to do is you want to take the people who are like aged 13 months,
1:16:56 and then,
1:16:57 you know,
1:16:57 like have quadruplets of them,
1:16:59 14 months,
1:16:59 15 months,
1:17:00 etc.
1:17:01 and then you want to then
1:17:02 randomized
1:17:04 within that.
1:17:04 So,
1:17:05 think of it as you have like
1:17:06 your strata,
1:17:07 you have two strata,
1:17:09 strata,
1:17:09 urban and rural.
1:17:11 Within your urban and rural,
1:17:12 you have a bunch of quadruplets,
1:17:15 right?
1:17:15 And then within those quadruplets,
1:17:17 now you're running a lottery and saying like 2
1:17:19 of these guys are going to be treatment,
1:17:21 2 of these guys are going to be in control.
1:17:23 So that's the kind of like the main takeaway.
1:17:26 And if you do this kind of thing,
1:17:29 And I think that I remember the thing that I was going to say in that debate,
1:17:32 people say,
1:17:33 well,
1:17:33 look,
1:17:33 you know,
1:17:35 If you do that,
1:17:36 then you don't really need to report the baseline balance,
1:17:39 like,
1:17:39 at least not as table one of a,
1:17:41 like a thing,
1:17:41 you can just toss it into the appendix.
1:17:43 So,
1:17:43 I think if you do this kind of thing and you're prepared,
1:17:47 my recommendation is that it's totally OK to
1:17:49 report the baseline balance in the appendix table,
1:17:52 and that You can also describe how you did this,
1:17:54 you know,
1:17:55 randomization,
1:17:56 essentially ensuring that on important key variables,
1:18:00 you will by definition have balance.
1:18:02 Then
1:18:03 it's not that important to show the baseline balance
1:18:05 because all you're doing is confirming you've done that.
1:18:08 You can just stick it into the appendix state.
1:18:10 Right?
1:18:11 The second thing people say is that,
1:18:12 well,
1:18:13 actually,
1:18:13 maybe you didn't do that,
1:18:15 but maybe
1:18:16 you had a plan for re-randomizing.
1:18:18 So you thought about this thing that can happen by chance,
1:18:21 and you said that you're going to minimize it.
1:18:23 So,
1:18:23 what you can do is essentially,
1:18:25 suppose that you have K variant,
1:18:27 K
1:18:28 characteristics at baseline,
1:18:29 you know,
1:18:30 age,
1:18:30 sex,
1:18:31 like test score,
1:18:32 da da da,
1:18:32 it's an urban,
1:18:33 rural,
1:18:34 the household size,
1:18:35 this and that.
1:18:36 And what you're saying is that,
1:18:39 let's say,
1:18:40 I don't want to see
1:18:42 any of the,
1:18:43 the
1:18:44 P,
1:18:45 the differences between
1:18:47 the treatment and control or any of those variables
1:18:50 to be bigger than a certain amount or the P value to
1:18:52 be like higher than a certain amount or something like that.
1:18:56 So,
1:18:56 you can pre-specify saying,
1:18:58 I'm going to randomize this like 100 times,
1:19:02 and then I'm going to look through,
1:19:04 and then I'm going to
1:19:06 Take the one that minimizes the maximum of this like T value,
1:19:10 right?
1:19:11 Like this difference,
1:19:12 like standardized by the um
1:19:14 by the standard errors,
1:19:16 right?
1:19:16 Or you can say,
1:19:18 I'm going to re-randomize until all T stats are below a certain threshold.
1:19:24 So suppose that you only cared about two variables,
1:19:26 like the test score and age,
1:19:28 right?
1:19:29 And you can say,
1:19:30 I'm going to re-randomize until the T tests for both of those are under 1.
1:19:35 Right?
1:19:36 Something like that.
1:19:37 So,
1:19:37 you have to be careful there,
1:19:38 like they,
1:19:38 they should be feasible.
1:19:40 If you have many variables and if you put a really
1:19:43 like
1:19:43 demanding threshold like T equals to 0.05 or something like that,
1:19:48 then
1:19:49 you're never going to get there.
1:19:51 So,
1:19:51 you want to be realistic,
1:19:52 but
1:19:53 re-randomizing is actually,
1:19:55 like,
1:19:55 in many cases,
1:19:56 if you pre-specify it and do it correctly,
1:19:59 it's equivalent to
1:20:01 You know,
1:20:01 like,
1:20:02 unconditionally randomizing
1:20:04 to get equal numbers of people in each group.
1:20:07 So,
1:20:07 you,
1:20:08 If you do this,
1:20:09 you should pre-specify for randomization inference,
1:20:12 which you guys covered with Florence,
1:20:14 and don't search over randomizations for best value,
1:20:17 meaning that
1:20:18 you didn't pre-specify,
1:20:19 you do it a bunch of times,
1:20:21 you're still not happy,
1:20:22 and then you say,
1:20:23 oh,
1:20:23 let me do one more time,
1:20:24 or now,
1:20:25 this one is OK.
1:20:27 Those are not OK because there's then very little randomness left,
1:20:30 like you're imposing your will on the kind of like the distribution.
1:20:35 But otherwise,
1:20:36 if you actually
1:20:37 do either of these two options,
1:20:39 you're fine.
1:20:40 And again,
1:20:41 if you do this,
1:20:42 you will have balance on the pre-specified variables according to your criteria,
1:20:47 right?
1:20:47 Then again,
1:20:48 in my mind,
1:20:49 then it's OK to put it in table,
1:20:51 your appendix table one,
1:20:53 right?
1:20:54 But if you haven't done either of these things,
1:20:56 let's say you inherited the
1:20:59 You know,
1:21:00 RCT or like something like that,
1:21:02 or,
1:21:02 you know,
1:21:03 just for whatever reason,
1:21:04 you didn't do it,
1:21:05 or you did do some of these things,
1:21:07 but like you did unconditional randomization,
1:21:10 but
1:21:11 you actually
1:21:12 don't know whether your RAs did a good job or maybe there was a public lottery,
1:21:17 and you don't know if the public lottery
1:21:19 went according to plan or something like that.
1:21:22 Then this is really a defensive thing.
1:21:24 You're saying,
1:21:24 look,
1:21:25 you know,
1:21:27 We randomized,
1:21:27 but we didn't do all these safeguards,
1:21:30 so now we're going to show you that,
1:21:32 like,
1:21:33 it's actually OK.
1:21:34 Like,
1:21:34 these,
1:21:34 the treatment and control look very similar to each other.
1:21:38 And like I said again,
1:21:39 like I was saying before,
1:21:41 if your study is well-powered,
1:21:43 then you're going to be more likely
1:21:45 to reject differences in covariant values between
1:21:48 arms,
1:21:49 and if it's underpowered,
1:21:51 then you're not going to be able to reject anything.
1:21:54 So,
1:21:55 You don't want
1:21:57 a badly powered study
1:21:59 to be helping you say,
1:22:01 oh,
1:22:02 I have balance because
1:22:03 there's these massive differences between treatment and control,
1:22:06 it's just that they're not statistically significant.
1:22:08 So you have to report
1:22:10 your minimum detectable effects and you,
1:22:12 your,
1:22:12 your
1:22:13 power,
1:22:14 so to speak.
1:22:15 And then you also have to have an
1:22:17 idea of like meaningful differences between these covariants.
1:22:20 Like if I see that like there's 10%
1:22:22 points more people with university education in Malawi,
1:22:26 but it's not significant,
1:22:27 I'm not going to sit there and say like,
1:22:29 oh,
1:22:29 treatment and control look exactly the same when like 10% points
1:22:33 difference in education,
1:22:35 the
1:22:35 university education is a huge deal in a place like Malawi.
1:22:40 The second thing you want to do,
1:22:41 which I don't see often enough,
1:22:42 is that
1:22:43 Instead of just looking pair-wise,
1:22:45 like this variable or is it different,
1:22:47 this variable,
1:22:48 is it different or counting them,
1:22:49 whatever,
1:22:50 you should be doing joint tests of orthogonality,
1:22:52 which means that
1:22:53 is this vector of variables for treatment
1:22:57 different than the vector of variables for control,
1:23:00 and then you want really high P values in these tests.
1:23:03 So like in the links,
1:23:04 like I have some examples here,
1:23:06 I'm very happy.
1:23:07 When I see that,
1:23:08 like,
1:23:08 the treatment and control,
1:23:10 the,
1:23:10 the difference between those vectors is like a F test,
1:23:13 like a P value of,
1:23:14 like,
1:23:14 0.862 or 0.698 or something like that.
1:23:18 And I do get actually worried when it's like 0.11 something,
1:23:23 because then I can see that actually,
1:23:24 some of the variables are looking different.
1:23:26 It's just that it hasn't quite reached
1:23:28 traditional significance level.
1:23:31 The last thing we do before we go is kind of attrition.
1:23:35 I think
1:23:36 the thing that I see with attrition that bothers me most is that
1:23:40 people generally say,
1:23:42 well,
1:23:42 OK.
1:23:44 Attrition is really kind of like
1:23:46 balanced in levels,
1:23:47 so I'm not too worried.
1:23:49 And that's essentially to me is a big cop-out
1:23:52 because you could have had like to be facetious,
1:23:55 let's say
1:23:56 you have 10% attrition in both the treatment and control group,
1:23:59 but in the treatment group,
1:24:01 all the right-handed people
1:24:03 left,
1:24:03 and then in the control group,
1:24:05 all of the 10% that left are left-handed people.
1:24:08 Now,
1:24:09 they're very different than Each other
1:24:11 and so that's going to affect things and so you can't just say,
1:24:14 OK,
1:24:15 I'm going to check levels,
1:24:17 you have to
1:24:17 check for differential attrition between treatment and control
1:24:20 in both levels as well as characteristics,
1:24:24 right?
1:24:24 If you can show,
1:24:25 show evidence that like you're OK in both of the,
1:24:28 I mean,
1:24:28 you're not gonna ever have a study that has zero attrition.
1:24:31 I mean,
1:24:31 I have one One,
1:24:32 but that's mainly because it's the administrative data.
1:24:35 So like,
1:24:36 people do their like counseling session,
1:24:38 I see them,
1:24:39 so nobody drops out.
1:24:40 But as soon as I have to follow them up,
1:24:42 like to ask them a question,
1:24:44 I'm gonna lose some of them,
1:24:45 right?
1:24:45 So you're always going to have attrition,
1:24:47 so you're always going to have to do these things.
1:24:50 If you're lucky,
1:24:51 your attrition will be minimal.
1:24:54 And then it will be balanced in levels and characteristics,
1:24:57 so that you can essentially proceed with like showing the data for like the 96%
1:25:02 of the people that you were able to follow up or something like that.
1:25:05 But in other cases,
1:25:06 you're gonna have to use,
1:25:08 like other,
1:25:09 other types of things.
1:25:11 And one of the
1:25:12 things that I,
1:25:13 uh,
1:25:14 tell people is that,
1:25:15 you know,
1:25:16 the bounds analysis.
1:25:17 I think it is really important and it needs to be much,
1:25:20 much more prominent in many studies
1:25:22 than it is.
1:25:23 Often what I see is like people say,
1:25:25 oh,
1:25:26 attrition,
1:25:26 yeah,
1:25:26 we have some attrition,
1:25:28 sometimes it's like
1:25:28 20% or something like that.
1:25:30 People say,
1:25:31 oh,
1:25:31 that's OK,
1:25:32 20% is pretty comparable to like other studies like ours,
1:25:36 and we see no differences between treatment and control,
1:25:39 so like we're just gonna go ahead.
1:25:41 I think in my mind,
1:25:42 that's more and more unacceptable,
1:25:44 and I think you want to be doing some sort of adjustment,
1:25:47 and that usually my preferred route is these
1:25:51 Kling and Liebman bounds,
1:25:52 and you can see these uh linked papers that I have here.
1:25:56 Um,
1:25:57 and what that does essentially is the following.
1:26:00 Suppose that I have like
1:26:01 5% attrition in both my cash transfer group and my control group,
1:26:06 right?
1:26:07 And then suppose that like the,
1:26:09 it's the,
1:26:09 you know,
1:26:10 something about,
1:26:11 let's say consumption
1:26:12 um that I'm interested in,
1:26:14 right?
1:26:15 So what I want,
1:26:16 so,
1:26:16 and suppose that when I just compared them without any adjustments,
1:26:19 I see that there's an effect,
1:26:21 like there's a large effect.
1:26:22 So now what I wanna say is like,
1:26:25 suppose that the 5% in the treatment group that dropped out
1:26:30 had 10% less
1:26:33 um
1:26:34 Consumption than the mean.
1:26:37 And then the control group
1:26:39 who dropped out,
1:26:40 had 10% more consumption than the meat.
1:26:43 Like,
1:26:43 what would have happened?
1:26:44 Usually,
1:26:45 we do this by standardizing,
1:26:46 so we say like,
1:26:47 what if they,
1:26:48 like,
1:26:48 treatment group had 0.1 standard deviation,
1:26:51 less consumption,
1:26:52 and the control group had 0.1% more.
1:26:54 I recalculate,
1:26:55 like,
1:26:56 I fill in those missing values for the treatment and control.
1:26:59 I recalculate the effect for all the 100%.
1:27:03 Now,
1:27:03 my effect is going to be smaller because what I did is I said,
1:27:06 I stacked the deck against myself,
1:27:09 right?
1:27:09 I said that treatment group attrition was selective in a
1:27:13 way that benefits treatment and control group selection was,
1:27:17 uh,
1:27:18 that,
1:27:19 that actually harmed their like estimates,
1:27:21 so now they got closer to each other.
1:27:23 But I do it,
1:27:25 and then there's still an effect and it's still significant.
1:27:27 Then I say,
1:27:28 OK,
1:27:29 uh,
1:27:29 let me do it again with 0.25 standard deviations.
1:27:32 Like,
1:27:32 I do it again and it's still,
1:27:34 it's smaller,
1:27:35 but it's still significant effect larger.
1:27:36 And then finally,
1:27:38 I do 0.5,
1:27:39 and that's where I lose like they become equal
1:27:42 and then there's the difference is no longer significant.
1:27:45 Now,
1:27:45 what I've done is I've said,
1:27:46 look,
1:27:47 my study is robust.
1:27:49 Even if the
1:27:51 differential attrition
1:27:53 was selected to such an extent that like even if
1:27:56 there was a
1:27:57 0.8 0.8 standard deviation difference between the consumption values
1:28:03 of the people that attrited from my study,
1:28:05 like 0.4 up for the treatment and 0.0.4% down for the control.
1:28:10 I still get
1:28:12 significant effect that cash transfers are working very well here,
1:28:16 right?
1:28:16 So then,
1:28:18 It's a matter of
1:28:20 debating with people,
1:28:22 like,
1:28:22 is it really realistic to think that attrition is so selected that,
1:28:27 like,
1:28:27 I would have these massive differences in the pattern of it,
1:28:30 like,
1:28:31 and then,
1:28:32 yes,
1:28:32 at this point now,
1:28:33 what you're doing is you're doing a little bit of a discussion
1:28:36 with your referees or with your audience and with the policymakers,
1:28:40 you know,
1:28:40 about something like
1:28:42 that you do with non-experimental studies,
1:28:44 like,
1:28:44 is the different,
1:28:45 difference in different
1:28:46 valid because the parallel trends assumption is there or is the IV estimates valid,
1:28:52 etc.
1:28:52 So you're in that world,
1:28:54 but at least you put some bounds on that world and you say,
1:28:57 you can actually say that I have differential attrition,
1:29:00 but I still have
1:29:02 A significant effect.
1:29:03 And only after you've gone to this crazy difference between these two groups,
1:29:07 which I don't think is plausible,
1:29:10 then I have like no effect.
1:29:11 So,
1:29:12 at least that's something.
1:29:13 So I think
1:29:14 that's what I suggest people do with attrition stuff.
1:29:17 And then I had one more thing on covariant adjustments,
1:29:20 but once again,
1:29:21 we've run out of time,
1:29:22 so I'm going to stop there,
1:29:23 and this time I'm not going to offer.
1:29:26 Um,
1:29:26 to do a 3rd lecture on this because we have actually very little.
1:29:30 This is the only thing that's left.
1:29:32 But,
1:29:32 um,
1:29:33 I've written so many blog posts on this,
1:29:35 and then actually there are much better ones that I linked to here
1:29:39 on this topic that,
1:29:40 like,
1:29:41 essentially,
1:29:42 Randomized experiments,
1:29:44 the use of covariates is,
1:29:46 is important.
1:29:47 I think it's very valuable,
1:29:49 but you wanna do it in a way that's transparent,
1:29:52 that's pre-specified,
1:29:53 and that's correctly specified,
1:29:56 and all the information that you need is here.
1:29:59 You don't need me to,
1:30:00 like,
1:30:00 talk about it.
1:30:01 Um,
1:30:02 but with that,
1:30:03 I think I'll stop here.
1:30:05 And I know that this is your last lecture,
1:30:07 and I know that you guys are
1:30:09 not going to be McNamara Fellows.
1:30:12 Uh,
1:30:12 soon.
1:30:13 I wish you the best.
1:30:14 I hope it was a good year for all of you.
1:30:16 We're sad that we couldn't have you here,
1:30:18 but,
1:30:19 uh,
1:30:19 I wish you all the best,
1:30:20 and if you have any questions,
1:30:21 feel free to email me.
1:30:24 Thank you so much,
1:30:25 Mark.
1:30:26 I really appreciate it.
- add-style
- lp-body-content