00:07 I decided to,
00:09 um,
00:10 to do a talk on,
00:11 uh,
00:12 difference in differences,
00:13 um,
00:13 and the,
00:14 the main reason
00:15 being,
00:16 um,
00:17 uh,
00:17 it's been the most tumultuous of the last uh few years and,
00:22 um,
00:23 probably the one that
00:25 I think it's been a little fatiguing for many people to try to um
00:30 read all the papers and uh I thought I might just try to give you
00:35 my own uh
00:36 walkthrough
00:37 of where,
00:38 where,
00:39 uh,
00:39 where we are
00:40 and where we were.
00:41 And so,
00:42 um,
00:43 I'm gonna walk through.
00:44 Uh,
00:45 the problems,
00:46 uh,
00:46 that had kind of haunted,
00:48 uh,
00:48 difference in differences that we now know of.
00:50 Uh,
00:50 it's not really difference in differences that was the problem,
00:53 but it was a
00:54 particular implementation of it
00:56 and,
00:56 uh,
00:57 sort of what we learned from it
00:59 and,
01:00 uh,
01:00 described sort of the,
01:01 the,
01:02 the path forward and kind of give,
01:03 give you some,
01:04 uh,
01:05 comments,
01:06 uh,
01:06 along the way about my own interpretation.
01:09 of this.
01:09 At any point,
01:10 uh,
01:11 if you want to just stop me,
01:12 uh,
01:12 and ask questions,
01:13 I have the Q&A and the chat both open on a different screen so I can
01:17 see it.
01:18 So I'm gonna,
01:18 I'm gonna talk about that,
01:19 this,
01:20 this,
01:20 this literature
01:21 on,
01:22 uh,
01:23 on difference in differences
01:25 is massive.
01:26 Uh,
01:26 it,
01:27 it wasn't always massive.
01:28 The econometrics of it actually
01:30 had virtually nothing,
01:33 um,
01:34 for a long,
01:35 long time.
01:35 Um,
01:36 there was
01:37 Uh,
01:38 to my knowledge,
01:40 uh,
01:40 a few
01:42 papers uh on difference in differences,
01:44 and they were not widely used
01:47 models,
01:48 uh,
01:48 one by Heckman,
01:50 Ikamura,
01:51 and Todd in '97,
01:52 and then uh Alberto Abadi in 05,
01:56 uh,
01:56 but they were,
01:58 uh,
01:58 seemed to make a bigger impact theoretically than they did uh in terms of practice.
02:03 And then there was,
02:04 of course,
02:04 the,
02:04 the,
02:05 the Bertrand Defloe and Mullinon.
02:07 Paper on,
02:08 uh,
02:09 on,
02:09 uh,
02:10 clustering.
02:11 Uh,
02:11 but otherwise,
02:12 it,
02:12 it really wasn't,
02:14 uh,
02:14 a tremendously active area in econometrics,
02:17 uh,
02:18 until around 2016.
02:20 And in 2016,
02:22 there was a um
02:23 paper posted to uh SSRN
02:27 by,
02:27 uh,
02:27 I'm not gonna get their names right,
02:29 even though I speak with them,
02:29 I never get their names right,
02:31 uh,
02:31 Correll Borsak at UCL
02:33 and uh Xavier Yarville,
02:35 and uh they appeared to be the first
02:38 people
02:39 that began to kind of bring attention to,
02:42 uh,
02:42 a problem that was kind of haunting.
02:45 Uh,
02:45 a particular
02:47 kind of estimator,
02:48 um,
02:49 the panel fixed effects estimator of a difference in differences design.
02:53 That's really the first ones that,
02:55 uh,
02:55 brought any attention to it.
02:56 The paper was never published,
02:59 um,
02:59 but sort of
03:01 starts a new,
03:02 uh,
03:03 large research agenda
03:05 that,
03:05 um,
03:06 well,
03:06 I don't know if it started it.
03:07 I don't think that's really true.
03:09 It really was essentially
03:11 about
03:12 I don't know,
03:13 4
03:14 or
03:15 so,
03:16 uh,
03:17 simultaneous
03:18 kind of,
03:19 uh,
03:19 discoveries by,
03:21 um,
03:22 an independent groups of econometricians and applied people,
03:25 uh,
03:26 that,
03:26 and you sort of saw these papers kind of appear all at once and then it was a bit of a,
03:31 a flurry of publications,
03:33 uh,
03:33 in,
03:34 in,
03:35 in,
03:35 uh,
03:35 top,
03:36 and,
03:37 uh,
03:37 top,
03:37 top journals,
03:38 AER.
03:39 Um,
03:40 Journal of Econometrics,
03:41 uh,
03:42 uh,
03:42 were the,
03:43 the main
03:44 places where these papers went.
03:45 And so,
03:46 um,
03:47 they,
03:48 they ended up
03:49 kind of,
03:49 uh,
03:50 sort of shattering a lot of people's,
03:52 uh,
03:53 confidence.
03:53 I think that's,
03:54 that were empiricists,
03:55 that were primarily empirical micro people.
03:58 And um I kinda wanna just sort of give you a,
04:01 a,
04:02 a,
04:02 a crash course in what that kind of earthquake was for a lot of people and,
04:07 you know,
04:08 where things are.
04:10 And they make,
04:10 I guess some comments
04:12 as to whether or not I think that it was all worth it and sort of where we are,
04:16 where,
04:17 uh,
04:17 you know,
04:17 your
04:18 overall confidence level should be.
04:20 So,
04:20 I'm gonna take us through,
04:22 um,
04:23 the basics of different diff,
04:24 uh,
04:25 how it relates to causality.
04:26 I'm gonna talk about this,
04:28 this model,
04:29 uh,
04:29 the two-way fixed effects model is what it's now been called.
04:32 Um,
04:33 I never called it that in graduate school,
04:35 but that's what it's now
04:37 basically associated with.
04:39 Uh,
04:39 and then we'll talk about a few alternatives
04:43 to that model.
04:44 Uh,
04:44 we'll also walk through a simulation and then
04:46 I'm gonna kind of conclude with some comments,
04:49 all right?
04:49 So what is the difference in differences?
04:52 It's the,
04:52 it's a very simple design.
04:54 A group of units
04:55 uh receive a treatment
04:57 at some time period.
04:58 Uh,
04:59 another group of units
05:00 do not receive that treatment,
05:02 or rather,
05:03 another group of units will be used as a comparison.
05:06 That's probably a better way of saying it.
05:08 So a group of units are treated
05:10 and another group
05:11 is going to function as a comparison group.
05:14 Um,
05:14 and we're going to observe them over time and then we're going to
05:16 go through a a really simple calculation to compare them to one another.
05:21 It was kind of historically used in the
05:24 19th century with some health policy debates,
05:26 a couple of different.
05:28 Uh people prominently associated with it,
05:30 uh,
05:30 but it gets reintroduced
05:32 in the 70s and 80s,
05:34 uh,
05:34 at Princeton,
05:36 uh,
05:36 with the industrial relations labor group,
05:38 uh Orly Ashenfelter and David Carter specifically sort of the,
05:42 the,
05:42 the fathers of it and Orly Ashenfelter in particular,
05:45 is kind of the,
05:47 the one that basically
05:48 creates it
05:49 uh by working on a lot of job training programs for the government.
05:53 Uh,
05:53 it was kind of one of the first ones at Princeton.
05:57 Even though they had kind of a strong preference for randomization,
06:01 it was sort of kind of presented as this,
06:04 it sort of kind of floats up as this uh
06:06 um
06:08 Uh,
06:08 non-randomization method that,
06:10 that you would kind of go to if you couldn't do randomization or if you didn't have
06:15 randomized instrumental variables.
06:17 Uh,
06:17 it's since become the most popular of all the quasi-experimental.
06:21 Uh,
06:22 methods.
06:22 This is from a paper in the AEA Papers and proceedings by Janet Curry and co-authors,
06:28 and you can see right here,
06:29 this scale goes up to 25% of all papers.
06:33 By 2018,
06:35 uh,
06:35 about 23% of all papers at MBER
06:40 are using difference in differences and about 16%.
06:44 At,
06:44 um,
06:44 the top 5 journals are using differences and differences.
06:47 So if you have
06:49 a good question,
06:49 and it's the way I interpret this is if you have a good question or an important,
06:53 a question that is viewed as important by the community,
06:57 uh,
06:57 and the editors,
06:58 and uh you have the appropriate data.
07:01 Then,
07:02 uh,
07:02 you can publish,
07:03 uh,
07:03 at the,
07:04 uh,
07:04 you can publish different different papers anywhere,
07:06 OK?
07:07 So they're,
07:07 they're,
07:07 uh,
07:08 they've only gotten more popular
07:10 over time,
07:10 not,
07:11 not,
07:11 not less.
07:12 Um,
07:13 so you really can't understand though,
07:16 uh,
07:16 you really can't understand difference in differences without kind of
07:20 committing upfront with some,
07:23 uh,
07:23 of that,
07:24 of that potential outcome,
07:26 uh,
07:27 notation.
07:28 And if,
07:28 if you're like me,
07:29 when you went to graduate school,
07:30 uh,
07:31 I graduated in '07,
07:32 econometrics
07:33 was my field course.
07:35 We didn't actually work with potential outcomes notation.
07:38 It,
07:38 it,
07:39 uh,
07:39 the back
07:40 when I was in graduate school,
07:41 I feel like an old person even talking this way,
07:43 but,
07:43 but when I was in graduate school,
07:45 we,
07:45 we did not use
07:47 this notation,
07:48 uh,
07:49 which
07:49 I think actually has been a real breakthrough for a lot
07:52 of econometricians um when it was first starting to be introduced
07:56 in the 90s.
07:58 Um,
07:59 it's really flexible,
07:59 very powerful way of expressing,
08:02 uh,
08:02 causal effects.
08:03 So I know many,
08:03 I know basically everyone on the call is familiar with it,
08:06 but I just want to make sure
08:07 that we introduce the notation so that we're comfortable using it.
08:10 So
08:11 the individual treatment effect.
08:13 Is uh Delta I,
08:15 uh,
08:16 and it is,
08:16 uh,
08:17 that's the,
08:17 the,
08:18 that's what I call a causal effect throughout the slide,
08:20 throughout the talk,
08:22 and,
08:22 uh,
08:22 it's the difference between
08:24 an
08:25 outcome.
08:26 In a world where you're treated
08:28 for me
08:29 and an outcome in a world where I was not treated
08:32 for me
08:32 at the exact same point in time.
08:34 And
08:35 you can tell when you frame it this way,
08:36 this is sort of the,
08:37 the interesting piece about this,
08:39 uh,
08:40 this potential outcome framework is how
08:42 you're immediately confronted
08:44 when you lay it out this way,
08:45 uh,
08:46 with this kind of like epistemological
08:48 problem,
08:49 which is,
08:49 uh,
08:50 if I have to,
08:50 if,
08:51 if I just kind of like
08:52 You know,
08:53 as a starting point,
08:54 I'm going to define causal effects
08:57 with respect to
08:58 counterfactuals,
09:00 then I've basically said it's not knowable.
09:02 At the individual level,
09:03 it cannot be calculated because
09:06 I don't have
09:07 both of these,
09:08 these,
09:09 these,
09:09 uh,
09:10 uh,
09:10 I don't,
09:11 I don't ever see both of these,
09:13 both Y1 and Y0 for the same person at the same point in time.
09:16 So there's a sense in which
09:18 Under the,
09:18 the Ruben Niman causal potential outcomes model,
09:21 causality is,
09:22 is,
09:23 is not knowable.
09:24 Uh,
09:24 and that,
09:24 that's kind of an old
09:26 view that you saw
09:27 with philosophers like David Hume too,
09:29 which always kind of mystified me for a long time and now,
09:32 uh,
09:32 I think for a lot of
09:34 people,
09:35 In economics is just taken for granted,
09:37 that it's the fundamental problem of causal imprints.
09:40 And
09:41 you,
09:41 since these are individual treatment effects and you've
09:43 got people in treatment groups and control groups,
09:46 um,
09:46 you can aggregate them.
09:47 You can take like,
09:48 you know,
09:48 weighted averages over each of these individual treatment effects
09:52 and kind of come up with these more interesting aggregate parameters,
09:55 right?
09:55 So this one's the,
09:56 the,
09:56 the average treatment effect on the treated.
09:59 That's just what,
10:00 that,
10:00 that would just be
10:02 the,
10:02 uh,
10:02 the weighted average.
10:04 Overall individual treatment effects for a group
10:07 of people that received the treatment.
10:09 And I put this in red right here,
10:11 uh,
10:12 because I wanted to illustrate that
10:13 while this one can be observed,
10:15 cause for the treatment group,
10:17 I do observe the outcome in a world where they were treated.
10:20 It's almost tautological to say it that way,
10:22 but it,
10:23 but it's not,
10:24 it's technically different.
10:25 I do not observe this one.
10:27 For a group of people that are treated,
10:29 I do not observe their outcome had they not been treated.
10:32 And so again,
10:32 we see that even when we aggregate up.
10:35 Even when we aggregate up,
10:36 we don't like avoid
10:38 the,
10:38 the problem that was sort of laid out before because
10:41 um
10:42 uh all these causal parameters rely on counterfactuals.
10:46 So all we can do
10:48 in,
10:48 uh,
10:49 in our work is estimate these parameters,
10:51 but there's a sense.
10:53 In sort of this mysterious way,
10:54 there's a sense in which that is a different type of knowledge.
10:58 And so like,
10:58 uh,
10:59 you know,
10:59 the,
11:00 the,
11:00 the,
11:00 the actual visibly witnessing why 1 and Y 0 would
11:04 seem to almost approach certainty and any of this kind of
11:08 estimation is knowledge.
11:09 But it's sort of a lower level,
11:11 but this is the one
11:12 here that difference in differences is going to be focused on.
11:14 It's gonna be focused on the ATT and in that sense,
11:17 it's gonna be a little bit different
11:19 from the kinds of aggregate parameters that you
11:21 might have the luxury of identifying under an RCT
11:24 because under an RCT.
11:26 Uh,
11:27 you've got
11:28 this,
11:28 uh,
11:28 this condition where
11:30 the,
11:31 uh,
11:31 the,
11:32 the,
11:32 the treatment assignment occurs
11:34 for reasons that are completely unrelated to the potential outcomes.
11:37 And when you do that,
11:38 and I'll show you in the next slide,
11:39 what happens with that,
11:41 you can recover.
11:42 All of the aggregate parameters,
11:43 but with difference in differences,
11:45 you can't.
11:46 All the difference in differences is ever going to get you to that ATT.
11:49 And since you don't use randomization
11:51 for difference in differences,
11:53 you're forced
11:54 to place restrictions
11:56 on this one.
11:57 This is the one that has to be uh restricted in some important way,
12:02 uh,
12:02 but because it's not random,
12:04 because the treatment is not randomized,
12:05 it's gonna be a fairly non-trivial,
12:07 uh,
12:08 restriction.
12:09 Um,
12:10 that isn't always widely appreciated as,
12:12 as to what a big deal the assumption is.
12:15 So,
12:16 this is the,
12:17 uh,
12:17 the,
12:17 the diff and diff equation.
12:18 I call it the diff and diff equation.
12:20 It is more commonly now being called the 2 by 2 because of
12:24 Andrew Goodman Bacon's 2021 paper
12:26 uh in the Journal of Econometrics,
12:28 which we're gonna review,
12:29 but,
12:29 uh,
12:30 I just kind of like to call it the diff and diff equation,
12:32 uh,
12:33 sort of to,
12:33 to sort of keep it within this framework that we're gonna use,
12:36 and this is all it is.
12:37 It's,
12:38 uh,
12:38 uh,
12:39 for a group of units that are treated,
12:40 Group K,
12:42 right,
12:42 and group U.
12:44 Right.
12:44 So I want to know like the effect of getting a PhD
12:47 on my happiness.
12:48 All right.
12:49 So what I could do is I could take a group of people that have PhDs,
12:52 graduated in like '07 when I graduated,
12:55 that would be like,
12:56 you know,
12:56 a year after they got their PhD and a year before they got their PhD.
13:00 I can compare them with like a group of people
13:03 that
13:04 didn't have their PhDs ever.
13:06 All right,
13:06 so that would be like,
13:07 you know,
13:08 this group of people here,
13:09 uh,
13:10 group you,
13:10 the no PhD people observed in the post-period,
13:13 observed in the pre-period.
13:14 Now the post
13:15 here
13:15 is referring to when,
13:17 uh,
13:17 when the treatment group got their treatment.
13:19 So this
13:20 concept of post,
13:22 uh,
13:22 is always over here with respect to Kay.
13:24 It's,
13:24 it's really got nothing to do with that group because that group,
13:27 that group wasn't treated.
13:28 So that's not the,
13:29 the reference point.
13:31 All right.
13:32 So what we do is we use uh what's called a switching equation.
13:35 We replace
13:37 uh all of those potent,
13:39 all of those actual outcomes.
13:40 Notice there's no superscripts anywhere,
13:42 so it's like average outcome
13:45 for group K in the post-period.
13:46 Well,
13:47 in the post-period,
13:48 the treatment group is treated.
13:50 So I don't just observe Y,
13:52 I actually observe Y1,
13:54 right?
13:54 But for every other period
13:56 in the pre-treatment for the treatment group or for the comparison group,
13:59 it's always Y 0.
14:01 All right.
14:01 So I can just replace,
14:03 this is just the different equation or the 2 by 2 with the,
14:06 with just kind of like replacing
14:09 uh potential outcomes for actual outcomes.
14:12 Seems like it's kind of trivial and in,
14:13 in a sense,
14:14 it's like that,
14:15 that step alone
14:16 isn't really,
14:17 you know,
14:18 important,
14:18 but it's like once we move into
14:20 this world of potential outcomes,
14:22 then we can kind of break out things into causality and bias.
14:25 So what I'm gonna do is
14:27 I'm gonna take this diff and diff equation
14:29 that has its uh
14:31 uh potential outcomes flipped in,
14:33 and I'm just gonna add in a 0.
14:35 All right?
14:36 I'm gonna add in a 0.
14:37 Now notice these are these,
14:38 uh
14:38 these um
14:40 counterfactuals.
14:41 So
14:42 for Group K,
14:44 all right,
14:44 in the post-period.
14:46 Uh,
14:46 what would be their outcome had they not been treated?
14:48 Well,
14:48 that didn't happen,
14:49 but we're gonna put it in anyway,
14:51 because part of the nice part about potential
14:53 outcomes is you can still define this stuff,
14:55 you're just not measuring it.
14:57 All right?
14:57 So I got this here,
14:59 I put in a 0,
15:00 and then I'm just gonna move some stuff around,
15:01 right?
15:02 So what I did was I just moved these around
15:04 and I get this expression here.
15:06 I notice that's the ATT.
15:07 That's the ATT from a couple of slides ago.
15:10 All right,
15:11 so I want to know when I take the 2x 2,
15:13 I actually have the ATT inside it.
15:17 But I also have this term here,
15:19 and this is a biased term,
15:21 all right?
15:21 This is a biased term.
15:23 And this is basically saying,
15:25 uh,
15:25 in a world where nobody was ever treated,
15:27 what would be the natural evolution in the outcome?
15:31 OK.
15:31 What would be the natural evolution in the outcome?
15:33 And so if the natural evolution in the outcomes
15:36 for the treatment group and control group were
15:38 exactly the same on average at the mean,
15:41 uh,
15:41 well,
15:41 then
15:42 there's no bias,
15:43 OK?
15:44 There's no bias and so you could just simply do a simple 2 by 2 and you'd isolate
15:48 the ATT.
15:50 OK.
15:50 That's called parallel trends.
15:52 Everybody knows it,
15:53 but I just wanted to put it out there.
15:54 Uh,
15:55 it's a counterfactual assumption.
15:58 And it is not trivial.
15:59 All right.
16:00 The reason it is not trivial
16:01 is,
16:03 uh,
16:04 here.
16:06 Something Don Rubin said,
16:07 uh,
16:08 in a lecture I saw him.
16:10 Don Rubin said,
16:11 you know,
16:11 the reason that randomization is really special.
16:15 You know,
16:15 it's like when you,
16:16 when you,
16:16 when you really think about
16:18 what a,
16:18 what a time to be alive,
16:20 you know,
16:20 uh,
16:21 1923,
16:22 1925,
16:23 Jersey Nieman
16:24 lays out the potential outcomes framework.
16:27 Ronald Fisher
16:28 realizes
16:29 that physical randomization
16:32 will
16:32 allow you to identify average treatment effects.
16:36 We've,
16:36 we've been running experiments
16:38 for centuries and we've never known that.
16:40 We've never known.
16:41 Best I can tell,
16:42 nobody really knew.
16:44 That physical randomization was so special,
16:47 right?
16:47 And the reason it was so special is because
16:49 when you physically randomize,
16:52 you can write down the qualities like this,
16:54 these conditional expectation,
16:56 equalities.
16:57 So the,
16:58 the conditional
16:59 expected potential outcome for the treatment group will be exactly equal to that
17:03 of the comparison group at the mean of actually the same distribution,
17:07 uh,
17:07 so long as independence holds,
17:08 which independence does hold with physical randomization.
17:12 Right?
17:12 And so Ruben says,
17:13 well,
17:13 that,
17:13 the reason we know that is because we know how science works.
17:15 We know,
17:16 we know how
17:17 uh probability works.
17:18 We know how randomization works.
17:20 There is no science.
17:22 There is no science
17:23 that gives us this,
17:25 OK.
17:26 So,
17:26 in,
17:26 in,
17:26 in a way,
17:27 like,
17:27 you know,
17:28 the,
17:29 the challenge of difference in differences is not that
17:32 it's a lesser brother.
17:34 Of physical randomization,
17:37 all of the causal inference methodologies identify causal effects.
17:42 None of them is,
17:43 none of them is inferior
17:45 to one,
17:46 to the other.
17:47 Uh,
17:48 it,
17:48 it's more like on the human side,
17:51 on the finite human side,
17:54 uh,
17:54 difference in differences is hard.
17:57 It's harder.
17:59 It's harder to believe.
18:00 It's not harder to do,
18:01 and it's not that it's less truthful,
18:03 but we don't have the same confidence in it
18:06 because
18:07 we don't have the same reason.
18:08 We don't have the same reason like we have with
18:11 independence because independence practically guarantees,
18:14 you know,
18:15 randomization practically guarantees independence in a large sample,
18:18 uh,
18:18 you know,
18:19 given
18:20 Uh,
18:21 some weird behavior by people inside the experiment.
18:23 Putting that aside,
18:25 it practically guarantees it.
18:26 There is no such guarantee
18:27 with parallel trends.
18:28 And so,
18:29 uh,
18:30 you have this epistemological uncertainty,
18:32 and I always kind of hammer this,
18:33 um,
18:35 because it's really important to me.
18:36 I don't know,
18:36 it's,
18:36 it's,
18:37 uh,
18:37 it,
18:38 it,
18:38 it's,
18:38 it's something I get really preoccupied with,
18:40 which is
18:41 All of causal inference is about these key identifying assumptions,
18:46 none of which can be
18:48 directly
18:48 observed,
18:50 but with randomization,
18:51 it's almost like we're given
18:53 permission
18:54 to not directly observe it,
18:56 and with difference,
18:57 and with difference in differences and this parallel trends assumption,
19:01 it's like a shaky permission.
19:03 It's like,
19:03 you know,
19:04 you're not positive that your parents gave you
19:06 permission to have the car.
19:08 You,
19:08 you,
19:08 you think they probably did,
19:09 but you're not positive.
19:10 But
19:11 with randomization,
19:12 you're 100%,
19:12 you're basically 100% certain,
19:14 right?
19:15 So,
19:17 So let's look at how uh
19:20 we have gone about estimating that ATT.
19:24 All right.
19:24 So,
19:24 you can use that simple definitive equation,
19:27 which is this one right here.
19:28 You can use this,
19:29 and it'll isolate the ATT because look,
19:31 under
19:31 parallel trends,
19:32 that simple
19:33 equation will isolate the ATT.
19:35 It's just the simple differences in means,
19:38 OK?
19:38 Right here.
19:39 It's just the simple differences in means and it'll isolate
19:42 the ATT as long as you have parallel trends.
19:44 And you can also use a regression.
19:46 You can also use a regression.
19:47 You just have to have this specification
19:50 for the regression,
19:51 OK?
19:52 You've got to have a treatment dummy,
19:54 you gotta have a post-treatment period,
19:56 and then you got to have an interaction.
19:59 So long as you have one treatment group,
20:01 right?
20:01 So if you have one treatment group and one comparison group,
20:04 you,
20:04 that's the OLS specification and with parallel trends,
20:08 it will isolate the ATT.
20:09 Now,
20:09 if you have to put in covariants,
20:11 that one won't work.
20:12 Uh,
20:13 you can,
20:13 you're,
20:14 you're gonna need additional assumptions with this OLS specification,
20:18 uh,
20:18 if you want to put in covariants,
20:20 and that's kind of laid out more clearly in
20:22 Santana and,
20:23 and,
20:24 uh,
20:24 Santana and Zal 2020.
20:26 Uh,
20:27 but it's actually like a longer literature,
20:29 uh,
20:29 that I mentioned a minute ago.
20:30 If you want to put in covariants,
20:31 you either can use this model,
20:32 put in covariants,
20:34 willing to make some additional assumptions,
20:35 or you can use,
20:37 uh,
20:37 a couple of other methods that don't require
20:40 those additional assumptions.
20:42 Um,
20:43 but it's when you have differential timing,
20:45 it's when you have differential timing that things get more complicated because
20:49 if you have differential timing,
20:50 it means you have groups that are getting treated at different points in time,
20:53 and that means that this idea of a post-period doesn't make any sense.
20:58 Cause
20:58 if I'm treated,
20:59 if I have the 07 group that gets their PhD and I have the 2017 group that gets their PhD
21:05 and I have a group of people that never get their their PhD,
21:09 I don't know what post means for that last group.
21:12 I don't know if it means 2008 or if it means 2018
21:15 because post
21:16 is a concept
21:18 with respect to a treatment group.
21:20 And if I've got multiple ones,
21:21 I can't define
21:23 what post is for an,
21:24 an,
21:25 a non-treated group,
21:26 right?
21:26 Or that up,
21:27 you know,
21:27 the untreated group.
21:29 So,
21:29 what was done
21:30 instead,
21:31 what was done instead was to use
21:34 Um,
21:36 Oh,
21:36 I took it out.
21:37 So what was,
21:38 what was done instead was to use a,
21:39 a panel fixed effects
21:41 estimator,
21:42 and a panel fixed effects estimator
21:45 was just simply to regress the outcome onto a dummy,
21:49 uh,
21:50 where,
21:50 uh,
21:51 the dummy turned on.
21:53 Uh,
21:53 0 or 1,
21:54 depending on if in that particular period a unit was treated or not.
21:58 And so you would have
21:59 all 0s for an untreated group that stayed untreated the whole time
22:03 and you'd have this kind of staggering of 0 to 1
22:06 for each of the treatment groups,
22:08 OK?
22:09 And
22:10 we know from the panel fixed effects literature,
22:12 and this is how I always learned,
22:14 uh,
22:14 was that that was called the panel fixed effects estimator.
22:17 It wasn't really called the two-way fixed effects because you,
22:20 you technically didn't have to put in.
22:22 Your fixed effects.
22:23 Actually,
22:24 in order to get an unbiased estimate of that parameter,
22:27 you don't have to put that in,
22:28 but
22:29 Uh,
22:30 it,
22:30 but it was usually done.
22:31 And you would put in this vector of panel unit fixed
22:35 effects and a vector of year fixed effects and then you would
22:39 have that treatment dummy.
22:40 And so long as
22:42 something called strict exogeneity held,
22:45 you would get a consistent estimate of this
22:47 parameter that had no subscripts on it.
22:49 It had no like
22:51 unit I subscripts or unit T subscripts.
22:53 It's just this constant
22:55 in the equation,
22:57 all right.
22:58 And
22:59 I,
22:59 I knew what strict exogeneity was
23:03 because I passed the,
23:05 the,
23:05 the,
23:05 the econometrics field exam.
23:07 Uh,
23:08 it requires that all the errors in all the
23:10 periods were independent of treatment status in all periods.
23:13 I just
23:14 never,
23:15 that was not
23:17 a quasi-experimental kind of design.
23:20 That was a specific estimator.
23:21 And so,
23:22 I didn't really know
23:24 how to express that in a difference in differences context.
23:28 In my book,
23:29 I just kind of was guessing and I basically was,
23:32 was saying,
23:33 well,
23:34 I think it means parallel trends,
23:36 but I think it more generally means uh the policy is an indulgence.
23:40 That's the only thing I could,
23:41 could conclude.
23:42 I was,
23:43 I was just thinking.
23:45 Uh,
23:45 I guess that strict exogeneity means,
23:48 um,
23:49 no policy endogeneity,
23:51 and maybe it also meant parallel trends,
23:53 but it turned out to be more complicated than that,
23:55 and I had no idea.
23:57 So this is from John Gardner's paper on two-stage diff and diff.
24:01 So let's define this aggregate treatment effect,
24:04 delta,
24:05 and let's allow it to be heterogeneous
24:08 across panel units,
24:10 which is very typical,
24:11 right?
24:11 I think like I did this Twitter poll one time and I was asked,
24:14 Um,
24:15 you know,
24:15 what,
24:16 what do you,
24:16 what do you guys think?
24:17 Do you,
24:18 do you think when you sort of,
24:19 if I have a gun to your head and I say,
24:20 you know,
24:21 do you think treatment effects are
24:22 at the individual level are sort of,
24:24 you know,
24:24 do they differ
24:26 or are they the same for everybody?
24:27 It was kind of interesting because
24:29 by and large,
24:29 best I could tell,
24:30 and I know that obviously it's selection bias,
24:32 but uh best that I could tell.
24:35 The,
24:35 the,
24:36 the economists
24:38 are the ones that just sort of commit
24:41 upfront to heterogeneity,
24:43 and it was like epidemiologists
24:45 for the most part,
24:46 sort of start with homogeneity.
24:48 I thought that was kind of interesting because I had no idea.
24:49 I don't know why that would be the case.
24:51 Uh,
24:52 they all were using potential,
24:54 they were all familiar with causal inference and observational data.
24:57 So it's kind of interesting that,
24:58 you know,
24:58 the economists are sort of
25:01 in 2012,
25:01 2022,
25:02 deeply committed to deep heterogeneity.
25:05 They just kind of assume it.
25:06 Uh,
25:07 I mean,
25:07 and it goes back to the Nobel laureates,
25:10 um,
25:10 and,
25:11 you know,
25:11 uh,
25:12 and just,
25:12 you know,
25:13 100 and something years of,
25:14 of thinking deep about econometrics,
25:17 but I just kind of wanted to bring it up.
25:19 Because in that panel fixed effects
25:21 specification,
25:23 the heterogeneity had not
25:24 traditionally been expressed.
25:26 It was usually just
25:27 Y regress onto a dummy with a coefficient that was constant,
25:30 it would just be this,
25:31 it'd just be a constant.
25:33 But Gardner sort of expresses it
25:35 now.
25:37 As a potentially heterogeneous.
25:40 Uh,
25:41 parameter that's going to differ
25:42 by panel units.
25:43 We're going to group it by group and group and period.
25:47 All right?
25:47 So it's gonna be a group of units that are treated
25:50 and,
25:50 uh,
25:50 so we have like panel units,
25:52 people,
25:52 but we're gonna have uh groups
25:54 be treated at a particular point in time,
25:57 all right?
25:58 This is calendar time and this is like relative,
26:00 relative time.
26:01 All right.
26:02 And so,
26:03 you've got some mean aggregation over all of those,
26:06 all right.
26:06 Now,
26:07 in the two-way fixed effects specification,
26:10 you have parallel trends here.
26:12 All right.
26:12 So,
26:13 so always,
26:13 that was where two-way fixed effects was grabbing out that parallel trend because
26:18 when,
26:19 um,
26:19 when this was a 0,
26:21 all right,
26:22 when this was a 0,
26:23 then that was just a Y 0
26:26 and it would evolve.
26:27 For everybody the same.
26:29 It would,
26:29 it would involve everybody kind of the same.
26:31 It evolved by this,
26:32 you know,
26:32 the,
26:32 these,
26:33 these,
26:34 these period fixed effects
26:35 and some kind of level,
26:37 some kind of level intercept,
26:38 and it would just kind of evolve.
26:39 So parallel trends was always
26:42 inside.
26:42 The two-way fixed fix.
26:43 It was actually like very strict though,
26:45 cause it was requiring it to hold
26:47 all the time even before treatment help,
26:49 but let's put that aside for now.
26:51 And then we have here the ATT,
26:53 all right?
26:54 So if you substitute this
26:57 into here,
26:58 all right,
26:59 what you get is this more complicated
27:01 regression
27:03 specification.
27:03 You get parallel trends here,
27:05 you get this ATT here.
27:08 But then you've got this new composite error term,
27:11 all right,
27:11 which is the deviations from that average and it's correlated
27:16 with the period fixed effects and the group fixed effects.
27:19 And so you can see here.
27:22 It's also correlated with D,
27:23 so
27:24 it,
27:25 it's no longer
27:26 the case
27:28 that
27:29 D
27:30 is independent of the new composite error term,
27:34 right?
27:34 Like,
27:34 so,
27:35 so here we see why this was all getting screwed up.
27:38 It wasn't that two-way fixed effects was a problem.
27:41 It was that
27:42 strict exogeneity was not holding with that specification.
27:46 OK,
27:46 so it was never,
27:48 it was never,
27:48 the problem was never
27:50 to a fixed effects.
27:51 The problem was
27:53 the model.
27:53 The model specification was always the problem.
27:57 It,
27:57 and it just wasn't,
27:58 I don't really know.
27:58 I mean,
27:59 I bet if I
28:01 Gone to Harvard and took,
28:02 you know,
28:03 econometrics from Gary Chamberlain,
28:05 he knew,
28:06 probably,
28:06 I mean,
28:07 probably like everybody knew,
28:08 uh,
28:08 there was an econometrician,
28:10 but I,
28:10 I didn't know,
28:11 you know,
28:12 I'm a normal person that never was paying attention
28:14 apparently if they were teaching it in class.
28:16 So,
28:18 so from this
28:19 kind of realization.
28:21 All right.
28:21 Uh,
28:21 and,
28:21 and it's kind of like something that's not really made super explicit.
28:26 I've actually not seen it
28:28 kind of spelled out super explicitly in every one of these papers,
28:32 um,
28:33 uh,
28:34 of exactly that kind of expression except for in John Gardner.
28:37 Uh,
28:38 but you would talk to the econometricians and like in,
28:40 in interviews,
28:40 I would,
28:41 they would articulate it.
28:43 Um,
28:44 uh,
28:45 there was more effort just kind of breaking down,
28:48 like,
28:48 OK,
28:48 fine,
28:49 then what exactly isn't,
28:51 so I've got this like,
28:52 you know,
28:53 in this regression,
28:54 I've got this coefficient,
28:56 right?
28:57 I've got this coefficient on D.
28:59 What is it?
29:00 And so they would kind of do two things.
29:02 The papers would do two things.
29:03 They'd say,
29:04 well,
29:05 what numbers add up?
29:06 When I run the regression,
29:07 what are the numbers that add up to that number in the regression?
29:11 That was kind of a numerical
29:13 decomposition.
29:14 So if you were confused at all around all these papers,
29:17 one of the ways that you might have been confused is with this word decomposition.
29:21 Of the two-way fixed effects coefficient,
29:24 and you would find people saying
29:26 different things.
29:27 You would find some people say,
29:29 talking about negative weights,
29:31 for instance.
29:32 Uh,
29:32 many of you probably have heard about this problem of negative weights.
29:35 But if you've read,
29:36 uh,
29:37 Andrew Goodman Bacon's paper,
29:38 he says all the weights are positive.
29:40 So how can they both be true?
29:42 And they're both true because they're not the same types
29:45 of decomposition.
29:47 Goodman Bacon,
29:48 what he's doing
29:50 primarily.
29:51 And he's,
29:52 he's breaking down
29:54 what numbers
29:56 add up to
29:57 the regression coefficient,
29:59 not in terms of causality,
30:01 but like just simply adding together,
30:03 all right?
30:04 That's a numerical,
30:05 that's why I just call it,
30:06 I'm just calling these days a numerical
30:07 decomposition of the two-way fixed effects coefficient versus
30:11 a theoretical one.
30:12 And a theoretical one says,
30:14 OK,
30:14 here's this regression coefficient,
30:16 what does it mean?
30:17 What does it mean in terms of causality and bias?
30:20 And that's where John Gardner,
30:22 Chase Martin and Dehoful Borsak and Yarraville,
30:24 that's where son and Abraham,
30:25 that's where all of them
30:27 have focused their main attention because
30:29 when the econometricians
30:31 are
30:32 working,
30:33 their goal is to,
30:34 once they figure out theoretically what is wrong with the,
30:38 with the number,
30:39 they can then design the solution,
30:42 right?
30:42 You can't really design the solution by just knowing
30:46 that um
30:47 You know,
30:48 so it's like,
30:48 it's like a cake.
30:50 A cake that you eat is based on eggs,
30:52 flour.
30:54 Um,
30:55 water,
30:56 yeast,
30:56 all this stuff,
30:57 right?
30:58 And so you,
30:59 and it even has ingredients like,
31:00 you know,
31:00 1 cup flour,
31:02 uh,
31:02 you know,
31:03 1/2 a cup water,
31:04 whatever.
31:04 So like that would be the equivalent of the numerical decomposition,
31:08 but the theoretical decomposition would be things like
31:11 when I eat a cake,
31:13 I remember.
31:15 My,
31:16 my last birthday,
31:17 or I remember my best memories,
31:19 right?
31:19 It's like that's what a theoretical decomposition is.
31:22 Theoretical decomposition is like this mysterious
31:25 meaning behind the number,
31:27 and this is kind of like what we're about in science,
31:30 but we've got to like use numbers to get there.
31:33 So,
31:34 let's look at Goodman Bacon and before we kind of move into these solutions.
31:38 So,
31:38 we've got
31:39 two groups,
31:40 early treaty group and a late treaty group,
31:42 right?
31:42 Because we're working with this differential timing scenario.
31:45 We're going to define Group K and Group L
31:48 as
31:49 different
31:50 based on when they got treated.
31:52 So,
31:52 L is going to be the later treated group and K is going to be the first one.
31:56 And uh
31:58 we're,
31:58 uh,
31:58 we're gonna have some weights
32:00 uh that we're going to put on
32:02 uh 2 by 2s.
32:03 We're gonna,
32:04 and we're gonna get these 2 x 2s,
32:06 OK?
32:06 So let's look at
32:07 what a 2x 2 is in the context of differential timing.
32:11 Well,
32:11 all that a 2x 2 is,
32:13 is an average.
32:14 It's a,
32:15 it's 4 averages,
32:17 4 averages,
32:18 an average here,
32:19 an average here,
32:19 an average here,
32:20 an average here,
32:21 right?
32:22 Where here,
32:23 Uh,
32:23 is with respect to whether it's after a treatment or before a treatment
32:28 for some reference group,
32:29 OK?
32:30 So we got 4 averages and we just go through that simple diff and diff equation
32:34 after minus before for the treated group,
32:36 after minus before for the untreated group,
32:39 after minus before for the later group,
32:41 after minus before for the still the untreated group,
32:44 and these are a special
32:45 kind of 2 by 2s.
32:47 This is using a,
32:49 what's now being called a never treated group
32:52 because it's never treated like that you group,
32:55 never treated over the sample,
32:56 right?
32:57 But you can also do these 2 by 2s.
32:59 You can,
32:59 you can start comparing treated groups to treated groups.
33:02 Now,
33:02 when you,
33:03 you probably,
33:04 you know,
33:04 since I'm speaking to people at the World Bank.
33:07 Um,
33:08 uh,
33:08 there's a sense in which I think people who,
33:11 who,
33:11 who cut their teeth every day running randomized control trials kind of know,
33:16 you obviously don't want your comparison group to be treated,
33:19 right?
33:19 But that doesn't mean that you can't use as a comparison group,
33:22 a group that hasn't been treated yet,
33:25 right?
33:25 That,
33:25 that might go to
33:26 school in the future,
33:27 but that's not at that moment going to school,
33:30 right?
33:30 And it's a not yet.
33:31 Treated group.
33:32 So what happens is we have
33:34 Group K
33:35 treated,
33:35 but in this little period of time,
33:38 what's weird,
33:38 the OLS kind of knows to do this.
33:40 It's like
33:41 the,
33:41 the mean of why in this in-between period,
33:45 in between when K and L were treated,
33:47 so you got average here and average here.
33:49 An average for the control group,
33:51 it's a treated group,
33:52 but they're in,
33:53 they haven't yet gotten treated,
33:54 right?
33:55 And so they're clean.
33:56 That's what uh Aaron Dube always calls them.
33:58 He says it's a clean control,
34:00 all right.
34:01 This one is not a clean control.
34:03 This is a dirty control,
34:04 right?
34:04 So here's our
34:05 treatment group,
34:06 Group L,
34:07 average,
34:08 average,
34:09 and here's our comparison group,
34:10 Group K.
34:11 But look,
34:12 Group K was already treated,
34:14 and this is the sin.
34:15 This is the mortal sin
34:17 uh
34:17 of any difference in differences design.
34:19 I mean,
34:19 it would be,
34:20 it's,
34:20 it's,
34:21 it's,
34:21 uh,
34:22 it's not a violation of SUTTVA,
34:24 uh,
34:25 because there's not really a spillover.
34:27 But it's almost kind of like
34:29 for the same reason that SAA is a problem,
34:32 you know,
34:32 SATVA violations are an issue.
34:34 Uh,
34:35 it,
34:35 it's kind of like
34:36 in the,
34:36 it's not exactly the same,
34:38 but,
34:38 well,
34:38 it's not the same as that,
34:39 but it's,
34:40 it's like if you already know the SUTVA violation,
34:42 kind of be thinking a little bit like,
34:44 I don't want to be comparing people to already treated units
34:47 because it's going to screw up the calculation in some way,
34:49 and Goodman Bacon is going to show exactly how it screws it up.
34:53 All right.
34:53 So what he does is he uses the Friswald level theorem,
34:56 all right,
34:57 to basically just start hammering out that regression
35:00 coefficient with each of those fixed effects.
35:02 So it's like,
35:03 when you put in the year fixed effects,
35:05 you can residualize
35:06 the,
35:07 the treatment variable
35:08 when you put in.
35:10 The other uh panel fixed effects,
35:11 you do another residualization
35:13 and you kind of can calculate exactly through those sets of differences,
35:18 precisely what,
35:20 uh,
35:20 you know,
35:21 what that
35:22 panel fixed effects regression coefficient was always doing,
35:25 what it was always doing,
35:26 and here's what it was always doing,
35:27 and I'll show you in a simulation that it works.
35:30 So
35:30 it was basically calculating,
35:33 right,
35:33 you run a regression onto a dummy.
35:36 Using that panel fixed effects specification,
35:39 you get a number.
35:40 Here's the number you get.
35:42 Right?
35:42 This is the cake,
35:43 right?
35:44 And that cake is a weighted average over a bunch of ingredients,
35:48 OK?
35:49 A certain number of eggs,
35:50 certain amount of water,
35:51 certain amount of flour,
35:52 right?
35:52 Where these are
35:53 the certain amount.
35:55 And these are the ingredients,
35:57 all right?
35:58 So it's like
35:58 1 cup flour,
36:00 1 cup
36:01 flour,
36:02 you know,
36:03 2
36:04 eggs,
36:05 right?
36:06 It's that,
36:06 it's that kind of thing,
36:07 all right.
36:08 But
36:08 each of these
36:10 has
36:11 different memories,
36:12 right?
36:12 Different memories.
36:13 And so
36:14 the untreated.
36:16 2 by 2s
36:17 are causal effects plus parallel trends.
36:20 Any time,
36:21 in fact,
36:22 here's the deal.
36:23 Anytime you calculate a 2 by 2
36:26 where you use as an untreated unit,
36:28 you can do that.
36:29 You can go through those steps that we went through earlier,
36:32 and it's always going to be
36:34 ATT plus parallel trend bias,
36:36 right?
36:37 And that's why.
36:39 This one here,
36:41 this one here,
36:41 and this one here are just.
36:44 ATT plus parallel trend.
36:47 ATT plus parallel trends,
36:49 but when,
36:49 when Andrew also did
36:52 the first of all,
36:53 when he did the first wall level theorem,
36:54 all right,
36:55 application to this two-way fixed effects coefficient,
36:58 uh,
36:58 or to,
36:59 to the,
36:59 uh,
36:59 I guess to the,
37:00 to the estimator,
37:01 which gives this coefficient,
37:02 he also found
37:04 That this
37:05 red one.
37:07 was the case.
37:08 And notice that this red one
37:09 is got a causal effect inside it,
37:12 it's got a parallel trend effect inside it,
37:14 but it's also got
37:16 This,
37:17 a dynamic treatment effect bias,
37:21 and notice
37:22 it's negative.
37:23 I want to dig into this a little bit,
37:24 but when we put all these together,
37:26 here's what we get.
37:28 That the two-way fixed effects coefficient that you,
37:31 you know,
37:31 you,
37:31 you regress outcomes onto a treatment dummy,
37:34 where you've got differential timing,
37:36 right?
37:36 You got 3 groups,
37:37 Group K,
37:37 Group L,
37:38 Group U.
37:39 What does that coefficient mean?
37:41 What are its memories?
37:42 Uh,
37:43 weighted average treatment effects,
37:45 all right?
37:45 The ones we care about,
37:46 these weights are weird,
37:47 but let's put that aside for this presentation.
37:49 So you've got this weighted average,
37:51 treated untreated.
37:52 You got weighted average parallel trends.
37:54 You can see it right here,
37:55 parallel trends,
37:56 parallel trends,
37:57 parallel trends,
37:58 right?
37:58 So they're all in there.
37:59 Here's the ATT,
38:00 the ATT,
38:01 the ATT.
38:02 Here's the,
38:03 the weights,
38:04 the weights.
38:05 Here's some more weights,
38:06 all right?
38:06 So that's why these are
38:07 weighted averages over causal parameters and weighted averages over biases,
38:13 right?
38:13 But then you've got
38:15 This,
38:15 uh,
38:16 you know,
38:16 inside this,
38:17 you've got this
38:18 weighted
38:19 average Delta ATT term,
38:22 right?
38:22 And that's the dynamics.
38:24 And notice what's going on.
38:26 First of all,
38:28 two-way fixed effects is being biased
38:30 towards 0,
38:32 so long as
38:34 the sign of this one
38:36 is kind of the sign of that one,
38:38 right?
38:38 So if it's like,
38:38 if you find positive treatment effects that are getting more positive over time,
38:42 that's like a positive
38:44 plus a negative,
38:46 right?
38:46 Because that'd be a positive
38:47 minus a positive.
38:49 And so,
38:50 what two-way fixed effects is usually gonna be,
38:52 what it's usually gonna be is just um
38:56 Too small
38:57 in absolute value.
38:58 It's just gonna be too small in absolute value,
39:00 but
39:02 Because
39:03 it's linear,
39:04 right?
39:04 It's not one of these things that's like,
39:06 you know,
39:06 the bias is approaching zero but never reaching it.
39:10 That's not what's going on.
39:11 Uh,
39:11 it's linear
39:12 in all these three terms.
39:13 This coefficient is linear in all these 3 terms.
39:15 And why do I say that?
39:16 Because if this is relatively small,
39:19 Compared to that
39:21 dynamic,
39:22 then the sign will flip.
39:24 And so,
39:25 this is one of the things
39:26 that you first saw
39:28 with Borisak Yarvil 2016 when they posted it to the working paper series SSRN.
39:34 They were like,
39:35 it does not have a no sign flip property,
39:37 which is,
39:37 that's why
39:38 I think,
39:39 I think there were two reasons why
39:41 uh
39:42 Goodman Bacon's paper kind of freaked everybody out.
39:45 First was
39:46 parallel trends is not sufficient.
39:48 It's necessary,
39:49 but it's not sufficient with differential timing.
39:51 Well,
39:51 that's scary because
39:52 basically,
39:53 every single person.
39:56 Uh,
39:56 up to that point had been saying,
39:59 uh,
39:59 they've been doing their
40:00 parallel trends,
40:01 bullet point in their slots,
40:03 you know,
40:04 and that was all they would do.
40:05 This one,
40:06 this one was,
40:07 uh,
40:08 nobody knew about this,
40:09 and this one is scary because they would oftentimes
40:11 kind of like be interested in the dynamics,
40:14 right?
40:14 So if you have dynamics,
40:16 it was part of the paper,
40:17 but if you have dynamics,
40:19 then you have bias and your sign might flip,
40:22 OK.
40:23 So here's some simulated data.
40:25 All right,
40:25 let's assume 1000 firms,
40:27 40 states,
40:29 25 firms per state over 30 years.
40:31 That gives us 30,000 observations in 4 groups.
40:34 We're gonna have here a two-way fixed effects
40:37 equation that is producing
40:40 uh Y 0,
40:42 right?
40:42 It's gonna be this one.
40:43 It's gonna be this one right here.
40:45 Um,
40:46 Y 0.
40:47 It's going to be determined in every period for every unit
40:50 by
40:51 a level fixed effect plus,
40:54 uh,
40:54 basically,
40:55 time,
40:56 time fixed effects in every period
40:58 and some error term.
40:59 And this error term is going to be IID.
41:00 I'm just gonna have it be IID.
41:02 So,
41:02 what this does is it does two things.
41:04 This right here
41:06 gives us parallel trends and this whole equation.
41:09 Gives us what's called no anticipation.
41:12 Because what it means is that
41:14 uh I don't get treated.
41:15 I,
41:16 my,
41:16 my treatment effect does not occur until I get treated.
41:20 All right.
41:21 And you could imagine that being violated,
41:23 you know,
41:23 if I
41:24 know that Baylor's gonna give me a raise in the fall,
41:26 maybe I go out and get a,
41:28 uh,
41:29 you know,
41:29 a,
41:29 a new,
41:30 a new outfit today,
41:32 right?
41:32 But that's,
41:33 that's anticipation of a treatment effect.
41:35 And so,
41:35 Diff and Diff says,
41:37 I also got to rule out no anticipation effects,
41:39 which nobody also knew about.
41:41 So here's what this looks like.
41:43 These are called the group time ATT parameters,
41:46 all right?
41:47 And what they are,
41:48 are
41:49 uh dynamic treat,
41:50 I,
41:50 I've got here dynamic treatment effects,
41:52 but the group time ATT is just the cells.
41:56 Within each
41:57 column year,
41:58 right?
41:59 So,
41:59 I've got four groups,
42:00 86,
42:01 92,
42:02 98,
42:02 04,
42:04 because remember,
42:04 I've got
42:05 data goes from 80 to 09,
42:07 4 groups,
42:09 all right?
42:10 And I'm just gonna,
42:11 I'm gonna look at the average,
42:12 and then I've got like
42:13 1986 having dynamic treatment effects.
42:15 They all have dynamic treatment effects,
42:17 all right?
42:17 They all have dynamic treatment effects.
42:19 But
42:20 the ATT,
42:21 OK,
42:22 the ATT is an 82,
42:24 and how did I get that?
42:25 All I did was
42:26 I added together all these colored cells
42:30 and then divided by 60 because there were 60
42:33 of the cells,
42:34 all right?
42:35 So you already kind of see like,
42:37 when I lay out,
42:38 like if I said to my daughter,
42:39 Wren,
42:40 you know,
42:40 she's a freshman,
42:41 and I say,
42:41 hey,
42:41 Wren,
42:42 uh,
42:43 what's the average of all these cells?
42:45 She would know to take a uniform weight of 1/60.
42:49 Right.
42:50 And that would probably be to me,
42:53 if I was going to a policymaker,
42:55 I would say
42:56 this is kind of an interesting parameter.
42:58 It's the weighted average overall
43:00 individual ATTs,
43:01 group time ATTs in every single period.
43:03 You could imagine doing it other ways,
43:05 but
43:06 that's not the craziest way to do it,
43:08 all right?
43:10 OK.
43:11 So,
43:12 the truth
43:13 is in 82.
43:14 If I estimate
43:16 that data,
43:17 30,000 observations rested onto
43:21 Uh,
43:21 firm fixed effects,
43:23 time fixed effects with that dummy,
43:26 all right.
43:26 I get a negative 6.69.
43:29 I,
43:29 I get the sign flip.
43:31 Right?
43:32 And uh why?
43:33 Why is it happening?
43:35 Because of extreme dynamics.
43:36 These dynamics here are really big relative to that 82,
43:40 right?
43:41 Relative to that 82.
43:42 So,
43:42 here's what's going on.
43:43 When I use Bacon's decomposition,
43:46 what's it,
43:46 what's really cool about Bacon's decomposition is that it uses data.
43:50 It's almost like Bacon's decomposition
43:53 is like a um
43:55 Uh,
43:55 it's like a magnifying glass,
43:56 like a,
43:56 a,
43:57 a magnifying glass that lets you go into the cake
43:58 and figure out exactly how many eggs were used,
44:01 exactly how much flour.
44:02 And this is what it's saying,
44:03 it's like,
44:04 it backs up the recipe that went into that -6.69.
44:08 And it basically says,
44:11 See,
44:11 there is no here in in that data,
44:13 there is no never treated group.
44:16 You can see that right here.
44:18 There's no never treated group.
44:19 A never treated group would be a column of zeros.
44:21 I don't have one.
44:22 So that means
44:23 I only have
44:25 the good,
44:26 not yet treated 2 by 2s and the bad.
44:29 Already treated 2 by 2s.
44:33 And they're each being weighted
44:35 the same 0.5,
44:37 but one of them is positive,
44:39 one of them is negative,
44:40 and in fact,
44:42 that,
44:42 that,
44:43 that second one
44:44 is
44:45 bigger in absolute value than the first one,
44:46 and then mechanically,
44:48 it's causing the sign to flip,
44:49 it's causing the sign to flip.
44:51 I'm having the,
44:51 I'm having,
44:52 it's almost like,
44:53 you know,
44:53 in my memory example,
44:55 uh,
44:55 I'm having like,
44:56 you know,
44:56 psychosis.
44:57 I'm not even remembering any of the things right.
44:59 It's not even like I'm kind of getting it wrong.
45:01 I'm like having
45:02 completely false memories,
45:04 right?
45:04 So,
45:05 the,
45:05 the problem was this late to early 2 by 2,
45:09 OK,
45:09 under heterogeneous treatment effects,
45:11 but the Bacon decomposition is kind of suggestive
45:14 that the problem is there.
45:15 It's not really conclusive because
45:18 Ultimately,
45:19 you know,
45:19 you don't know that these are wrong.
45:21 If you have constant treatment effects,
45:22 these aren't wrong.
45:23 So a high 0.5 doesn't tell you it's wrong.
45:27 It's just
45:29 You know,
45:29 it's
45:30 It's just breaking down the recipe.
45:31 It's not telling you what the true memories are.
45:34 So let's look at
45:36 what
45:37 you can do instead of two-way fixed effects.
45:39 And let's figure out a little bit,
45:40 at least mechanically,
45:41 I'm not gonna go through proofs or anything,
45:43 but let's just kind of go through mechanically
45:45 what the heck these things are doing,
45:46 all right?
45:48 So there's,
45:48 there's basically two types
45:51 of alternatives to the two-way fixed effects model.
45:55 They're,
45:55 but they're both
45:57 imputation
45:58 methodologies,
46:00 right?
46:00 And,
46:00 but,
46:01 but let's back up for a second.
46:03 If we're estimating causal effects.
46:06 And in causal effects,
46:08 every unit
46:09 is missing its counterfactual,
46:11 that anytime I get an unbiased estimate
46:14 of some aggregate parameter,
46:16 it must be true I've imputed
46:18 the counterfactuals because
46:20 the idea of a treatment effect is based on a counterfactual.
46:24 So if I'm getting
46:25 a treatment effect of 0.07
46:28 returns
46:29 for every year of schooling,
46:31 that must mean that I have a,
46:34 have actually imputed.
46:36 A
46:37 missing counterfactual cause that's,
46:38 that's what the meaning
46:40 of an unbiased parameter means,
46:42 right?
46:42 Well,
46:42 but,
46:43 so that,
46:44 that's kind of interesting to kind of keep in your mind
46:46 cause we don't always think that way,
46:47 that we've actually imputed
46:49 missing counterfactuals.
46:50 And the reason we don't think that way
46:52 is because when I run a regression,
46:54 I don't get
46:55 a new column of variables
46:57 called Y zero,
46:59 right?
46:59 Uh,
46:59 it doesn't actually make
47:02 New data.
47:03 But see,
47:04 some,
47:05 some methods do.
47:06 Some methods do make
47:08 new columns of data,
47:09 various forms of like nearest neighbor matching.
47:12 will do that kind of thing,
47:13 all right?
47:14 Synthetic control
47:15 will do that kind of thing,
47:16 or at least it'll kind of calculate weights
47:18 that can kind of be used to sort of
47:20 construct
47:21 counterfactuals,
47:22 right?
47:22 So sometimes it happens
47:24 explicitly,
47:25 right?
47:26 And sometimes it happens implicitly.
47:27 And this is a quote from Hito Emins and Don Rubin in their 2015 book.
47:32 At some level,
47:33 all methods for causal inference are imputation methods.
47:36 It's just that some do it more explicitly than others.
47:40 Right?
47:41 RCTs.
47:43 They don't really seem to be imputing counterfactuals,
47:46 but under independence,
47:47 it's as though they are.
47:49 It's as though they are,
47:51 all right?
47:51 And these new solutions,
47:53 I like to think of them
47:54 as either the implicit.
47:57 Or the explicit
47:59 imputation methods,
48:00 all right?
48:02 So here are the assumptions that are used in the ones I'm gonna talk about,
48:06 OK?
48:06 The ones I'm gonna talk about.
48:08 The um
48:10 Uh,
48:10 5 or so.
48:11 I'm not gonna go into a lot of detail on all of them just for,
48:13 for the sake of time because the purpose is really
48:16 bird's-eye view help you kind of feel like you've got,
48:18 uh,
48:19 you know,
48:19 like some breadcrumbs through this whole thing so that
48:22 you just sort of have in your mind a compass
48:24 for walking through them.
48:25 So here's,
48:26 here's the five assumptions.
48:27 Now,
48:27 a few of these are kind of trivial,
48:29 all right?
48:30 Uh,
48:30 assume you have panel data,
48:32 assume you have repeated cross-section data,
48:34 right?
48:35 Now,
48:35 I know that in developing countries,
48:37 you know,
48:37 panel data is not very common,
48:39 so you can use
48:41 repeated cross-section data with difference in differences
48:43 so long as you are willing to assume
48:46 modularity.
48:47 What is
48:47 Modularity.
48:48 I talked about it in my book actually,
48:50 when I talk about this Napster example,
48:53 uh,
48:53 and using the consumer expenditure survey,
48:56 uh,
48:57 the Consumer expenditure survey is
48:58 a repeated cross-section of American households
49:01 and it's got data on their spending and it had data on their,
49:05 um,
49:06 internet use and it had data on their,
49:09 um,
49:11 Music spending,
49:12 right?
49:13 And you kind of look at like a kind of a,
49:15 a little bit of like a dip dip.
49:16 It's kind of like a,
49:17 it was more of like a
49:20 It's like a continuous treatment,
49:22 different diff group,
49:23 sort of,
49:23 it's a little weird.
49:24 But anyway,
49:25 you've got like an internet group and a
49:26 non-internet group and you can sort of see in this paper by
49:29 this one author
49:31 that uh
49:31 he had this like correlation between as Napster grew,
49:35 um,
49:35 you would see fewer and fewer people
49:38 uh that use the internet
49:40 buying.
49:42 Uh,
49:42 albums,
49:43 spending money on albums.
49:44 But the,
49:45 the thing is,
49:45 you could also see that
49:47 over the sample,
49:48 the demographics
49:49 of the sample was shifting.
49:51 They were getting younger and younger
49:52 over this particular period of time,
49:54 and they were also getting poorer and poorer
49:56 in the sample.
49:57 That's a violation of modularity.
49:59 Modularity is kind of saying that none of that
50:02 sample selection
50:03 is kind of happening.
50:04 Right?
50:05 So if you can,
50:05 if you,
50:06 if you have that,
50:07 you can use it.
50:08 You can always use difference in differences.
50:10 The,
50:10 the,
50:10 the construction of it is going to be
50:11 a little bit different in terms of calculating the
50:14 um confidence intervals,
50:15 but it,
50:15 it's also possible to do it.
50:17 So,
50:18 you know,
50:18 assuming you have data,
50:19 all right,
50:20 uh,
50:21 parallel trends,
50:22 you have to assume conditional parallel trends if you have covariants,
50:24 but if not,
50:25 you,
50:25 you don't have to.
50:26 If you have covariants,
50:27 you've got to assume common support.
50:30 You also need this no anticipation assumption,
50:32 all right,
50:32 which meant
50:35 Zeroes pre-treatment.
50:37 No treatment effect
50:39 before the treatment period,
50:40 right?
50:40 Now,
50:40 that,
50:41 that may or may not be trivial,
50:43 right?
50:44 And then the treatment cannot,
50:45 if it turns on,
50:46 can't turn off,
50:47 right?
50:47 Now,
50:47 there is a paper
50:48 by the Chase Martin and Dehoful 2020 AER
50:51 that does allow for that.
50:52 It's the only one that I know that allows for it.
50:55 Um,
50:55 it'll identify a particular treatment parameter around that switching,
51:00 but we're gonna look at
51:01 Uh,
51:02 the more general case,
51:03 uh,
51:03 the more common case
51:05 of a,
51:05 of,
51:05 of a non-switching scenario.
51:08 And this is the Calloway and Santana
51:10 estimator
51:12 using the inverse probability weight.
51:13 So I just want to show you a couple of things.
51:15 In here,
51:16 our hour before and after.
51:19 Values of Y,
51:21 right?
51:21 Cause remember what,
51:22 remember what different,
51:23 different equations always are doing
51:24 after minus before for the treatment group,
51:26 after-4 for the comparison group,
51:28 after
51:29 uh calculating means
51:32 of an outcome.
51:32 So here are the outcomes,
51:34 here's the means.
51:35 So this is
51:36 after,
51:36 right here,
51:37 it's after minus before,
51:39 right?
51:39 So this is like G means the year of treatment.
51:42 And so this would be like baseline prior to treatment and under no anticipation,
51:46 it's not treated.
51:47 So it's like a,
51:48 you know,
51:48 it's a true baseline
51:49 prior to treatment.
51:50 So I've got this long difference,
51:53 right?
51:53 So I've,
51:53 I,
51:53 I'm now like calculating treatment effects way out in time or immediately,
51:57 right?
51:58 And under these five assumptions,
51:59 it'll be unbiased,
52:01 right?
52:01 So I've got here,
52:02 average Y
52:04 for the treatment group.
52:05 So this here is,
52:06 when it's a 1,
52:08 All right,
52:09 when it's a treatment group,
52:10 that'll be a 1,
52:11 and that'll be a 0,
52:12 cause that's gonna be the comparison number.
52:14 So look what you're getting.
52:15 When you're calculating.
52:18 The different equation.
52:20 4
52:22 The differential timing scenario,
52:24 all right.
52:25 With Calloway Santana,
52:26 you don't
52:27 wait.
52:29 The
52:30 treatment group.
52:30 You,
52:31 you wait by this like share of the units that are in the treatment group,
52:34 but everybody's getting weighted.
52:36 Uh,
52:36 well,
52:36 yes,
52:36 you,
52:37 you're gonna do that weight,
52:37 but you're not gonna do this like propensity score weighting,
52:40 right?
52:41 So you're gonna get one over the share of units
52:43 in the treatment group and that's gonna be weight,
52:45 weight.
52:46 All right.
52:46 So you weighted
52:48 after -4
52:49 and then for the comparison group,
52:50 look at what you're doing.
52:52 So this is way,
52:52 this is the way that inverse probability weighting
52:55 is doing
52:57 what
52:58 Embi and Rubin said,
52:59 which is an implicit imputation.
53:02 What it's doing is
53:03 it's weighting up
53:06 control group units
53:07 that have high propensity scores,
53:09 and it's weighting down control group units that
53:12 have low propensity scores because the propensity score.
53:15 is basically capturing
53:17 the pro the selection and the treatment based on covariants.
53:20 If there's no
53:21 covariance,
53:22 it's just the frequentest weight.
53:23 It's the share of units.
53:25 It's like if I just pull
53:26 a random unit out of the data set,
53:28 what's the probability?
53:30 Uh,
53:30 what's the probability that they're in
53:32 the treatment group?
53:33 So I'll have here like maybe,
53:35 you know,
53:36 I have 4 groups,
53:37 1/4.
53:39 Uh,
53:40 or,
53:40 but if I have covariants,
53:41 then it's going to be some like covariantate adjusted
53:44 propensity score,
53:45 and I'm gonna wait this.
53:47 So the weights are different,
53:49 depending on if it's treatment group or comparison group,
53:51 but notice what you're doing.
53:53 The C here,
53:55 it will never be
53:57 an already treated unit.
53:59 Uh,
53:59 it will,
53:59 so,
54:00 so basically,
54:00 what happens is with Callaway-S Santana,
54:02 as it goes through
54:04 and calculates each of these,
54:07 all right,
54:07 calculates each of them.
54:08 Here it is over in Callaway-S Santana.
54:10 It's always looking to the right,
54:13 all right?
54:13 So it's always looking to the right.
54:15 And it's saying,
54:16 do I have any control group units over here?
54:18 So over here for 1986,
54:20 yes,
54:21 it's gonna use these
54:23 as my comparison group,
54:24 and it's gonna get it right.
54:26 When it gets over here.
54:28 It ignores 1986.
54:31 So when I'm trying to get to 1992
54:33 ATTs,
54:34 the group time ATTs,
54:35 I,
54:35 I dropped 1986 because they're already treated.
54:38 So the C
54:40 that's happening here,
54:41 it's just a simple trick of the data.
54:43 It's just through subsetting of the data.
54:44 It's actually just,
54:45 you know,
54:45 it's kind of interesting.
54:46 It's,
54:47 it's like trivial
54:49 programming.
54:50 It's calculating it,
54:51 all right?
54:52 Just simple steps.
54:52 So every time I got zeros to the right,
54:55 I can calculate it.
54:56 So look what happens with Calie Santana.
54:58 Calloway Santana
55:00 cannot calculate two things.
55:03 First of all,
55:04 it cannot calculate
55:06 any of the late
55:08 Group time ATTs for my
55:10 86,
55:11 92,
55:11 98 04 groups.
55:13 Why?
55:13 Because at that period,
55:15 I don't have any,
55:16 any,
55:16 any untreated units left,
55:17 left.
55:18 I don't have any never treated,
55:19 and I don't have any not yet treated.
55:21 So that's,
55:21 that's kind of like what's interesting is that two-way fixed effects
55:25 did not stop.
55:27 Two-way fixed effects did not look at the situation and say,
55:30 well,
55:30 I'm worried because two-way fixed effects does not care
55:33 if a unit
55:34 is treated or not.
55:36 two-way fixed effects loves variation in the data.
55:40 Any variation in the data,
55:41 any comparison it can take,
55:42 it can make,
55:43 it will exploit it.
55:44 But that's not what causal inference is about.
55:46 Causal inference is always about selective comparisons,
55:50 appropriate
55:51 counterfactuals,
55:52 and there are no
55:53 appropriate counterfactuals for these groups in these periods.
55:57 So I cannot calculate
55:59 these treatment effects for these years,
56:01 nor can I calculate any
56:03 of the 04.
56:05 ATTs ever,
56:07 right?
56:08 So that means that
56:10 the feasible
56:12 ATT
56:13 is different.
56:15 I thought it was an 82
56:17 because that was the average over everything,
56:18 but it's like,
56:20 I don't even know if that's really the right way to communicate anymore because
56:22 I'm gonna get unbiased estimates of each group time ATT but it'll never be 82
56:27 because I'm not actually working with those specific group time ATTs anymore.
56:32 The,
56:33 the feasible one
56:34 is a 68,
56:36 right?
56:36 So if I just do these,
56:38 oops,
56:38 if I just do,
56:39 uh,
56:40 you know,
56:41 these right here,
56:42 this triangle right here,
56:43 um,
56:44 I can't remember how many there are.
56:45 There's like,
56:45 uh,
56:46 however many,
56:46 it's like,
56:47 I think I have 24 less or something.
56:49 So,
56:49 um,
56:51 Uh,
56:51 or maybe 24 total.
56:52 So I get 68.3.
56:54 Well,
56:56 Callo Santana gets it,
56:57 gets that right.
56:58 He just doesn't get this,
56:59 but that's not the target parameter anymore.
57:01 See,
57:02 so one of the things,
57:03 one of the things that
57:05 I think we are learning
57:07 from this explosion
57:09 of literature isn't just the estimator.
57:12 It's not just the estimator.
57:14 The other thing that they are forcing us to do,
57:17 that these econometricians have said we've got to be doing
57:20 is they're telling us now to say,
57:23 what
57:23 is
57:24 the parameter that you are trying to calculate,
57:27 right?
57:28 And going through this group time ATT concept,
57:32 what's really nice with it
57:34 is
57:34 how
57:35 it's
57:37 so clear
57:38 what
57:38 I'm trying to do and what I'm not trying to do.
57:41 Not only that though,
57:43 Look at how the weights make sense,
57:46 right?
57:47 They're uniform weights,
57:49 right?
57:49 They're weighted averages,
57:50 but what are the weights?
57:51 They're positive weights and everybody's being weighted the same.
57:53 Well,
57:54 what if I have a different
57:55 number of people in every group?
57:57 What if I've got 1000 people in this one,
57:59 but I've only got 500 in this one?
58:01 Well,
58:01 I could imagine
58:03 probability weights
58:04 that weighted those differently.
58:06 Calloway Santana lets you incorporate those.
58:09 Right?
58:09 So it's really kind of nice about Calloway-S
58:11 Santana is that compared to two-way fixed effects,
58:14 Callaway-S Santana
58:16 lets you
58:17 pick
58:18 your weights,
58:19 and two-way fixed effects as a model
58:22 forces
58:23 weights on you,
58:24 the variance weights,
58:25 which I didn't get into,
58:26 but they're not,
58:26 they're not particularly
58:29 interesting to say the least.
58:30 They're weird.
58:31 It would be just like
58:32 if all of a sudden these things got weighted more
58:35 than these at the end.
58:36 It's all kinds of weird things like that.
58:39 All right.
58:40 So what I did was
58:41 I reran
58:42 to a fixed effects on
58:45 this.
58:47 1980 to 2003.
58:49 I thought that would be fair,
58:51 right?
58:51 Cause that's the full sample,
58:53 and I know the truth is 68.34.
58:55 Am I still gonna get a negative?
58:56 And look what happened,
58:57 I don't get the negative.
58:59 You're not getting the negative anymore.
59:00 Now,
59:00 it's still got the attenuation bias.
59:03 It's still got the attenuation bias,
59:05 but
59:06 it's not negative.
59:07 Now,
59:07 why is that?
59:09 Think back,
59:10 think back to the bacon decomposition.
59:12 In the bacon decomposition that we did with this simulated data.
59:16 Right?
59:17 There was no
59:18 never treated group.
59:20 There was only
59:24 Oops.
59:24 There was only these two,
59:26 this group not yet treated,
59:27 and this group.
59:29 That already treated
59:30 and because of just the structuring of the data,
59:32 the weights were exactly the same,
59:33 but there were only two types of 2 by 2.
59:35 Well,
59:37 necessarily,
59:38 all right,
59:39 the weights on this one have got to fall
59:42 if I put in another never treated group,
59:45 which is what I now have
59:48 if I'm only working with 81 to 03,
59:51 right?
59:51 So you can kind of start to see how things are like subtly changing.
59:56 They're kind of subtly changing.
59:57 As I introduce
59:59 the never treated group,
1:00:01 both the dynamics are changing
1:00:03 and
1:00:04 like,
1:00:06 Things with the weights are changing.
1:00:07 So like the actual 2 by 2s are changing,
1:00:09 but also the weights are changing and
1:00:12 the types of 2x 2s.
1:00:14 The types of problematic 2x2s are falling,
1:00:17 right?
1:00:18 So it's interesting that we see
1:00:20 a return
1:00:22 to positive effects,
1:00:23 but
1:00:27 Still got the dynamics.
1:00:29 So it's still biased downward,
1:00:31 which is why.
1:00:35 It's biased below 68.
1:00:38 It's down to 26.881,
1:00:40 but you know,
1:00:40 that might seem like good news.
1:00:43 That might seem like good news,
1:00:44 but,
1:00:45 I mean,
1:00:46 it's almost like,
1:00:47 again,
1:00:47 I go in my memory,
1:00:48 memory,
1:00:48 memory metaphor.
1:00:49 It's like,
1:00:50 sure,
1:00:50 it's nice that you're not having false memories,
1:00:53 right?
1:00:54 But it's not great that you can't remember
1:00:57 everything,
1:00:57 right?
1:00:57 That you're,
1:00:58 it's,
1:00:58 it's great that you're not like thinking that you,
1:01:01 you know,
1:01:02 some people struggle with,
1:01:03 you know,
1:01:03 psychosis and they like,
1:01:04 you know,
1:01:05 remember things that aren't true.
1:01:06 I mean,
1:01:06 that's a horrible
1:01:08 struggle,
1:01:08 but some people,
1:01:09 you know,
1:01:09 they just forget,
1:01:10 they're forgetful,
1:01:11 and two-way fixed effects
1:01:13 here is forgetful.
1:01:14 It's forgetting a whole,
1:01:15 you know,
1:01:15 35,
1:01:16 42
1:01:18 units on the treatment effect,
1:01:19 and that's not helpful,
1:01:20 but look,
1:01:21 Cie Santana is getting them right,
1:01:22 right?
1:01:24 So
1:01:25 Randomization,
1:01:26 it's like I love to say this over and over because I,
1:01:28 this is something that at least right now it's kind of been a preoccupying my mind.
1:01:32 This whole thing that Ruben said in a conference,
1:01:34 he said,
1:01:35 uh,
1:01:35 randomization gives us,
1:01:36 he,
1:01:36 he said something to this effect,
1:01:37 he's like randomization gives us confidence
1:01:39 because we know how the science works
1:01:41 and there is no science of parallel trends,
1:01:42 but economists never give up.
1:01:44 That's one of the things I,
1:01:45 I feel like I really,
1:01:47 uh,
1:01:47 appreciate about,
1:01:48 uh,
1:01:49 you know,
1:01:49 the movement towards design-based cause of inference is,
1:01:52 is like this movement towards
1:01:54 You know,
1:01:55 defending
1:01:56 the assumptions,
1:01:57 right?
1:01:57 It's like it's all about,
1:01:58 can,
1:01:59 I'm not,
1:01:59 you know,
1:01:59 cause like the old days,
1:02:00 I was,
1:02:01 I,
1:02:01 I interviewed Orly Ashenfelter,
1:02:03 uh,
1:02:04 a couple of weeks ago for my podcast and
1:02:08 He told me,
1:02:08 he said,
1:02:09 I remember the days
1:02:11 when papers wouldn't even tell you what their instruments were,
1:02:15 you know,
1:02:15 that's,
1:02:16 I could,
1:02:17 I mean,
1:02:18 that is not the good old days,
1:02:19 right?
1:02:20 It's like,
1:02:20 you know,
1:02:21 not even putting that,
1:02:22 you're like,
1:02:23 using instruments.
1:02:23 I'm not even gonna tell you what they are.
1:02:25 I'm gonna have a table of coefficients.
1:02:26 It's,
1:02:27 it's just ridiculous.
1:02:28 It's just,
1:02:29 just,
1:02:29 you know,
1:02:30 absurd,
1:02:31 probably
1:02:32 the,
1:02:32 uh,
1:02:33 the thing,
1:02:33 the,
1:02:33 the,
1:02:34 the like communications that were made.
1:02:37 Between the profession and
1:02:39 people that made actual decisions in real world,
1:02:42 you know,
1:02:42 it's just
1:02:44 Criminal,
1:02:45 but,
1:02:45 um,
1:02:46 uh,
1:02:47 you know,
1:02:47 so parallel trends like there's no,
1:02:48 there's no science of it,
1:02:49 but economists don't give up,
1:02:50 so what did they do?
1:02:51 So it's interesting what,
1:02:52 it's interesting what,
1:02:53 if you really kind of think really
1:02:55 closely about what empirical microeconomists have typically done
1:02:58 when they don't have randomization.
1:03:00 They even do it with randomization.
1:03:01 They do the balance test
1:03:02 on covariants.
1:03:03 But what do they do in,
1:03:04 in the quasi-experimental design?
1:03:07 What they typically do
1:03:08 is they look at that assumption
1:03:10 and they try to find
1:03:13 like
1:03:13 Uh,
1:03:14 a mimic of it.
1:03:16 In
1:03:17 the world.
1:03:18 It's not perfect,
1:03:19 but it's almost like a a mimic of it.
1:03:21 It's like,
1:03:22 it's like,
1:03:22 it's like Plato.
1:03:24 And the philosopher Plato and his thing about the,
1:03:28 the shadows on the,
1:03:29 on the cave wall,
1:03:30 right?
1:03:30 Like the identifying assumption
1:03:32 are these
1:03:33 ideals
1:03:34 out in the world that you can't see.
1:03:36 So you look over here at the wall and you're like,
1:03:38 OK,
1:03:38 well,
1:03:39 I wonder if
1:03:40 I were to look for parallel trends somewhere else.
1:03:43 And that was the creation of the event study,
1:03:45 right?
1:03:46 That,
1:03:46 that,
1:03:46 that was the event study.
1:03:47 For a long time,
1:03:47 they weren't doing event studies.
1:03:49 Uh,
1:03:49 and then,
1:03:50 uh,
1:03:50 event studies were,
1:03:51 you know,
1:03:51 they were doing them in finance,
1:03:52 but it means something different there.
1:03:54 Event studies were essentially a way for us
1:03:56 to have confidence that parallel trends held,
1:03:59 right?
1:04:00 And,
1:04:01 uh,
1:04:01 but it turns out two-way fixed effects was screwed up for them too,
1:04:04 at least in the,
1:04:05 the,
1:04:05 the classical
1:04:07 specification of it.
1:04:08 So I'm gonna go through this
1:04:09 son and Abraham
1:04:11 specification and try to give you some intuition
1:04:13 about why two-way fixed effects
1:04:15 is flawed,
1:04:16 all right?
1:04:17 I,
1:04:17 I,
1:04:18 in the,
1:04:18 in the dynamic specification is flawed
1:04:21 as an estimate of the coefficients
1:04:24 on um
1:04:26 On the leads and the lacks.
1:04:28 The leads uh are the pre-treatment coefficients.
1:04:32 And the lags are the post,
1:04:34 right?
1:04:34 So the leads
1:04:36 are meant to be,
1:04:37 uh,
1:04:38 you know,
1:04:38 the shadows on the wall
1:04:40 of parallel trends probably holding,
1:04:43 right?
1:04:43 So that's the idea.
1:04:44 It's like,
1:04:44 well,
1:04:44 they're,
1:04:45 they're probably holding because they're holding on the wall.
1:04:47 So that means they're probably holding,
1:04:49 uh,
1:04:49 in this period of time where I can't observe them,
1:04:52 right?
1:04:52 And then the,
1:04:53 the,
1:04:53 the lags are the dynamic treatment effects,
1:04:55 right?
1:04:56 They're both interesting in different ways.
1:04:58 OK.
1:04:59 So we're gonna have here,
1:05:00 uh,
1:05:01 again,
1:05:01 units are gonna be part of a group,
1:05:03 1986 group,
1:05:04 1992 group.
1:05:05 Uh,
1:05:06 they use the letter E though,
1:05:08 uh,
1:05:08 because Son and Abraham,
1:05:10 uh,
1:05:10 they were,
1:05:10 they were Amy Finkelstein,
1:05:12 Amy Finkelstein's class at MIT.
1:05:15 And
1:05:16 Dr.
1:05:16 Finkelstein had
1:05:17 made a comment in class about event studies,
1:05:20 and Sophie's Sun and Sarah Abraham
1:05:23 wrote this paper.
1:05:24 It was just a phenomenal paper.
1:05:25 I highly encourage you to read it.
1:05:26 I have an interview with
1:05:28 Sophie's Sun
1:05:29 on my sub stack.
1:05:30 She's absolutely brilliant,
1:05:32 and
1:05:33 they use this.
1:05:34 They use this notation.
1:05:35 They call it a cohort,
1:05:37 not a group.
1:05:38 Callie C call it a group.
1:05:38 It's the same thing.
1:05:41 Let's talk about benning.
1:05:42 So you have leads and lags,
1:05:44 which would be like,
1:05:44 you know,
1:05:45 the T minus 1,
1:05:46 the T minus 2
1:05:48 relative time period,
1:05:49 the T +1,
1:05:49 the T + 2
1:05:51 relative time period.
1:05:52 I'll show you an example of what I mean,
1:05:54 but you can imagine binning them
1:05:56 and like saying something like,
1:05:58 well,
1:05:59 I have,
1:06:00 you know.
1:06:01 Uh,
1:06:02 you know,
1:06:02 I have 20 lags,
1:06:04 uh,
1:06:04 20 leads,
1:06:05 and that's a lot.
1:06:06 Why don't I just estimate,
1:06:08 you know,
1:06:08 4 of them.
1:06:09 I'll just group them,
1:06:10 you know,
1:06:10 group 5 at a time,
1:06:11 right?
1:06:12 You could have maybe some reason that you want to do it.
1:06:13 Those are called bends,
1:06:15 right?
1:06:15 So you can have bends,
1:06:16 and we're gonna have bends be G,
1:06:17 and we're gonna have leads and lags be L,
1:06:20 and if they're bend,
1:06:21 then G is our focus,
1:06:23 and if they're not bend,
1:06:23 L is our focus,
1:06:25 right?
1:06:25 And the building block.
1:06:27 is again,
1:06:28 the group time ATT,
1:06:29 but they're gonna call it the CATT cohort,
1:06:32 cohort ATT.
1:06:33 But all it is,
1:06:34 all it is is the ATT
1:06:36 for a relative time period,
1:06:37 right?
1:06:39 OK,
1:06:40 I'm gonna skip this because it's a lot easier to do it.
1:06:42 So here's the assumptions,
1:06:43 parallel trends,
1:06:44 no,
1:06:44 no anticipation,
1:06:46 treatment effect,
1:06:47 homogeneity,
1:06:48 right?
1:06:48 Now,
1:06:49 if you have all three of those.
1:06:51 You have all three of those and you estimate this dynamic model,
1:06:55 all right,
1:06:56 mu,
1:06:57 the coefficient on a relative time period,
1:07:01 um,
1:07:02 uh,
1:07:02 you know,
1:07:02 relative to the,
1:07:03 when the treatment occurs,
1:07:05 uh,
1:07:05 will be an unbiased effect of the CATT for that period,
1:07:09 which is,
1:07:10 um,
1:07:11 Basically,
1:07:11 like,
1:07:12 you know,
1:07:13 this would be,
1:07:14 I'll,
1:07:14 I'll sort of talk you through it real quick,
1:07:16 what,
1:07:16 what they are.
1:07:17 We'll,
1:07:17 we'll,
1:07:18 we'll skip this for a second,
1:07:19 but I just want to kind of show you
1:07:20 what they are.
1:07:21 Here's what a relative time period is,
1:07:23 right?
1:07:23 So,
1:07:24 1986,
1:07:25 1992,
1:07:25 1998,
1:07:27 right?
1:07:28 1986,
1:07:29 in the year of treatment,
1:07:31 all right?
1:07:32 Let's get a 10,
1:07:33 an 8,
1:07:33 and a 6,
1:07:35 right?
1:07:35 10 plus 8 plus 6 divided by 3 is an 8.
1:07:38 So the CATT
1:07:42 For
1:07:43 the year of treatment
1:07:44 is an 8.
1:07:46 Then it goes 2016,
1:07:49 12.
1:07:50 Divided by 3,
1:07:51 that's a 16.
1:07:53 So,
1:07:54 the
1:07:55 T plus 1.
1:07:57 Mu,
1:07:58 the,
1:07:58 the T + 1
1:08:00 CATT is a 16,
1:08:02 all right,
1:08:02 and so forth,
1:08:03 right?
1:08:03 So we're gonna see if
1:08:04 Son and Abraham can get it and we're gonna see how two-way fixed effects works.
1:08:07 Now,
1:08:07 if we have
1:08:09 Those 3 things,
1:08:11 right?
1:08:12 If we have those 3,
1:08:13 well,
1:08:13 let's put aside if we have them or not,
1:08:15 right?
1:08:15 Let's say I run
1:08:17 this
1:08:17 regression right here.
1:08:18 I say I run this regression,
1:08:20 and I want to know
1:08:21 the,
1:08:22 the memories,
1:08:23 the theoretical meaning of the mu coefficients,
1:08:26 the population regression
1:08:27 coefficient on each of those leads and lags,
1:08:31 uh,
1:08:31 dummies.
1:08:32 What is it,
1:08:32 what is,
1:08:33 what does mu mean,
1:08:34 right?
1:08:35 And here's what mu means.
1:08:36 It's a weighted average.
1:08:38 Over 3 different kinds.
1:08:42 Of
1:08:43 treatment effects,
1:08:43 but the problem is,
1:08:44 here's what,
1:08:45 here's what's crazy
1:08:46 about the event studies,
1:08:47 and you got to kind of see it to just believe that this is absurd.
1:08:50 So
1:08:50 basically what you get
1:08:52 with the,
1:08:53 the two-way fixed effects regression of this.
1:08:56 Is
1:08:58 weights on um
1:09:00 weights that sum to one,
1:09:03 or let's just define the weights for a second.
1:09:04 We've got weights
1:09:06 in your own period,
1:09:08 right?
1:09:09 Weights
1:09:10 in other periods.
1:09:12 Weights in the periods that you dropped,
1:09:15 right,
1:09:15 because you have to drop
1:09:16 when you run this.
1:09:17 You got all these muses,
1:09:18 you got all these like,
1:09:19 you know,
1:09:19 ones can't Coca-Cola in a year,
1:09:20 you can't estimate all of them,
1:09:22 you got to drop some.
1:09:24 So each regression coefficient estimated with two-way
1:09:27 fixed effects is the sum of three things
1:09:29 weighted average,
1:09:30 CATT,
1:09:31 where the weight sum to 1.
1:09:33 Weighted average CATT from
1:09:36 the excluded
1:09:38 period,
1:09:38 which you're going to sum to
1:09:41 here,
1:09:41 they're going to sum to -1,
1:09:43 and the CATT from every other period that is in the model,
1:09:47 and those are going to sum to 0.
1:09:49 OLS is just
1:09:51 so
1:09:52 deep.
1:09:53 It's just such a deep model,
1:09:55 you know,
1:09:56 a lot of stuff is problematic for causal inference,
1:09:58 but the fact that it even is doing all this stuff
1:10:00 and we didn't really know it or I didn't know it is just so weird.
1:10:03 It's not
1:10:05 something that has been like really spelled out lots of times.
1:10:09 So,
1:10:09 when you have all three of them,
1:10:10 here's what happens.
1:10:13 When you have all three of them.
1:10:16 No anticipation treatment effects
1:10:19 go to zero,
1:10:20 right?
1:10:21 So,
1:10:21 those are gone.
1:10:23 So if you're looking at the mu.
1:10:26 If you're looking at the muse on like,
1:10:29 you know,
1:10:29 on this one,
1:10:30 T minus 2,
1:10:32 right?
1:10:32 Then,
1:10:34 Well,
1:10:34 T minus 2 is basically 000.
1:10:38 So that would mean,
1:10:39 uh,
1:10:40 you know,
1:10:41 it,
1:10:41 that,
1:10:41 that's a 0.
1:10:43 And any other
1:10:44 leads and lags,
1:10:45 any other leads and lags where no anticipation was holding,
1:10:49 this would be from all the other relevant time periods,
1:10:51 those are also zero.
1:10:53 But the problem is,
1:10:54 the problem is this one.
1:10:56 Under
1:10:58 Without homogeneous treatment effects,
1:11:00 without homogeneous treatment effects,
1:11:03 even the treatment effects from these other periods,
1:11:06 from these other periods
1:11:08 end up getting stacked
1:11:11 inside
1:11:12 the regression coefficients
1:11:14 for these pre and post-treatment
1:11:16 relative lacks,
1:11:18 OK?
1:11:19 So look what happens when I estimate two-way fixed effects,
1:11:22 when I estimate two-way fixed effects,
1:11:23 this is two-way fixed effects,
1:11:25 OK?
1:11:25 This is two-way fixed effects.
1:11:27 And look what I get.
1:11:28 All right,
1:11:29 I know
1:11:31 by construction,
1:11:32 I know by construction,
1:11:34 right,
1:11:34 that the data generating process
1:11:36 guaranteed
1:11:38 a zero treatment effect
1:11:40 pre-treatment.
1:11:41 So that means that truth is a zero and it's finding a gigantic treatment effect.
1:11:46 Why?
1:11:47 Because
1:11:50 All right.
1:11:51 Here's an example.
1:11:52 All right.
1:11:52 So here I look at mu-2.
1:11:54 All right,
1:11:55 mu -2.
1:11:56 So that,
1:11:56 that would be uh
1:11:58 this one right here,
1:11:59 should be a 0.
1:12:01 Mug2,
1:12:02 right?
1:12:03 Under no anticipation,
1:12:05 right?
1:12:05 All the CATTs for the negative periods are 0.
1:12:09 So that one's a 0.
1:12:12 That one's a 0,
1:12:13 that one's a 0.
1:12:14 So those are gone,
1:12:16 right?
1:12:16 So under no anticipation,
1:12:17 I kill every.
1:12:21 Pre-treatment lead
1:12:23 contaminating any
1:12:25 pre-treatment lead
1:12:26 or even any post-treatment.
1:12:27 It's like,
1:12:28 so,
1:12:28 uh,
1:12:28 if you have no anticipation already you're dealing
1:12:31 with a ton of the problems,
1:12:33 right?
1:12:34 And secondly,
1:12:36 right?
1:12:37 Secondly,
1:12:38 let's say that I have homogeneous treatment effects,
1:12:42 right?
1:12:42 So this is uh homogeneous treatment effects,
1:12:45 and I have not dropped period zero,
1:12:48 right?
1:12:48 So we've got here,
1:12:49 I'm gonna include Ben zero,
1:12:52 not going to drop it,
1:12:53 but I have homogeneous treatment effects,
1:12:55 right?
1:12:55 Well,
1:12:55 look,
1:12:59 Relative time periods belonging to some other band,
1:13:02 they sum to zero.
1:13:04 It'll be 15
1:13:06 minus 1/2.
1:13:07 That sums to zero,
1:13:09 and if this number is the same as that number,
1:13:12 then the whole thing sums to zero.
1:13:13 So I'm,
1:13:14 I'm doing good still.
1:13:14 So I got,
1:13:15 you know,
1:13:16 for the mu g2.
1:13:18 Right,
1:13:18 for the mu negative2,
1:13:20 I got
1:13:20 000.
1:13:23 So what's this one?
1:13:26 Well,
1:13:26 in some crazy world,
1:13:28 uh,
1:13:28 I decided to drop,
1:13:31 um,
1:13:31 as my,
1:13:32 you know,
1:13:32 to avoid my dummy variable trap,
1:13:34 I decided to drop a post-treatment.
1:13:37 Period.
1:13:38 And because it's dropped.
1:13:40 Um,
1:13:42 It's,
1:13:43 it's in the mute.
1:13:44 It's in the mute,
1:13:45 right?
1:13:46 Now,
1:13:46 here's the deal.
1:13:48 Under my scenario,
1:13:49 under my data generating process,
1:13:50 all right,
1:13:51 I'm gonna drop
1:13:52 the,
1:13:52 I'm gonna drop the,
1:13:53 um,
1:13:54 uh,
1:13:55 I'm gonna drop only the,
1:13:56 the pre-treatment period.
1:13:57 So those are gonna be 0,
1:13:59 OK?
1:13:59 Those are all gonna be 0 under my two-way fixed effects specification.
1:14:02 So those are 0s.
1:14:04 And every pre-treatment
1:14:06 average treated on the treated is also 0.
1:14:09 But
1:14:09 two-way fixed effects in any pre-treatment period
1:14:12 is going to be this weighted average.
1:14:15 Over
1:14:16 every single
1:14:18 other.
1:14:22 Like every single one of those.
1:14:25 Every single one of them are going into
1:14:28 that coefficient.
1:14:30 That's why it's positive.
1:14:32 That's why it ends up being,
1:14:33 that's,
1:14:33 that's like,
1:14:33 that's what's going on,
1:14:35 OK?
1:14:35 And not only that,
1:14:36 look what other thing you're getting,
1:14:37 right?
1:14:38 You're getting these really long
1:14:40 24
1:14:42 lacks.
1:14:42 You're getting 24 leads.
1:14:44 Why are you getting 24 leads?
1:14:46 Because
1:14:47 I'm using all the data.
1:14:49 I'm including the 04 to 09 group.
1:14:52 And in 004,
1:14:54 there are 24
1:14:56 leads,
1:14:57 right?
1:14:57 So again,
1:14:58 you're seeing two-way fixed specs kind of doing this like extrapolation thing,
1:15:01 right,
1:15:02 beyond the support of the data
1:15:03 because it's making those comparisons even for its leads and lacks.
1:15:06 But when I use Sun and Abraham,
1:15:08 it's interaction weighted
1:15:09 three-step estimator,
1:15:11 does not get all of them,
1:15:13 does not get all of them.
1:15:14 And the reason is because
1:15:16 Sun and Abraham is basically a version of Calloway and Santana.
1:15:20 It uses this group
1:15:23 right here.
1:15:24 It's got to use an untreated group.
1:15:26 So what it does is it uses
1:15:28 the last
1:15:29 cohort
1:15:30 as the control,
1:15:31 but it only uses it
1:15:33 for the 1980 to 2003.
1:15:35 So look what else,
1:15:36 so that's one thing going on,
1:15:38 right?
1:15:38 So it's only going to get at most
1:15:40 17
1:15:41 lags.
1:15:43 Which is right here.
1:15:44 That's 17 lags.
1:15:45 Notice that
1:15:46 two-way fixed effects is going all the way to 009,
1:15:49 sorry,
1:15:49 it's going to 24 lags,
1:15:50 but Sun and Abraham stopped there and that's because they only work with this.
1:15:53 That's not even the most,
1:15:54 that's not even everything,
1:15:55 right?
1:15:56 The other thing that you've got to realize
1:15:58 when you're working with these event studies is exactly
1:16:01 what
1:16:02 units are contributing
1:16:04 to those leads and lags.
1:16:05 Now look,
1:16:07 If I want to look at 5 lags,
1:16:09 5 lags,
1:16:10 right?
1:16:11 Well,
1:16:11 they have 5 lags,
1:16:12 they have 5 lags,
1:16:13 and they have 5 lags.
1:16:14 So I'm getting weighted averages over
1:16:17 those
1:16:18 5
1:16:19 CATTs,
1:16:20 those 3 CATTs,
1:16:21 and it's getting it right.
1:16:23 True effect is a 48,
1:16:24 it's getting 48.
1:16:25 But when I go beyond 5,
1:16:27 look what happens.
1:16:29 I don't have.
1:16:31 The 98 group doesn't have
1:16:33 11 lakhs,
1:16:35 right?
1:16:35 Only these first two groups are.
1:16:37 So this,
1:16:38 this treatment parameter here,
1:16:41 right,
1:16:41 is based on these,
1:16:43 and
1:16:44 it,
1:16:45 and when I calculate it,
1:16:47 all right,
1:16:47 when I calculate,
1:16:48 when I estimate it.
1:16:50 I'm getting sample selection.
1:16:52 Right?
1:16:53 I'm not using everybody
1:16:55 anymore,
1:16:55 and the same thing here,
1:16:57 I'm not using it.
1:16:57 So watch what's happening,
1:16:58 right?
1:16:59 This one is going up by 8,
1:17:01 OK?
1:17:01 It's going up by 8 at a time,
1:17:03 but now look what's happening.
1:17:04 It's going up by 9.
1:17:06 Then it goes up by 10.
1:17:07 So the dynamics,
1:17:08 look what happens.
1:17:09 You can't really quite tell,
1:17:10 but you can sort of see it shift up a little bit.
1:17:12 It's shifting up
1:17:14 because
1:17:15 the slope
1:17:16 of the,
1:17:17 um,
1:17:18 like the time path
1:17:20 is getting a little bit steeper,
1:17:21 it's uh,
1:17:22 steeper,
1:17:22 and the reason it's getting a little bit steeper is because we're losing
1:17:25 the late,
1:17:26 the,
1:17:26 the like,
1:17:27 you know,
1:17:27 later treated groups and just in this data generating process,
1:17:30 they have
1:17:30 smaller dynamics.
1:17:32 So it's just something to keep in mind,
1:17:33 it's not necessarily a problem,
1:17:34 but it's something that again,
1:17:36 uh,
1:17:37 by understanding what exactly these methods are doing,
1:17:40 that's kind of why I do like these implicit,
1:17:42 uh,
1:17:43 methods a lot is because they've really helped me,
1:17:46 uh,
1:17:47 conceptualize what's going on.
1:17:49 But then there are the explicit
1:17:51 amputation methods.
1:17:52 They're all unbiased,
1:17:54 right?
1:17:54 They're all consistent
1:17:56 under
1:17:57 slight.
1:18:00 slightly different data generating processes,
1:18:02 slightly different data generating process,
1:18:04 and the imputation methods,
1:18:06 the imputation methods here.
1:18:08 They use a stronger parallel trends assumption.
1:18:11 Right,
1:18:12 they use a slightly stronger parallel trends assumption that requires
1:18:15 parallel trends to follow
1:18:18 that two-way fixed effects model.
1:18:19 It's going to be a two-way fixed effects model that I put in here.
1:18:24 It's gonna be a two-way fixed effects model that's going to uh
1:18:27 model Y 0,
1:18:29 right?
1:18:29 Which means that parallel trends holds in every single period,
1:18:33 not just
1:18:35 in the period.
1:18:37 Uh,
1:18:38 that Calloway and Santana needed to do.
1:18:40 Calloway-S Santana is the weakest of all the parallel trends assumption,
1:18:43 only needs it
1:18:44 in the post-treatment period.
1:18:46 I don't know how big of a deal that is though in practice,
1:18:50 because if you are relying on event studies to confirm
1:18:53 the parallel trend assumption in your mind,
1:18:55 then you're not gonna really be able to differentiate
1:18:58 between the two.
1:18:59 But I put it out there
1:19:01 for you anyway.
1:19:02 So above of the imputation methods,
1:19:04 there's basically 3.
1:19:05 There's 4 if you count matrix completion with nuclear norm regularization,
1:19:09 which is the Susan Aheal paper in JASA,
1:19:12 but um
1:19:13 it's really a synthetic control model because it
1:19:15 does not rely on a parallel trends assumption,
1:19:18 and you can kind of say like,
1:19:19 well,
1:19:19 Cunning,
1:19:20 why do you,
1:19:20 why are we even doing it that way?
1:19:21 It's like you can do it however you want to do it.
1:19:23 That's how I,
1:19:24 that's how I group all these things.
1:19:25 And so I'll just kind of say like
1:19:27 the,
1:19:27 the imputation methods,
1:19:29 they all are kind of similar.
1:19:30 Well,
1:19:31 in fact like
1:19:32 Moonblock
1:19:33 by Wooldridge actually nests all of them,
1:19:36 right?
1:19:36 But what they're basically doing,
1:19:38 there's a history
1:19:39 in causal inference of doing what they're doing,
1:19:42 right?
1:19:42 And it goes back at least to Heckman,
1:19:44 Ikamura,
1:19:45 and Todd.
1:19:46 What they're gonna do
1:19:47 is they're gonna impute
1:19:49 the missing counterfactuals,
1:19:51 all right,
1:19:52 for the treated units
1:19:54 by estimating
1:19:55 the fixed effects.
1:19:58 Using only the control group units.
1:20:01 So here's like an example here with,
1:20:03 uh,
1:20:03 so there's the John Gardner model,
1:20:05 which uses GMM
1:20:06 and then there's the robust efficient imputation
1:20:08 model by Krill Borsak and co-authors.
1:20:11 Uh,
1:20:11 it's very similar
1:20:12 to two-step DID,
1:20:14 two-stage DID,
1:20:15 but um,
1:20:17 Uh,
1:20:17 a little,
1:20:18 a little bit different.
1:20:18 So here,
1:20:19 here's what they do.
1:20:19 It doesn't use GMM for one.
1:20:21 So they're gonna estimate the expected potential outcomes using OLS
1:20:26 but only
1:20:27 the untreated
1:20:28 observation.
1:20:29 So the way they're gonna do it,
1:20:31 actually,
1:20:31 yeah,
1:20:32 they are estimating the expected,
1:20:33 but,
1:20:33 but the way that they're doing that
1:20:35 is they're estimating.
1:20:38 The,
1:20:38 the panel unit fixed effects and the time fixed effects.
1:20:41 They're getting consistent estimates of those
1:20:44 from running regression on the control group units.
1:20:47 Then taking those,
1:20:50 they impute.
1:20:51 All of the treatment group
1:20:53 counterfactuals because if the data is following
1:20:57 a fixed effects model
1:20:59 and you can get consistent estimates of the fixed effects,
1:21:03 then
1:21:04 it's simple,
1:21:04 which is probably why Gardner and Carrell Borsak and all
1:21:07 kind of both thought of it at the same time.
1:21:09 And Moonlock.
1:21:10 Uh,
1:21:11 it doesn't appear that it would be the same,
1:21:13 but it,
1:21:13 it,
1:21:14 it,
1:21:14 it is able to get to the same answer
1:21:16 as Borsak at all.
1:21:18 It's able to get the same answer as Borsack all with this like massively saturated
1:21:22 two-way fixed effects model,
1:21:24 right?
1:21:26 So,
1:21:27 um,
1:21:29 So here you see it,
1:21:30 Borsak al,
1:21:31 son and Abraham,
1:21:32 both get to the same answer,
1:21:33 68.33.
1:21:34 Now it was easy to do it.
1:21:36 Why was it easy to do it?
1:21:38 Because my data generating process was a fixed effects model.
1:21:41 So it's like it set them all up to be successful.
1:21:44 But see what's interesting,
1:21:45 here's what's kind of interesting.
1:21:47 It was a fixed effects model that made the data,
1:21:49 and it was a fixed effects model that didn't work.
1:21:51 This should actually be
1:21:53 that positive 24.
1:21:54 It was a fixed effects model that didn't work even
1:21:56 though the data was generated by a fixed effects model,
1:21:59 and that's because strict exogeneity was violated
1:22:02 with that heterogeneity.
1:22:04 So there's software to do all this stuff.
1:22:06 There's software to do all these things.
1:22:07 And there's even software for
1:22:09 Wooldridge's moonlock estimator,
1:22:10 but it's like,
1:22:11 uh,
1:22:11 it's like do files.
1:22:13 Uh,
1:22:13 you have to go to his website,
1:22:14 go to his Dropbox,
1:22:16 he'll like distribute the do files.
1:22:18 I mean,
1:22:18 uh,
1:22:19 you know,
1:22:19 so,
1:22:19 so it,
1:22:20 it,
1:22:20 that,
1:22:21 that's,
1:22:21 that,
1:22:22 that's an alternative too.
1:22:23 So with Calloway Santana,
1:22:25 uh,
1:22:25 there's something called CSDID but in R,
1:22:27 it's called DID.
1:22:29 Brandt Calloway is basically like a
1:22:30 Blue.
1:22:31 I'm really more of like a,
1:22:32 you know,
1:22:32 black belt
1:22:33 amateur computer sciences.
1:22:35 So like,
1:22:36 this is the one I prefer to use.
1:22:37 In fact,
1:22:38 like,
1:22:38 uh,
1:22:38 my preference personally
1:22:40 is to only use the models,
1:22:42 only use the packages that the
1:22:44 authors of the econometric estimators created because
1:22:47 the reason is because
1:22:49 they have every incentive
1:22:51 to maintain it because of like the kinship affection
1:22:53 they have for the model and any kind of like
1:22:56 other things
1:22:57 like that somebody else makes,
1:22:58 those guys are the last to learn the bugs.
1:23:01 But anyway,
1:23:01 it's available.
1:23:02 So Sun and Abraham,
1:23:04 Sophie Sun made event Study Interact,
1:23:06 but there's an R package FS that does it too.
1:23:09 It uses the Sun ab
1:23:11 function,
1:23:12 Borseal.
1:23:13 They made it in theta DD imputation,
1:23:15 but there's also a DID imputation,
1:23:17 and then Gardner,
1:23:18 uh,
1:23:19 uh,
1:23:20 Gardner made it in the with Kyle Butts,
1:23:22 and then Kyle made it in R.
1:23:26 So I only have a couple of minutes.
1:23:27 Here's my
1:23:27 concluding remarks.
1:23:29 The diff and diff econometrics has been big money for these guys,
1:23:32 so it's gonna continue.
1:23:33 If you think about there being no barriers to entry in a competitive market
1:23:37 and that you're gonna continue to see entry as long as there's profits,
1:23:40 you're gonna continue to see diff and
1:23:41 diff papers because they're publishing really well.
1:23:44 Reud,
1:23:45 AER,
1:23:46 Journal of Econometrics,
1:23:47 right?
1:23:48 And so,
1:23:49 uh,
1:23:49 so here's some other kinds of things,
1:23:51 fuzzy diff and diff.
1:23:53 When the treatment and control group both get treated a little bit,
1:23:55 that's based on
1:23:57 uh stuff that Esther Deflo had done,
1:23:59 uh,
1:23:59 reversal,
1:23:59 reversible treatments,
1:24:00 treatments turned on and off,
1:24:02 geographic
1:24:03 dip and diffs where the dip,
1:24:05 where the,
1:24:05 the,
1:24:05 the control is in the same area with a ring,
1:24:08 uh,
1:24:09 continuous treatments,
1:24:11 uh,
1:24:11 time-varying controls.
1:24:12 It just seems like it's not ending anytime soon.
1:24:15 So,
1:24:15 you know,
1:24:16 this,
1:24:17 this,
1:24:17 this is,
1:24:18 you know,
1:24:18 a big,
1:24:19 big deal.
1:24:19 These papers all seem to be,
1:24:20 you know,
1:24:21 this has been career-making.
1:24:22 For a lot of these people,
1:24:24 OK?
1:24:26 Uh,
1:24:26 but the encouraging thing,
1:24:27 here's the encouragement,
1:24:29 these models are giving identical results for the strong parallel trends,
1:24:33 and in fact,
1:24:34 probably a lot of us may have been thinking of strong parallel trends.
1:24:37 All right.
1:24:38 The confidence intervals will differ,
1:24:40 um,
1:24:42 and you probably want to dig a little bit into that.
1:24:43 So you want to choose the,
1:24:44 here's my suggestion.
1:24:46 Choose the estimate.
1:24:47 So here,
1:24:47 here's my suggestion.
1:24:48 Choose the estimators,
1:24:49 you understand.
1:24:51 Choose the estimators where that parallel trend assumption you feel holds.
1:24:56 I encourage you to use the author's packages,
1:24:59 because they have the incentives to fix the bugs.
1:25:02 But is two-way fixed effects done?
1:25:04 No.
1:25:05 The problem was always model misspecification.
1:25:09 That's what Wooldridge has shown.
1:25:10 The moonlock estimator estimates a properly specified model
1:25:15 with two-way fixed effects,
1:25:16 and it is unbiased.
1:25:17 It's just
1:25:18 not the one anyone had ever done.
1:25:20 It's this completely saturated model
1:25:23 and uh so we now realize the problem was not regression.
1:25:27 The problem was
1:25:28 the canonical specification.
1:25:30 It's just that,
1:25:31 you know,
1:25:31 these other things have been useful because
1:25:34 That's what everybody was using,
1:25:36 but what I'm noticing is that the issues do seem to shrink
1:25:41 when you do have more never treated units,
1:25:43 and that's partly because the weights change
1:25:46 by introducing more 2x2s
1:25:49 that are not problematic,
1:25:51 or if you have less heterogeneity,
1:25:52 or if the treatment timing is closer in time.
1:25:56 But lately,
1:25:57 here's what I've been thinking lately.
1:26:00 I have been wondering.
1:26:02 I wonder if I go back to that,
1:26:05 that Twitter poll I did
1:26:07 between the,
1:26:07 the epidemiologists
1:26:09 and the economists answers to whether or not
1:26:12 you
1:26:12 are committed to heterogeneity
1:26:14 and homogeneity
1:26:15 as a baseline assumption.
1:26:17 I wonder
1:26:18 if,
1:26:19 uh,
1:26:20 if,
1:26:21 if treatment effects might not be nearly as heterogeneous
1:26:26 as you can rig it
1:26:27 in a simulation.
1:26:29 I wonder if,
1:26:30 um,
1:26:32 If,
1:26:33 uh,
1:26:34 some of this might not be as
1:26:37 scary
1:26:38 as
1:26:38 we've kind of been thinking it might be.
1:26:41 Um,
1:26:42 not to say that you continue to use an,
1:26:44 you know,
1:26:45 an inconsistent
1:26:46 estimator,
1:26:47 but as people go back and relook at old papers,
1:26:50 they might be perfectly fine.
1:26:52 They might be perfectly fine.
1:26:54 OK.
1:26:55 I hope that's the kind of thing that the,
1:26:57 the McNamara,
1:26:58 um,
1:27:00 Uh,
1:27:01 uh.
1:27:03 Uh,
1:27:03 the,
1:27:03 uh,
1:27:04 the,
1:27:05 the,
1:27:05 the fellowship lecture,
1:27:06 the McNamara Fellowship lecture doesn't mind me,
1:27:08 me doing,
1:27:09 but I,
1:27:10 I appreciate,
1:27:11 uh,
1:27:11 you've given me
1:27:12 90 minutes to talk about this.
1:27:13 I,
1:27:14 um,
1:27:15 hope it was helpful.
1:27:18 Yeah,
1:27:18 thank you so much for joining us,
1:27:19 professor.
1:27:20 Um,
1:27:21 we are just about at time.
1:27:22 So let me just ask really quickly if anybody has anything in the Q&A.
1:27:25 I think I,
1:27:26 I had a couple of questions that were successfully answered at the end,
1:27:29 namely,
1:27:30 where's the code?
1:27:31 And
1:27:32 can we just solve this by using a bunch of interactions in OLS and those are very nice,
1:27:37 um,
1:27:38 answers,
1:27:39 especially I should just go and check out Professor Wooldridge's Dropbox.
1:27:43 So I will,
1:27:43 I'll text him for his passwords and stuff and then see what's up.
1:27:46 Um thank you very much for joining us.
1:27:49 Uh,
1:27:49 for everyone who will be returning,
1:27:51 uh,
1:27:51 we will have details for the next
1:27:54 lecture available soon.
1:27:56 Um,
1:27:56 as soon as we resolve a quick scheduling conflict,
1:27:59 those will come out to you from the normal,
1:28:01 normal channels.
1:28:01 So keep an eye out for those.
1:28:03 And we look forward to seeing you again.
1:28:05 Um.
1:28:06 Uh,
1:28:06 as always,
1:28:07 you can
1:28:08 follow Professor Cunningham on Twitter.
1:28:09 You can see his new book that covers a lot of these topics in detail,
1:28:13 and I want to thank him for covering
1:28:16 this particular
1:28:18 topic because it has been such
1:28:20 a busy conversation,
1:28:21 one of a lot of recent developments.
1:28:24 And
1:28:25 uh as you said,
1:28:26 one of the most popular sources of identification and papers,
1:28:29 especially in
1:28:30 the kinds of um impactful
1:28:33 journals,
1:28:33 both for policy and for academic careers for a lot of the
1:28:37 people who are who are here and who are interested today.
1:28:40 So thank you again very much.
1:28:42 Um,
1:28:42 we hope to see you all at the next lecture
1:28:44 and uh have a great day and the rest of the week to everyone here.
1:28:50 Thank you.
1:28:50 Bye,
1:28:51 bye,
1:28:51 Ben.
1:28:51 Thanks for having me.
1:28:52 Thank you.
- add-style
- lp-body-content