00:00 All right,
00:00 I think we're going to get started
00:02 remarkably a minute early.
00:04 So if you're Dave Evans,
00:05 please have a seat.
00:10 All right,
00:10 this session is titled Education Policy,
00:13 and,
00:13 uh,
00:14 it is moderated by Dion Filmer,
00:15 who is the director of the research department at the World Bank.
00:18 Uh,
00:18 this will go for an hour and a half,
00:20 and then we're gonna,
00:21 uh,
00:21 go straight to a cocktail reception
00:23 after that.
00:24 So I will not be back up,
00:25 um,
00:25 after this session,
00:27 um,
00:28 but see you all tomorrow.
00:41 Thanks,
00:41 great,
00:42 um.
00:44 It's actually a better turnout than I feared for the,
00:47 you know,
00:48 4 o'clock session.
00:50 Um,
00:51 So I don't know that a
00:52 session on education really needs a lot of motivation,
00:55 but I do want to tie a little bit to this,
00:57 the conference.
00:58 So this morning,
00:59 Rachel kicked us off by saying there's no sustainable,
01:01 I'm paraphrasing,
01:02 no sustainable growth without human capital,
01:05 um,
01:05 so obviously human capital is central to the development part of this,
01:10 but then
01:10 on the first day Masood,
01:12 um,
01:14 In response to,
01:15 I don't know if it was a question or in his talk where he said,
01:18 what,
01:19 what should we do?
01:20 What,
01:20 what should countries do,
01:20 what should
01:22 agencies,
01:22 multilateral agencies like the World Bank do
01:25 in the context of declining multilateralism?
01:28 And his answer was invest in the basics of growth.
01:33 including human capital and in particular education.
01:36 So there we have some motivation coming from within our conference
01:40 for this session.
01:41 So as I said,
01:42 the session is gonna be focused on education policy.
01:45 Uh,
01:45 this morning we learned we heard a lot about service delivery.
01:49 This session takes a slightly different angle.
01:52 We're going to start with a kind of a big picture
01:54 story about the global distribution of skills,
01:57 and then we're going to go into three specific constraints to learning,
02:01 and they're sort of not the typical ones we think about.
02:04 So the idea is to simulate a little bit of new thinking in this area.
02:08 So the speakers are gonna be Dev Patel.
02:11 He's gonna talk to us about the global distribution of skills.
02:14 He's the prize fellow in economics,
02:16 history and politics
02:17 at the Center for History and Economics at Harvard University.
02:21 He's gonna be followed by Christina Brown.
02:23 Who is gonna talk to us about cognitive endurance and if you don't know what that is,
02:28 she's gonna explain it.
02:29 Uh,
02:30 she is an assistant professor at the Department
02:31 of Economics at the University of Chicago.
02:34 Then we're gonna hear from uh Lee Crawford.
02:36 He's gonna talk to us about lead exposure and how that impacts learning.
02:40 He's a senior research fellow at the Center for Global Development.
02:44 And then we're gonna hear from Gabriella
02:45 Smarelli,
02:46 who's gonna talk to us about violence in schools.
02:49 She's a senior research associate
02:51 in the global education program at the Center for Global Development.
02:55 So let me invite without further ado,
02:57 uh,
02:57 Dev.
03:06 Thank you all so much for
03:07 for being here and I'm delighted to present this joint
03:10 work with the brilliant Justin Sandifer who's right here.
03:14 Um,
03:14 so this part is entitled The Rosetta Stone for Human Capital,
03:18 and it's about trying to answer very basic questions
03:21 like this one.
03:23 Who can read better,
03:24 a 4th grader in *** or Peru?
03:27 So while questions like this may seem very simple,
03:30 they're actually deceptively difficult to answer.
03:32 And the reason why is that we lack a global standardized test.
03:36 So one way of visualizing that is with this map.
03:39 This is coverage of the
03:41 primary school standardized test exams with the largest coverage,
03:45 the TIMS and Perls exams.
03:47 As you can see from this map,
03:48 most of the world is actually excluded from the sampling frame,
03:52 including all low-income countries.
03:55 Parts of the developing world that are included in standardized tests tend
03:59 to participate in these regional exams like the ones shown here,
04:03 and they're
04:05 strikingly non-overlapping and also themselves quite sparse.
04:10 So what this project is trying to do is
04:12 try to answer questions like if we took a student
04:15 from Brazil who took essay.
04:17 And then we took a student from the United States who took TIMS.
04:22 How could we actually compare those test scores?
04:24 So for example,
04:25 students in both of these countries
04:27 have participated in these standardized tests,
04:30 but the test scales on which these scores are measured
04:33 are completely different.
04:35 So how can we compare,
04:36 for example,
04:37 a 600 for a student in Brazil
04:39 to a 500 for a student in DC?
04:43 So that's really the question we're gonna tackle today.
04:46 This paper has two parts,
04:48 and I'll just give you a high level overview of where we're going.
04:52 The first part is trying to understand
04:54 how we can actually translate test scores from
04:57 one test to another when they're completely,
04:59 uh,
05:00 separate on these different scales.
05:02 So from a high level what we do is we take different tests.
05:05 So for example,
05:06 you can imagine we take a yellow test,
05:08 a green test,
05:09 a red test,
05:09 and a blue test.
05:11 We build a hybrid exam that takes questions from each
05:14 of these tests and put them together in one booklet.
05:18 Then we give that booklet to some students,
05:21 and when we grade that test score for that student,
05:24 we do it on each original test score scale.
05:27 So we grade that student's performance on the yellow scale,
05:30 the green scale,
05:31 the red scale,
05:32 and the blue scale.
05:34 We do this a bunch of times for many,
05:35 many students,
05:37 we can then actually compare
05:39 for any other
05:40 test score from a student who only took one of the original exams,
05:44 or what they would be like on another exam,
05:45 so we can kind of create this
05:47 Rosetta Stone translation from one exam to another.
05:51 And then once we have that,
05:52 we can go and find a bunch of other students who may have only taken one of these tests,
05:57 pretend like that green student actually took the yellow exam,
06:00 and put them all on the same scale.
06:03 once we've done this methodology,
06:05 we actually implement this to try and link
06:08 4 of the world's largest standardized tests for primary school students.
06:12 So this is an important component of,
06:14 for example,
06:14 the SDGs.
06:16 We'll do this to actually translate test
06:18 scores for about 600,000 kids from 80 countries
06:22 for both reading and math.
06:25 And what I want to kind of bring to this panel is
06:27 some basic statistics about the distribution of learning around the world.
06:30 So we're going to start with 4 main facts.
06:33 The first is that
06:34 where a child lives turns out to be much
06:37 more important than how much money their household earns
06:40 in terms of predicting their test score.
06:42 So in other words,
06:43 if I took two students with the same household income
06:46 but who happen to live in very different countries,
06:49 it turns out that that is going to matter a lot more which country you live in
06:52 as opposed to how rich your household is.
06:56 The second factor we're gonna find is that the
06:58 relationship between your household's income and your test score
07:02 is gonna be steeper
07:04 in countries that are more unequal.
07:07 This is going to be very reminiscent of the types of
07:10 other
07:10 findings on inequality
07:12 and the so-called Great Gatsby curve that you might be familiar with.
07:16 The third fact is that we can look across gender,
07:20 and at a household level,
07:21 if we look at your household income,
07:23 we see that no matter what
07:24 household income you're,
07:25 you're in,
07:26 whatever bin you're in,
07:27 it turns out that girls tend to do better than boys in reading.
07:31 However,
07:32 that's not true for math,
07:33 and so for math,
07:34 only the kind of poorest households are we going
07:36 to see that the girls do better than boys,
07:38 and in fact,
07:39 we're going to see kind of a reversal as households get richer.
07:43 And then the 4th fact we're gonna find is that
07:45 if you look at the test score gap between public schools
07:49 and private schools,
07:50 we're gonna see that the difference in scores is gonna be bigger
07:55 in countries that have more income inequality.
07:58 So I'm gonna dive into a little bit more detail now
08:01 on all of these facts and our methodology.
08:04 First,
08:04 I'm going to start with just providing a very high level overview
08:07 of the method that we developed
08:08 and how we implemented that using fieldwork we conducted in India
08:12 and in Florida.
08:14 So
08:16 imagine we took with our our 4 different color tests,
08:18 as you can tell I'm a big fan of clip art in this talk.
08:20 So we start with our 4 color tests.
08:23 One thing you might do is just give each of those tests to students.
08:27 And then you would grade those tests.
08:29 Now the problem with that is that tests are really long
08:31 and no student really wants to sit through that many tests,
08:33 let alone one.
08:35 So that's the first obstacle.
08:37 And the second obstacle is that
08:39 a lot of the tests that we use as researchers are not actually publicly available.
08:43 So even if we wanted to do that,
08:45 those testing companies are,
08:46 are quite protective of their,
08:48 uh,
08:48 standardized tests,
08:49 and so they only release a small number of questions.
08:52 So the way we try to overcome this is we actually build an exam
08:56 that takes
08:57 test questions from each of the original source tests and puts them all together
09:01 in a new test booklet.
09:03 This is what I'm gonna call a hybrid exam.
09:07 So then from a high level what we can do is I take this hybrid exam.
09:11 And give it to students,
09:13 and so we did this both in Bihar,
09:15 India
09:16 and in Florida in the United States.
09:18 And so for each student,
09:21 kind of the core methodological part of our
09:23 paper is to use statistical methods from psychometrics
09:26 from this item response theory that's used to grade the original tests.
09:31 And
09:31 for each student who takes our hybrid exam,
09:33 we're gonna grade their performance as if it were the original full test
09:37 from the source exams.
09:39 This is the exact same type of method that's used
09:41 by the SATs to compare test scores over time or across students,
09:46 uh,
09:46 and we're just going to apply that to the study.
09:50 We're gonna do that not just for one student,
09:51 but for many,
09:52 many students,
09:54 and what we're gonna find is that we,
09:55 for,
09:56 you know,
09:56 any given person on that same day that they took this exam,
10:00 we can kind of assume they had the same ability.
10:03 And so we can say,
10:04 OK,
10:05 they scored a 486 on the red test,
10:08 that must be equivalent to a 312 on the blue test.
10:12 We're gonna do this for a bunch of bunch of students,
10:15 and that means that across the income sorry ability distribution,
10:18 we should be able to compare students and kind
10:20 of do these mappings from one test to another.
10:24 So this is the kind of core method we've developed.
10:27 We're gonna do that and apply that to 4 big standardized tests,
10:31 the TIMS and Perls exams,
10:32 which cover kind of the OECD,
10:35 ESA,
10:35 which covers many Latin American countries,
10:37 and then PASEC from West Africa.
10:40 We're gonna develop these hybrid exams,
10:42 pulling questions from these different source tests
10:45 and administer them in both Florida and Bihar.
10:48 And this is important because this actually gives us
10:51 a pretty big coverage across the skill distribution.
10:55 And
10:55 for the rest of today's talk really focus on some of the math results that we find.
11:00 So once we do this,
11:01 what we can come up with at the end of the day
11:04 are test conversion functions that look like this.
11:06 So just walk you through this,
11:08 on the x-axis of these graphs are the scores on the TIMS exam.
11:13 This is the,
11:13 you know,
11:13 that OECD exam.
11:15 And because we graded for each student their performance on another exam,
11:19 so on the Y axis in the graph on the left is the West African exam,
11:23 and the Y axis on the graph on the right is Latin American exam,
11:26 we can map from one test to another.
11:29 And what you can see is that we can just convert,
11:31 you know,
11:31 put in a test score on one axis,
11:34 and you can draw a line and figure out where it would be on the other one.
11:37 And so these are the types of test score conversion functions we can estimate
11:40 and then apply around the world.
11:42 So what that allows us to do is,
11:44 for example,
11:45 take those original test score distributions from Brazil and the United States
11:49 and now put them on the same x-axis,
11:52 so we can see exactly how those students now line up.
11:56 So once we've done that,
11:58 we're gonna now talk through a little bit more of the different types of facts we can
12:02 learn from this type of big learning data
12:04 once we've harmonized test scores around the world.
12:07 I'm gonna,
12:08 the rest of today's talk focus on test scores that are
12:10 in the units of that TIMS and Perls OECD exam.
12:14 So you should think of kind of 100 points there as
12:17 one standard deviation
12:18 in that kind of OECD units.
12:21 OK,
12:21 so fact one that we're gonna find
12:24 is that where students live turns out to
12:26 be much more important than their household income.
12:29 So I want to visualize that in a simple graph.
12:31 So here on the x axis
12:33 is the household income of the student who took the exam.
12:38 And I've drawn in dotted vertical lines here,
12:41 the World Bank income classifications.
12:44 The Y axis of this graph
12:46 is the student's test score on that kind of TIMS and Perls OECD exam.
12:51 And so what you can see is that first there's a really big difference in
12:56 math and reading ability as you move up the income distribution.
13:00 So richer students are performing significantly better
13:04 than poorer students,
13:05 and so when we talk about the learning crisis around the world,
13:08 this is kind of showing a lot of data towards that fact.
13:12 But one thing that this,
13:14 uh,
13:14 bin the scatter plot masks is that if you then try to write,
13:18 do the same thing
13:19 separately by country,
13:21 you see there's a huge amount of variation depending on where students live.
13:25 So now what we've done is,
13:27 um,
13:27 in each shade of blue here,
13:29 the kind of lighter shades are high income countries
13:32 and the darker shades are lower income countries.
13:35 And what you can see is if you just take
13:36 a vertical line and draw it somewhere on this graph,
13:40 so you're holding fixed the household income of a student,
13:43 it turns out that what country you live in is going
13:46 to be a huge determinant of what your test score is.
13:49 And it turns out that that kind of variation across
13:52 countries is going to be much more important in our data
13:55 than the variation within country across household incomes.
14:00 So that's fact one.
14:02 The fact 2 is about how incomes and test scores relate to one another.
14:07 So
14:08 I'm gonna plot in this graph two very simple statistics.
14:12 On the x axis is the Gini coefficient that we're all familiar with,
14:16 about how unequal the income distribution is within a country.
14:20 And the y axis is the relationship between a country's student's math score
14:26 and their household income within that country,
14:28 OK,
14:29 so you should think of a separate
14:30 parameter for every country.
14:33 So for a country like Finland,
14:35 which is a very unequal country in terms of income distribution,
14:39 you see that also the relationship between math score and income is also very low.
14:43 So
14:44 in other words,
14:44 if I increase my income quite a bit in income in Finland,
14:48 my test score does not go up very much.
14:51 By contrast,
14:52 a country like Brazil,
14:54 that's a lot more unequal on the income side
14:56 is also gonna have a much steeper relationship,
14:58 so that the richer the students are going to do significantly better
15:01 in terms of their math score,
15:03 and this tends to hold across countries too,
15:06 so we can now plot all the countries in our data set,
15:08 and we see the same kind of relationship.
15:12 OK,
15:12 the third fact I want to show you today is looking at the gender gap in test scores.
15:17 So here,
15:17 what we're gonna do is that the same graph I showed you before,
15:20 where the x axis is the household income,
15:22 and the Y axis now
15:24 is going to be the ratio
15:26 of test scores for girls to boys.
15:29 So if it's above that kind of horizontal dotted line,
15:31 that means girls are going to do better.
15:34 So when we look at reading,
15:35 we see that across the whole household income distribution,
15:38 girls are actually outperforming boys in terms of reading.
15:43 When we look at math,
15:43 we see a different story,
15:44 however,
15:46 only in the poorest households are,
15:48 are girls outperforming boys in math,
15:50 and actually you see that relationship reverse as you households get richer.
15:56 And then the last fact I want to talk to you about today is about private schooling.
16:00 This is obviously a huge phenomenon around the world.
16:03 One thing we can do with our test score data is just compare
16:06 the test scores of students in private schools to those in public schools.
16:10 So here in Colombia,
16:11 for example,
16:13 private school students tend to score about 50 points,
16:15 so it's like 0.5 standard deviations higher
16:19 than students in public schools.
16:21 We can do this across all of the countries
16:23 in our data,
16:25 and you can see that there's a lot of variation across countries in
16:27 the differences between private and public schools in the raw test scores.
16:32 Of course,
16:32 there's,
16:33 you know,
16:33 differences in the types of students that go to private and public schools here,
16:37 this is not a causal impact,
16:39 but if we put in a bunch of controls for things like gender and wealth,
16:43 things change,
16:44 but they don't change a huge amount.
16:45 There's still kind of quite a bit of variation.
16:49 One thing we can do then
16:50 is see how this varies with inequality in a country.
16:54 So now on the x axis is how unequal country in terms of its income,
16:58 and the y axis is that kind of private school premium.
17:02 What you see is a positive relationship here,
17:05 OK,
17:05 so the,
17:06 the more unequal a country is in terms of income,
17:09 you also see that there's kind of more of a
17:11 gap between the private and the public school students.
17:15 Now I'll be very consistent with,
17:16 for example,
17:16 in unequal countries,
17:17 if there's more
17:18 socioeconomic segregation across schools,
17:21 that's gonna manifest itself in the same way.
17:24 So,
17:24 um,
17:25 I'm gonna just conclude now by saying this paper
17:28 was about trying to compare test scores across countries,
17:31 even when those countries participate in different standardized tests.
17:34 What we do is develop a method to link those exams together.
17:38 There's a lot of learning inequality and that is certainly popping out in our data.
17:43 Very excited for the rest of the panel.
17:44 Thank you so much for your time.
17:49 Thanks,
17:50 Dave.
17:50 Uh,
17:50 next up we have Christina.
18:01 Great.
18:01 Well,
18:02 thank you so much for having me.
18:03 Uh,
18:03 delighted to present this research,
18:05 which is joined with Supreme Court,
18:06 Geeta Kingdon,
18:07 and Heather Schofield.
18:10 So,
18:10 you know,
18:11 I think I don't need to convince this room that
18:13 we have lots and lots of evidence on the
18:15 effect of schooling on many long-run outcomes for individuals,
18:19 everything from income,
18:20 health,
18:20 criminal behavior,
18:21 social outcomes.
18:22 We think,
18:23 you know,
18:23 schooling is,
18:24 is really,
18:24 really important for lots of long-run outcomes.
18:27 And
18:28 we have some evidence on what are the specific
18:31 channels that might contribute to those long-run outcomes,
18:34 but I would say they kind of fall into these two broad categories,
18:37 right?
18:37 There's some research that focuses on this idea that it's the specific
18:41 curriculum,
18:42 the academic content,
18:43 the literacy and numeracy skills,
18:45 which are contributing to these long-run outcomes.
18:48 And then there's this other idea that schooling probably does lots of
18:51 other things too in terms of building people's kind of cognitive capacity.
18:54 It might influence their ability to,
18:57 um,
18:57 kind of improve their working memory or their attention skills.
19:01 And on that second piece,
19:02 we have much less evidence.
19:04 We really don't have a lot of great measurement about those
19:06 kind of fuzzier cognitive skills
19:09 as compared to all of the really rich evidence that we just saw
19:12 on measuring
19:14 concrete literacy and numeracy skills.
19:16 And so what we're going to try to do in this paper
19:19 is really focus on one specific aspect of schooling,
19:22 in particular,
19:23 when we're going to think about the fact that
19:25 in most schools or most good schools,
19:28 you're spending a large part of your day
19:31 on effortful thinking for an extended period of time,
19:33 right?
19:33 That's kind of the experience of,
19:35 of what it means to go to school.
19:37 And then we're going to look at the effect of that type of schooling
19:40 on
19:41 a particular
19:43 cognitive capacity,
19:44 which we're going to call cognitive endurance.
19:47 So what do I mean by cognitive endurance?
19:50 We're going to define this as the ability to sustain,
19:53 uh,
19:53 performance on a cognitively challenging task over time,
19:57 right?
19:57 So this is,
19:58 can you,
19:59 you know,
19:59 how focused,
20:00 how much are you learning from this talk in the 1st minute
20:02 versus the 15th minute,
20:04 right?
20:04 Or at the end of this panel versus the beginning of the panel,
20:07 how much are you taking in?
20:08 That's really going to be our measure here.
20:11 And there's been lots of evidence showing that this has
20:13 kind of broad relevance for a lot of different settings.
20:16 We've seen that
20:17 in the workplace,
20:19 people tend to make
20:20 more mistakes later on in the day,
20:23 even in the situation of voting.
20:25 Further down the ballot,
20:27 people actually make less active,
20:29 kind of effortful decisions,
20:31 and they tend to result,
20:33 rely more on defaults if a ballot item appears later down.
20:38 But interestingly,
20:39 this really varies by overall income of the population.
20:42 So it's not just that kind of everyone gets cognitively fatigued,
20:46 um,
20:46 but that this might be particularly worse for certain populations.
20:52 OK,
20:53 um,
20:53 oh,
20:53 I realized that
20:54 the slides are slightly old,
20:55 so I'll have to,
20:56 ah,
20:56 improvise a little bit on some of them.
20:58 Um,
20:59 but what I'm showing you here is that,
21:01 um,
21:01 I,
21:02 not only do we see this kind of cognitive fatigue
21:05 happening in,
21:06 in the workplace,
21:07 it's also a really common feature of standardized tests.
21:11 Um,
21:11 so in the TIS tests,
21:12 uh,
21:12 again,
21:13 more TIS data building off of Dev's work,
21:15 um,
21:16 we tend to see that over the length of that exam,
21:18 so this is over within a single subject,
21:21 um,
21:21 which is about a 30 to 40 minute exam.
21:23 This is for 4th graders.
21:25 That students tend to perform worse over the length of that task.
21:30 So relative to an identical question item,
21:33 because we can actually control for question items,
21:35 there's variation in the ordering of questions that kids get in at TIMS tests.
21:39 For an identical question item,
21:41 kids are performing worse
21:42 when they happen to have gotten it as the last item versus the first item.
21:47 But this really varies by income level.
21:49 So this,
21:49 this fatigue that students are facing
21:52 is substantially worse if they're coming from a low-income country.
21:56 And there could be lots of explanations for that.
21:58 Um,
21:59 it could be that,
21:59 for example,
22:00 you know,
22:00 there's differences in nutrition,
22:03 maybe students didn't have time to finish the test.
22:05 There could be,
22:05 you know,
22:05 lots of drivers of that fact.
22:08 Um,
22:08 but we're going to focus on one in particular,
22:10 which is that maybe the type of
22:11 schooling that kids experience in low-income settings
22:14 is such that,
22:16 you know,
22:16 in,
22:16 in high-income classroom,
22:18 they're getting opportunities to kind of practice training this,
22:21 this
22:21 cognitive endurance muscle,
22:23 right?
22:23 They have the types of instruction where they're getting to practice
22:26 attending to a task over an extended period of time.
22:29 And so that way they end up being able to deploy that skill in lots of settings.
22:36 And in particular,
22:37 you can see this when you look at um
22:39 classroom observation data as well as teacher self-report.
22:42 Um,
22:42 this graph is showing you,
22:44 um,
22:44 on the
22:45 x axis is income per capita,
22:48 um,
22:48 or the percent of students coming from,
22:50 uh,
22:51 disadvantaged groups
22:53 and the fraction of time that students spend as part of the school day on.
22:57 Individual focused practice,
22:58 right?
22:59 So in low-income settings,
23:00 we typically think of classrooms as being much more teacher led,
23:04 um,
23:04 that students are kind of listening,
23:06 um,
23:07 copying down what they're hearing from their teacher,
23:08 but not spending a lot of time on individual focused practice.
23:12 Whereas in high-income countries,
23:14 they're spending a much larger fraction of their day on that type of activity.
23:21 So what are we going to do?
23:22 Um,
23:22 we're going to conduct an experiment,
23:23 um,
23:24 with a series of private schools in Lucknow,
23:26 India.
23:27 These are going to be low-cost private schools and they're going to
23:29 have all of those same features that I just talked about where,
23:32 um,
23:33 most of the instruction that students are going to
23:34 be experiencing is kind of teacher led lecture.
23:37 Where students are kind of passively listening,
23:39 um,
23:40 and,
23:40 uh,
23:41 maybe copying down notes,
23:42 um,
23:43 but they're only spending,
23:44 you know,
23:45 maybe a few minutes during each,
23:47 um,
23:47 uh,
23:48 period of the day on
23:50 kind of concentrated,
23:51 um,
23:51 uh,
23:52 cognitively challenging work.
23:55 And so what we're going to do is try to come in and say,
23:57 OK,
23:57 let's try to get
23:58 it so that,
23:59 uh,
23:59 a larger fraction of their school day is spent on
24:04 kind of extended practice on some cognitively challenging activity.
24:06 We don't really care what the activity is,
24:08 we just want them to spend more
24:10 extended time on some sort of thinking activity.
24:14 And in particular we need this to kind of hold their attention for 20 minutes or so.
24:18 This is kind of,
24:19 uh,
24:19 uh,
24:20 from the
24:21 length of their
24:22 kind of periods,
24:24 um,
24:24 in these classes.
24:25 And so our idea was to basically introduce um
24:29 a series of different kind of very basic apps,
24:31 um,
24:32 like not gamified,
24:33 very stripped down,
24:34 um,
24:34 somewhat boring,
24:35 um,
24:36 but that were dynamically adaptive.
24:38 The idea being that if we wanted something
24:40 that just barely is holding these kids' attention,
24:42 so it's stretching that attention muscle,
24:45 the content can't be too hard that students are tuning out and it can't be so easy
24:49 that they're,
24:50 um,
24:50 you know,
24:51 ah,
24:51 tuning out for,
24:52 for that reason.
24:54 And so what we did exactly was that kind of uh coming into their existing,
24:58 um,
24:59 elective period.
25:00 So these are thirty-minute periods that they have a couple of times a week.
25:04 We randomize students to either,
25:06 um,
25:06 receive the kind of math practice,
25:08 um,
25:08 tablet application
25:10 or the games practice one
25:12 relative to the kind of status quo study hall period.
25:16 So in the math and the games group,
25:17 they're getting a tablet during this 30-minute period.
25:20 They're working on either math problems or in the games group,
25:23 it's like puzzles,
25:24 mazes,
25:25 um,
25:26 apps that we specifically chose to be,
25:28 again,
25:28 kind of very,
25:29 uh,
25:29 kind of boring to a certain extent,
25:31 um,
25:32 but also free of any academic content.
25:34 We wanted them to be doing something cognitively challenging,
25:36 but that didn't have any specific literacy or numeracy numeracy requirements.
25:42 And that's going to be compared to the status quo where students,
25:45 um,
25:46 basically get kind of a study hall period.
25:48 The teacher spends most of that time doing grading or other
25:50 activities and they write a few problems up on the chalkboard.
25:53 Students work on those,
25:55 um,
25:55 and then usually kind of talk to their friends or do some other activity.
25:59 And so in practice what this ends up meaning is
26:01 that the treatment group is going to get an additional
26:04 about 15 hours of focused practice over the 6 month intervention period
26:09 relative to the control group.
26:11 So they're getting,
26:12 you know,
26:13 you know,
26:13 15-ish additional hours spread out over these different periods of time.
26:20 So what do we find?
26:22 Well,
26:22 what we were fundamentally interested in was,
26:25 if you get them to focus at practicing on a,
26:27 on a math,
26:28 on math problems or on puzzles and games,
26:30 is this a general skill that then they can deploy in lots of different settings?
26:35 Or is this a very specific skill,
26:37 right?
26:37 That if I practice attending to math problems,
26:40 I can pay great attention to math problems,
26:42 but that's it,
26:43 right?
26:43 Is this,
26:43 is this an underlying,
26:45 um,
26:46 general skill that can be transferred to different settings?
26:49 So what we did was we tested students in 3 different domains,
26:52 some that were more or less related
26:54 to the activity that they had been doing in their elective classes.
26:59 And we randomize the question order so we can look at
27:02 our,
27:02 our kind of outcome of interest here is not just
27:04 how well you perform on this listening comprehension task,
27:07 which is one of them,
27:09 but
27:09 how much worse do you do at the end of the test relative to the beginning.
27:15 So this first graph is showing you for the control group.
27:17 Sorry,
27:18 it's a little bit small,
27:19 that,
27:19 um,
27:20 labels here.
27:21 What is students performance look like over time.
27:24 And for the dotted line here,
27:27 the control group,
27:27 we see that students are doing about 3% points worse on an identical question item
27:32 when it appears at the end of the test versus the beginning.
27:35 And this was a listening comprehension test that
27:37 kids took basically like listening to headphones,
27:40 they listen to a short story and then have some.
27:42 Kind of understanding check questions that they have to complete,
27:45 um,
27:45 based on,
27:47 uh,
27:47 it was all pictorial,
27:48 so there was no literacy,
27:49 written literacy requirements to complete this,
27:52 but it's kind of a boring story that they're listening to.
27:54 It's kind of boring audio that they have to follow along to.
27:56 And so really we're trying to capture like,
27:58 can you sit and pay attention
28:00 similar to the maybe this setting currently,
28:02 right?
28:02 Can you sit and pay attention to something where,
28:04 you know,
28:05 it might not be doing a great job of necessarily holding your attention.
28:10 And so then,
28:11 uh,
28:11 the,
28:12 the great line was for the control group,
28:14 um,
28:14 and the blue and orange lines here are for the treatment group.
28:17 And what this is showing you is the first kind of
28:19 interesting pattern is that at the beginning of the exam,
28:22 we actually don't find any difference in performance,
28:25 and that's kind of what we would expect,
28:26 right?
28:26 Neither of the treatments were doing anything that
28:29 would improve your underlying listening comprehension skills.
28:32 Skills.
28:32 It wasn't kind of building those particular skills,
28:35 but what it was allowing you to do was
28:37 practice focusing on something for an extended period of time.
28:40 And so we see by the end of the task,
28:42 now both treatment groups are doing significantly better than the control group.
28:46 They still suffer,
28:47 you know,
28:48 a decent amount of,
28:49 of,
28:49 uh,
28:49 cognitive fatigue over the task,
28:51 but it's substantially less than the control group.
28:55 Similarly,
28:55 for the Ravens,
28:56 um,
28:57 test and the math test,
28:58 we see a somewhat similar pattern,
29:00 though a little bit more noisy.
29:02 So for the Ravens test,
29:03 we see a little bit of a difference between the control and the treatment groups,
29:06 though it actually isn't statistically significant,
29:09 um,
29:09 because this one we're zoomed in a little bit.
29:11 Um,
29:12 but we do see by the end of the test,
29:14 um,
29:14 you can kind of see that's like a little bit of
29:15 a flatter slope essentially for the that blue and orange line
29:19 compared to the gray line,
29:20 which has a steeper decline over the length of the task.
29:24 On the math,
29:25 um,
29:25 test,
29:26 you also see this really interesting pattern.
29:27 I don't know if you can see kind of closely enough what's happening here,
29:30 but at the beginning of the test,
29:32 we actually see that the games group,
29:34 which is the orange line,
29:36 is doing slightly worse than the control group.
29:38 And that is also kind of what we would expect because if you remember,
29:41 the control group is doing some math problems,
29:43 they have those initial few problems that they're working
29:45 on at the beginning of the study hall period,
29:48 and then they're,
29:48 you know,
29:49 talking to their friends for the rest of the 30 minutes.
29:52 Whereas in the games group,
29:53 they're not doing any math-related content.
29:55 So it's maybe not too surprising that in terms of their kind of,
29:58 you know,
29:58 initial content knowledge,
30:00 um,
30:00 the control group would,
30:01 would be doing a little bit better.
30:04 Um,
30:04 but over the length of the task,
30:05 and this was our,
30:06 um,
30:07 longest task that we had,
30:08 this was a 30 minute task.
30:09 You see this real stark,
30:11 stark decline,
30:12 um,
30:12 over the length of the task,
30:13 and in particular,
30:14 a slightly flatter slope,
30:16 um,
30:16 for those who were in the treatment groups.
30:19 OK.
30:20 Um,
30:21 and so then what we try to do in the rest of the paper is try to really understand,
30:24 OK,
30:24 was this fundamentally about us training attention skills,
30:28 or,
30:28 you know,
30:28 we were introducing tablets in the schools,
30:30 maybe this was doing something else.
30:32 It was,
30:33 um,
30:33 you know,
30:33 maybe kids were more motivated or they were showing up more for school,
30:37 or they just felt more,
30:38 you know,
30:38 they felt special because they were selected for the treatment.
30:41 So we try to do a few things to figure out,
30:43 you know,
30:43 which of these,
30:44 um,
30:44 explanations,
30:46 uh,
30:46 are driving the results.
30:48 Um,
30:48 since I'm short on time,
30:49 I'll go through this a tad quickly,
30:51 but
30:51 the key thing to take away is that
30:54 for a lot of the other channels that you might have in the
30:56 back of your mind that we could be impacting potentially by this treatment.
30:59 We kind of expect there also to be an effect at the beginning of the test,
31:02 right?
31:03 The fact that we're only really seeing,
31:04 um,
31:05 positive effects later on,
31:06 you know,
31:07 into the,
31:07 the 10 or 20 minute or 30 minute task
31:10 says something about
31:11 kind of attention potentially being the channel.
31:14 We also are able to measure some other channels and
31:16 don't seem to find evidence that that's what's driving results.
31:22 And then the other thing we try to do to kind of narrow down these channels is that
31:25 there are some um different measurement tools from the
31:27 psychology literature of how to measure sustained attention,
31:30 which we think of as really analogous to this idea of cognitive endurance.
31:34 Um,
31:34 and we find,
31:35 um,
31:35 uh,
31:35 positive evidence for,
31:36 for two out of these three measures.
31:40 Um,
31:40 and then the last thing that we do is see,
31:42 OK,
31:43 you know,
31:43 we've gotten kids to,
31:44 to pay better attention on our specific tasks.
31:47 It doesn't have any real effect on other outcomes we might care about?
31:51 And we find that,
31:51 yes,
31:52 actually,
31:52 having kids focus on essentially puzzles and games.
31:56 We actually find improvements in their,
31:58 um,
31:59 English,
31:59 um,
31:59 Hindi,
32:00 and math grades
32:01 at the end of the school year.
32:03 And,
32:04 and kind of quite interestingly,
32:06 you might think,
32:06 well,
32:07 this is just coming from the fact that because they can,
32:09 you know,
32:10 pay better attention on tests,
32:11 they're not fatiguing as much on their end of term exams
32:14 and all of this great improvement is really because they're,
32:16 they're focused better on those tests.
32:19 When you do kind of a back of the envelope calculation,
32:21 that would only explain about a 0.02,
32:24 0.03 standard deviation increase in,
32:27 um,
32:28 in,
32:28 uh,
32:28 their exam scores.
32:30 And so actually,
32:30 we think most of these great effects are coming from the fact that
32:34 if they can pay better attention in class,
32:36 they're actually learning more.
32:37 And so that's why we're seeing these improvements in grades is because this
32:40 ability to attend this is now allowing them to learn more content as well.
32:46 Um,
32:46 and then the last thing we do is try to say,
32:48 OK,
32:49 you know,
32:49 this was maybe a little bit of a niche experiment that we did where we came in and,
32:52 and introduced a particular
32:54 type of elective period.
32:56 If kids are just exposed to,
32:58 you know,
32:58 traditional schooling,
32:59 does this also improve their cognitive endurance?
33:02 And so we use like a fuzzy regression discontinuity design
33:05 comparing,
33:06 um,
33:06 very similar HP.
33:08 who are on either side of a birthday cutoff
33:11 and see that,
33:11 you know,
33:12 for a,
33:12 a 9 year old who was either in 4th grade or 5th grade,
33:16 depending on their birthday,
33:17 um,
33:17 those who have an additional year of
33:19 schooling have significantly more cognitive endurance,
33:22 specifically if they attend schools where
33:25 there's more time spent on this individualized,
33:28 um,
33:28 focused practice.
33:30 Um,
33:31 so I'll just leave you with,
33:32 ah,
33:32 you know,
33:32 kind of a few takeaways.
33:34 I think the kind of goal of this paper was really to think about,
33:37 you know,
33:38 There's lots of things that we think schooling might do in terms of improving,
33:42 uh,
33:42 students' skills.
33:44 We're focusing on one in particular,
33:46 one in particular that we think,
33:48 uh,
33:49 from the data looks like
33:50 schools aren't spending as much time on in low-income settings.
33:54 And so our hope is that not only we can show kind of
33:56 that there's ways to measure these other skills in a relatively feasible way.
34:00 If you have an assessment where there's randomized question order,
34:02 you can look at,
34:04 um,
34:04 uh,
34:04 declines in,
34:05 in performance over time.
34:08 And so,
34:08 you know,
34:08 we really encourage people to start thinking about what are
34:11 these other cognitive capacities in terms of measuring working memory,
34:14 um,
34:15 or other measures of cognitive and non-cognitive outcomes
34:20 and see what are the underlying
34:22 aspects of teaching that could contribute to those skills.
34:24 So I'll leave it at that.
34:25 Thank you so much.
34:31 Thanks,
34:31 Christina,
34:32 that was fascinating.
34:32 Up next is Lee Crawford.
34:39 Thanks very much,
34:40 and
34:41 thank you all for being here.
34:42 Um,
34:43 so I'm gonna talk about lead exposure,
34:45 which is,
34:46 um,
34:46 I think still,
34:48 uh,
34:48 attention has been really growing in this space in the last couple of years,
34:51 but I think it's still a massively neglected,
34:53 um,
34:53 space.
34:54 And so,
34:55 uh,
34:55 in this paper,
34:55 we're trying to look at,
34:56 you know,
34:57 what does it,
34:57 what does this mean for education in,
34:59 in poor countries.
35:01 Um,
35:01 and this is a joint paper with colleagues at CGD that was
35:04 just published in this month's edition of the World Bank Research Observer.
35:08 So if there are any copies in the building,
35:09 I'd love to get one before we,
35:11 before I leave.
35:12 Um,
35:13 so the fact that lead is correlated with IQ is really not a new finding.
35:18 Uh,
35:18 this paper is from 1979,
35:20 um,
35:20 in the New England Journal of Medicine.
35:22 Um,
35:22 but what we,
35:23 you know,
35:23 IQ points doesn't mean a huge amount to most
35:26 of us development economists who work on education.
35:29 We tend to think about standardized effect sizes.
35:31 Um,
35:32 and so we,
35:33 we came to this with a question like how big of a deal is lead really for,
35:36 um,
35:37 education,
35:38 in poorer countries.
35:39 Uh,
35:39 what's the effect size and,
35:41 and does it,
35:41 does it,
35:42 Carry across from IQ into
35:45 the literacy and numeracy skills that we care about as well.
35:48 Um,
35:49 and
35:50 so this,
35:50 this
35:51 graph really frames the puzzle.
35:53 So on the left hand side,
35:55 most children in high income countries can now read a sentence by age 10.
36:01 It's not most children in low and middle income countries,
36:04 whereas 29%
36:06 we think around there
36:08 can read a sentence by age 10.
36:10 And at the same time,
36:11 lead is,
36:12 you know,
36:12 nearly a solved problem in high income countries,
36:14 just 2% of children with elevated blood lead levels,
36:18 but nearly half of children in low and low middle income countries
36:21 also have
36:23 elevated blood levels.
36:25 So we know that there's some relationship between lead exposure and
36:28 cognitive skills,
36:29 but we don't know
36:30 how much of this,
36:31 this gap is lead explaining.
36:33 And so that's what we set out to do with this paper.
36:37 And so we started by doing a systematic review of the literature out
36:41 there looking at all of the observational
36:42 studies that look at a correlation between
36:45 blood lead in a child and their test scores either on
36:48 IQ or on reading or,
36:49 or numeracy.
36:51 Um.
36:52 We make some adjustments to all of these studies to
36:54 try and make them comparable in the first time,
36:56 uh,
36:56 first place,
36:57 um,
36:57 which they aren't always necessarily.
37:00 We try and adjust for unobserved confounding
37:02 because most of these are observational studies
37:04 and do what we can to show you that this is a causal relationship,
37:08 um,
37:08 adjust for publication bias in the literature,
37:10 of which it does seem there's some,
37:11 but it's not,
37:12 um,
37:13 it doesn't fundamentally,
37:14 uh,
37:14 under,
37:15 under,
37:16 um,
37:17 ruin this relationship.
37:18 And then finally we do the simulation,
37:20 so we go to each country and we look at
37:22 what is the average blood lead level
37:23 and then we have this parameter from our
37:25 meta analysis showing the relationship between blood lead
37:28 and learning and say,
37:29 OK,
37:30 so what would happen if we could
37:32 reduce blood lead levels
37:33 in
37:34 low low and middle income countries
37:36 to the levels of lead in the US,
37:38 which is very low.
37:40 What would that do to learning?
37:42 Um,
37:43 so this is our meta analysis.
37:45 We find
37:46 286 estimates from 47 studies.
37:50 Most of these studies are on IQ,
37:51 40 studies.
37:54 The average
37:55 effect size there is 0.22 standard deviations for each log unit,
38:00 so that's roughly
38:01 a doubling or halving in blood lead.
38:03 So a reasonable effect size.
38:06 We also found a reasonable number of studies
38:08 that measured impacts on reading and maths,
38:11 and what's striking is that the
38:13 average effect size there is almost identical across these.
38:16 So it's,
38:17 we kind of think of IQ as maybe being something different,
38:20 but the correlation
38:22 with blood lead at least is very similar for IQ as it is for
38:26 reading and numeracy,
38:27 these
38:28 foundational skills that we tend to care about
38:31 as economists focused on human capital.
38:34 Um,
38:35 so that's a simple unadjusted correlation,
38:37 and obviously,
38:38 we,
38:39 we don't just want to take that.
38:40 So,
38:41 um,
38:42 we try to adjust that with apologies to,
38:44 uh,
38:45 Xavier Salah Martin.
38:47 We ran not quite 2 million regressions,
38:49 but we ran every possible
38:51 meta-analytic regression,
38:53 adjusting for different features of these studies.
38:55 So
38:56 a lot of these studies don't adjust for parental income or parental IQ,
39:00 um,
39:00 parental education,
39:01 which you really should.
39:03 And so we adjust in the meta analytic regression for
39:07 the studies that do and don't make that adjustment.
39:12 Our ideal study would be in a,
39:13 in a,
39:14 not in a high-income country and would be measuring reading or mathematics.
39:17 And so,
39:19 essentially,
39:19 when we,
39:20 when we go from a
39:21 completely
39:22 unadjusted estimate,
39:23 which is the red dot here,
39:25 which is the 0.22 standard deviation,
39:27 our preferred estimate
39:29 is over on the right,
39:30 the blue dots,
39:31 and it roughly halves,
39:32 so it's 0.12 standard deviations.
39:34 Um.
39:35 But you know,
39:36 it's fairly arbitrary and so you can see all the different
39:40 choices you could have made in this metro analytic progression.
39:43 There are some differences in the estimate,
39:45 but it's,
39:45 it's around the same kind of
39:47 uh
39:48 ballpark regardless of which choice you make
39:51 there.
39:52 So we,
39:53 we proceed with our 0.12 standard deviation reduction in learning for
39:58 a doubling in
40:00 a rough doubling in blood lead levels.
40:03 Um,
40:05 now it's a room of economists.
40:06 So are these correlations really causal is the obvious question.
40:09 Um,
40:09 we have,
40:10 I think we have a number of reasons to think they are.
40:12 Uh,
40:12 the first
40:13 being simply that,
40:15 These correlations are relatively robust to,
40:18 including
40:18 different controls and also the coefficient stability
40:23 methods for unobserved
40:25 confounders as well.
40:27 Second,
40:27 there's a really clear theory of change here.
40:29 There's a clear,
40:30 well understood biological mechanism by which
40:32 lead prevents brain making connections.
40:37 Third,
40:37 you know,
40:38 perhaps most persuasive is there's this increasing crop of natural experiments
40:43 in economics which does
40:45 which do claim to show causal effects,
40:47 and by and large the effect
40:49 when using a natural experiment is larger
40:52 than when just looking at the observational.
40:55 That's true,
40:55 I think,
40:55 for all 5 or so
40:58 studies which
40:59 allow you to show both an OLS estimate and an instrumental variable estimate.
41:04 And they all show bigger effects with the
41:07 instrumental variable estimate.
41:09 And finally,
41:11 perhaps ethically ethically questionable,
41:13 but there are some direct experiments with treating animals with lead
41:17 that do show causal effects on the cognition of
41:20 monkeys and rats.
41:22 So
41:23 I think it's quite a strong case that there's a causal relationship
41:26 here.
41:28 Um,
41:29 and so what,
41:30 what does this do?
41:31 Here's our,
41:31 here's our simulation.
41:32 Um,
41:33 we take for each country,
41:35 the,
41:35 uh,
41:36 the harmonized learning outcome,
41:37 which is a,
41:38 a similar kind of approach,
41:40 not as fine-grained as,
41:41 as Dev's approach to putting different countries on the same learning scale.
41:45 And on the scale,
41:46 it's roughly 500 points is the
41:48 rich country average.
41:50 And
41:51 this set of developing countries,
41:53 um,
41:54 around 100 points,
41:55 150 points below that.
41:57 And for each of these countries,
41:59 we say,
42:00 you know,
42:00 they have different levels of blood lead in each country,
42:02 and so the effect of
42:04 eliminating that lead would be different for each country,
42:06 but it ranges between 20 or 30 points on this
42:10 scale that,
42:10 that they could potentially gain
42:12 if they could completely eliminate,
42:14 um,
42:14 blood lead.
42:15 And so that's enough to
42:17 reduce 21% of the,
42:19 the gap between high-income countries and,
42:21 um,
42:22 poorer countries,
42:23 which I think is a big deal.
42:24 Like it's not something that we,
42:26 you know,
42:27 the education sector is rightly focused on improving
42:30 teaching and improving access to school.
42:32 Um,
42:33 but,
42:33 you know,
42:34 it really shows that,
42:35 um,
42:36 there are these other factors which really affect how children,
42:40 how prepared children are to learn when they turn up in the classroom.
42:43 It reminds me of,
42:44 you know,
42:44 when I was in,
42:45 uh,
42:45 graduate school learning about deworming as being the big flashy thing
42:49 and like the most cost-effective thing you could do in education,
42:52 which to
42:53 most people working in the education
42:55 in the Ministry of Education was just bizarre,
42:57 but I think there's deworming,
42:59 there's
43:00 lead,
43:00 there's growing evidence about air pollution,
43:02 there's all these other factors which affect children's readiness to learn.
43:06 Um,
43:06 so I think this is important.
43:09 Um,
43:11 Perhaps eliminating lead altogether is,
43:13 is a big ask,
43:14 and that's not something we can just wave a wand and,
43:18 uh,
43:18 easily accomplish.
43:19 Unlike deworming,
43:20 there's not just a simple pill you can take.
43:22 We need to
43:23 remove multiple sources of lead exposure.
43:26 Um,
43:26 so here we benchmark
43:28 both that full reduction
43:29 and some
43:31 feasible,
43:32 uh,
43:32 demonstrated interventions
43:33 and benchmark them against some education interventions.
43:37 So two popular,
43:38 um,
43:39 education approaches at the bottom.
43:40 Um,
43:42 providing information about how the,
43:44 the financial returns to more schooling
43:46 or teaching at the right level,
43:48 they're gonna get you between a
43:50 0.1 and 0.2 standard deviation effect size,
43:53 uh,
43:53 increase in learning.
43:55 Um,
43:56 so the complete reduction of blood lead
43:58 would get you more than either of those interventions.
44:01 Um,
44:02 but some of these other approaches,
44:04 they're gonna get you some,
44:05 Some gains.
44:06 So soil remediation is the top there.
44:08 That's quite expensive,
44:09 a particular polluted site if you dig up and remove all of the soil,
44:13 so perhaps not
44:14 super cost effective just on educational
44:17 outcomes.
44:18 But then some of these other interventions are not necessarily very expensive.
44:22 So
44:22 the leaded petrol ban,
44:24 which has been accomplished.
44:26 That had quite large effects on blood lead
44:29 paint remediation.
44:30 There's a lot of space left.
44:33 There's lots of countries that don't have bans on leaded paint yet,
44:35 so there's some
44:36 really low cost regulatory interventions there
44:39 which
44:40 could,
44:40 you know,
44:41 produce these meaningful
44:42 gains in in learning,
44:44 as well as all the health benefits.
44:47 Um,
44:50 So
44:51 I'll conclude.
44:53 Uh,
44:54 I have some time,
44:55 but that's cool,
44:57 um.
44:58 It's a,
44:58 it's a big deal for
45:00 education and it's,
45:01 I think,
45:02 massively neglected still with,
45:04 uh,
45:05 huge,
45:05 most of,
45:05 I think the impact
45:07 is potentially the health impacts.
45:10 Um,
45:10 there's a lot of uncertainty there,
45:12 um.
45:13 I think when people
45:15 calculate big numbers for the ultimate impact of lead,
45:18 probably most of the impact comes from
45:20 cognition and productivity and IQ detriment,
45:23 but there's also a big chunk which comes from
45:25 cardiovascular disease.
45:28 So
45:28 it's
45:30 we tried to put together the amount of
45:31 international funding which goes on lead exposure,
45:34 and it was,
45:35 I think,
45:35 about $15 million that we could find.
45:38 Um,
45:39 in recent years,
45:39 which is just a tiny amount of money compared
45:42 to the amount that's spent on health and education,
45:44 um,
45:45 globally.
45:46 Including from aid,
45:48 despite all the cuts,
45:49 there's still.
45:51 Yeah,
45:52 still a massively,
45:53 massively neglected issue.
45:55 Um,
45:56 and
45:57 the next question which we're kind of working on now is,
46:00 OK,
46:00 so what do you do about it?
46:01 What are the most important sources?
46:03 And it's really tricky.
46:04 We don't have a great
46:06 understanding of what the most important sources of lead exposure are.
46:09 There's,
46:09 I guess,
46:09 two big buckets consumer products and industrial sources.
46:15 And even at that high level,
46:16 it's not totally clear.
46:18 Um,
46:18 but there is some real low-hanging fruit there,
46:20 like,
46:21 um,
46:21 the,
46:22 the fact that lead paint is not banned in every country.
46:24 There's some great organizations working on this,
46:26 but
46:27 that's some real obvious basic
46:29 regulatory issues,
46:31 lead,
46:31 uh,
46:32 limits in water.
46:33 There's a whole list of different
46:35 regulations that exist in many OECD countries and exist in the US,
46:38 um,
46:39 but don't,
46:40 um,
46:40 everywhere.
46:42 national surveys
46:44 have shown to be quite influential in
46:48 Bangladesh and Georgia and Bhutan recently.
46:51 I think the World Bank is supporting a survey in Indonesia.
46:54 So in many countries,
46:56 including the UK where I come from,
46:57 we haven't done a national blood lead survey in
47:00 decades,
47:00 if ever.
47:02 And so this can be really powerful in showing policymakers like wow,
47:06 you know,
47:06 50% of your children,
47:07 33% of your children are
47:09 poisoned
47:10 has shown to be really catalytic in
47:13 creating action.
47:14 And then,
47:16 yeah,
47:16 we
47:17 hate to end every presentation saying we need more research,
47:19 but we really do need more to try and understand
47:22 those different sources.
47:23 And so
47:24 we're doing work at CGD on
47:27 battery recycling,
47:28 understanding how much of a problem is coming from informal
47:31 car battery recycling,
47:32 which seems to be a lot.
47:35 People just in their backyard are smashing open batteries which have lead in them
47:39 in order to recycle them,
47:40 and that dust just goes everywhere.
47:42 And metallic cookware seems to be a really big one.
47:45 So roughly half of the
47:47 aluminum
47:49 cooking pans that you can buy in Africa
47:51 have
47:53 an elevated lead level in them.
47:55 And because people are making them with scrap metal,
47:57 any metal they can get their hands on,
47:59 that's a big constraint.
48:00 And sometimes a little piece of lead will make its way into into that stew.
48:04 And so every time you cook with that pan,
48:06 you get a little dose of lead,
48:08 but we don't know how big of a problem that is.
48:11 So lots more research.
48:15 Thanks.
48:19 Thank you,
48:19 Lee.
48:20 And,
48:21 and here,
48:22 that's aluminum.
48:27 Next up we have Gabriella.
48:31 Thanks,
48:32 thank you everyone here and also online.
48:34 I'm very excited to be part of this panel.
48:37 Today I'm going to talk about school violence and I think
48:41 everyone in this room will agree that violence is intrinsically bad
48:45 and evidence has also shown that
48:47 violence
48:50 has negative effects in child development,
48:52 including in learning.
48:53 In this presentation,
48:54 I'm going to take you a step back to think about the measurement of violence.
48:59 So I'm gonna be talking about how can we
49:01 Advance or they improve the way we measure violence
49:05 in schools and I will be using evidence from Malawi.
49:09 I will start by sharing with you three facts
49:12 of what we know based on the existing data.
49:15 First of all,
49:16 we know that
49:17 children are experiencing high levels of violence in the school.
49:21 If we take the case of Malawi,
49:23 we observe that
49:24 42% of children have self-reported experience
49:29 a school bullying.
49:30 And if we look at sexual violence,
49:32 we observed that around 22% of girls
49:36 have experienced sexual violence and 16% of this is happening
49:41 in schools.
49:43 Another thing that we observed from the data is
49:46 that most of this data is focusing on adolescents,
49:49 so we know little about what is happening
49:52 to the younger population.
49:55 A third key point is that the existing global stories
49:59 or international stories that collect data on violence against children
50:03 are using different methods to collect data.
50:07 And are doing this at different locations.
50:10 For example,
50:11 existing services are done either using
50:14 face to face service or self-administered questionnaires,
50:18 and they are done either
50:20 at the school setting or at home,
50:22 but we don't have enough understanding of how this is
50:25 affecting the estimates or the measures we have about violence.
50:32 Across these surveys and also
50:34 um among us that are collecting data on sensitive issues,
50:38 there are
50:39 some of the challenges when we collect this data.
50:42 One of the key challenges is that this might have
50:45 some misreporting and will be collected with some degree of error.
50:50 Um,
50:50 and there are different factors that influence um.
50:53 That influence this.
50:55 One of them is that children that experience violence
50:59 might not want to speak up or share the
51:01 truth of their experiences of violence in a survey.
51:05 This might be explained by
51:06 fears of breaches of privacy,
51:08 retaliation,
51:09 stigma,
51:10 or embarrassment,
51:12 and this can lead them to having some systematic misreporting in our data.
51:17 Another factor that can lead to some error
51:21 relates to the fact that especially when we work with
51:23 young children there can be some misunderstanding of the questions
51:27 as well as some lack of attention when we are collecting this data
51:33 and that
51:33 this also can result in having
51:36 some error in our data,
51:38 an error that is likely to be more random.
51:42 So addressing this and trying to minimize
51:45 these um error is of crucial importance
51:48 both to have more
51:50 accurate numbers of the dimension of the problem,
51:53 but most importantly to ensure that when we are measuring
51:56 the effectiveness of interventions tackling violence or targeting violence,
52:01 we ensure that we are measuring this accurately and
52:04 not with bias.
52:07 So
52:08 this research is going to think
52:10 a bit more about this and it's going to try to contribute to this.
52:13 What we are going to do is that we
52:15 are going to try to identify which data collection methods
52:19 would give us more reliable measures of
52:22 a school bullying,
52:23 corporal punishment,
52:25 and sexual violence,
52:26 and we are going to focus on a younger population of children,
52:29 specifically on children aged 8 to 12.
52:33 And the way we're going to do this is that we design an individual level,
52:38 uh,
52:39 survey experiment
52:41 with 4 treatment arms where children
52:43 are going to be randomly allocated to any of these 4 treatment arms.
52:47 In this,
52:48 we
52:49 bury two main components.
52:51 On the one side we vary the data collection method
52:54 where we use either a face to face method
52:57 or
52:58 something that is called an audio computed assisted interviewing method.
53:03 For this,
53:03 what you have to imagine is a child that will have a tablet
53:06 and a head of cell phones,
53:07 and they will be hearing a recording
53:09 that is asking them different questions
53:12 um of violence.
53:13 This method is at the end in a bit more privacy relative to a
53:18 face to face method.
53:19 The other component that will be burying in the survey experiment is the location.
53:25 In other words,
53:26 the location where we administered the survey.
53:29 This will be either
53:31 at the school setting or at the home of the respondent.
53:36 And we decided to
53:37 use these different
53:40 methods of data collection to see to
53:42 what extent varying different levels of privacy
53:45 will lead to different levels of reporting of violence.
53:50 Something important is that for this study what we use as the control group
53:55 is the group of children that were allocated to
53:58 respond to this survey using a face to face method
54:02 at
54:02 the home,
54:03 and we use it,
54:04 uh,
54:04 we decide this to be the control group
54:07 considering that
54:08 a lot of surveys including DHS prioritize this type of data collection.
54:14 Before showing you the results,
54:16 two points that I wanted to highlight
54:18 and include that we don't observe any attrition,
54:23 uh,
54:23 differential attrition across our treatment arms,
54:24 and another key point that I wanted to highlight is that we took into account
54:28 a safeguarding measures
54:30 when we were collecting these types of data.
54:33 I'm happy to answer questions about this at the end.
54:38 So what do we observe in the results?
54:40 So I'm gonna be showing you two graphs with the key results.
54:44 This first result is showing you,
54:46 um,
54:47 uh,
54:48 measures for violence related to physical and emotional violence.
54:53 And here what you're going to see is a percentage point difference
54:57 relative to the control group that in this case is the children that
55:01 responded to the survey using a face to face method at home.
55:05 If we start by observing the first part of the graph,
55:09 we are in that case,
55:11 we are comparing
55:12 the face to face survey at home relative to a face to face survey at school.
55:17 So mainly here we are varying the location of the survey.
55:20 And as you can see here,
55:23 location doesn't seem to matter.
55:24 That much,
55:25 particularly for cases related to reporting
55:28 of corporal punishment or emotional violence
55:31 from teachers.
55:32 We do observe that
55:34 um
55:35 with at school,
55:37 we observe that it's eliciting higher responses in terms of
55:41 more experiences of physical bullying.
55:45 Now I'm gonna come we're going to talk more about Akasi.
55:49 So if we compare the face to face at home relative to the Akasi method,
55:54 this is the one where
55:56 children are listening to the questions in an audio.
55:59 What we observe is that regardless of the um of the location
56:04 there is
56:05 higher reporting of violence in the Akasi method.
56:10 For example,
56:10 if we look at the case of physical bullying,
56:13 we observe that there is a 9% point increase when we are collecting the data
56:20 with Akasi.
56:21 This is approximately a 20%
56:24 increase in the reports of
56:26 a physical bullying.
56:29 Let me move now to sexual violence.
56:32 In this case,
56:33 it's important to mention that sexual violence considers
56:36 different forms of sexual violence.
56:39 It covers
56:40 sexual comments,
56:41 uh,
56:42 unwanted or forced touch,
56:44 forced kiss,
56:45 pornographic videos,
56:47 or nude pictures,
56:48 as well as forced sex.
56:52 First,
56:52 we do the same.
56:53 Let's compare face to face
56:55 at home relative to face to face at school.
56:58 Here we observe again that children seem to be more comfortable
57:03 responding to the survey in the school environment and we observe higher levels
57:08 of reporting sexual violence in the school environment.
57:12 We then compare the face to face
57:15 um at home with Aasi
57:18 and
57:19 regardless of the location we observe high levels
57:23 of reporting of sexual violence with Akasi.
57:27 In fact,
57:27 here the disclosures of violence,
57:30 sexual violence are
57:32 very high,
57:33 more than 100%,
57:34 and even 4 times higher,
57:36 um,
57:37 in the case of sexual violence perpetrated by members of the school staff.
57:43 So two key points to mention at this stage is that it seems that location does
57:49 matter and that children are feeling more comfortable
57:51 reporting these experiences in the school environment.
57:55 And some of you might be
57:56 asking yourself
57:58 why school versus the home.
57:59 Something that we
58:01 identify in the study is that in the school there were more spaces
58:06 to conduct the survey.
58:07 So for example,
58:08 50% of the numerators
58:10 end up doing the survey in the schoolyard.
58:12 So at this point it was very clear for the child
58:15 that you were far away from the listening range of others.
58:19 So this provided,
58:20 let's say,
58:21 higher privacy than perhaps what what could be achieved.
58:25 At home where the story could be either inside the
58:27 home or at the front door and it was perhaps
58:30 harder to show that you were farther away from the listening range of others.
58:37 Keeping that in mind,
58:38 I will now show you,
58:40 uh,
58:41 I will now discuss briefly
58:42 the cost effectiveness of these different ways of collecting the data.
58:47 And here what I'm thinking of effectiveness,
58:50 what we consider to be more,
58:52 more effective is the fact that we are able to
58:55 elicit higher reporting of violence and of course
58:59 here we are assuming that this higher reporting
59:02 is coming from accurate reporting,
59:04 meaning that we are closer to the truth of what are the levels of violence.
59:09 And what we observe is that increasing
59:11 by 1% point the probability of reporting violence
59:16 costs around $5 when we will be using an Aasi method in a school,
59:23 and the amount almost uh doubles when we do the same but using a face to face method
59:28 so.
59:30 This,
59:30 uh,
59:30 is indicating that Aasi seems to be also a cost-effective tool to consider
59:36 when we are thinking of how to collect data
59:39 on violence.
59:41 Before concluding,
59:42 I would like to also discuss now two,
59:46 brief points.
59:47 Um,
59:48 the first one is that perhaps
59:50 also some of you are thinking to what
59:51 extent AA is actually giving us reliable measures.
59:54 Do we worry that
59:57 perhaps these are not.
59:58 Uh,
59:59 accurate.
1:00:00 And here we do several robustness checks.
1:00:03 One first thing that we find is that under
1:00:05 AAi there is a lower social desirability bias.
1:00:09 So this in a way is telling us that
1:00:11 it's likely that with Aasi children feel uh more comfortable
1:00:15 telling us their true experiences of violence or or have
1:00:19 a lower incentive to,
1:00:20 uh,
1:00:21 miss report.
1:00:22 A second thing that we do observe also in AAI is that
1:00:27 more children fail attention checks that we put in the questions
1:00:31 and that there were more children misunderstanding some of the questions.
1:00:35 We,
1:00:35 however,
1:00:36 do,
1:00:36 um,
1:00:38 a series of tests to see to what extent this would affect the
1:00:42 results I just explained and we observed that even when accounting for this,
1:00:48 our results hold.
1:00:50 A second question that you might have is
1:00:54 the extent,
1:00:54 uh,
1:00:55 how
1:00:55 do these numbers that I spoke today compared
1:00:58 to what we observed from existing surveys.
1:01:01 So to think more about this,
1:01:04 uh,
1:01:04 we look at the data from 15 countries that have
1:01:07 a survey that is called the Violence Against Children survey,
1:01:11 and they have a question on forced sex.
1:01:13 So we took that data and we observed that
1:01:16 based on that data,
1:01:17 1 in 500 children
1:01:21 reported experience forced sex.
1:01:24 If we assume the same levels of underreporting that we observed in this study,
1:01:30 those numbers should actually be something closer to
1:01:34 1 in 50 children that are.
1:01:36 Are being experienced and that have
1:01:39 actually experienced for sex
1:01:41 and it's possible that these numbers are still a conservative number because
1:01:46 in our study we focus on a younger population of children and for sex
1:01:51 tends to increase uh
1:01:53 increase with age so
1:01:54 it's possible that these numbers
1:01:57 are even conservative.
1:01:59 So just to conclude,
1:02:01 two main messages.
1:02:02 The first one is that
1:02:04 um the location of the survey seems to be important and it is important to consider
1:02:09 uh spaces that provide higher privacy.
1:02:12 A second point is that Aasi
1:02:15 seems to be effective in eliciting higher levels of true
1:02:19 reporting of violence and it's also more cost effective.
1:02:23 This,
1:02:24 however,
1:02:25 should not be interpreted as saying that
1:02:27 Aasi should be the gold standard,
1:02:29 but instead that relative to face to face
1:02:33 Aasi might be giving us
1:02:35 lower systematic misreporting or underreporting of violence.
1:02:39 So it's worth
1:02:41 considering these different
1:02:42 alternatives when we are thinking of collecting data
1:02:46 on on violence against children.
1:02:48 Thank you.
1:02:55 OK,
1:02:55 let me invite
1:02:56 all the panelists up.
1:02:57 Um,
1:02:58 can I just have by a show of hands,
1:03:00 um,
1:03:01 who is interested in asking a question?
1:03:03 I just want to get a sense of how,
1:03:05 how many questions in the room,
1:03:06 OK.
1:03:08 So I'll refrain from asking my own name.
1:03:10 I'll I'll.
1:03:16 Yeah,
1:03:16 so we have,
1:03:16 we have some time,
1:03:18 um,
1:03:18 so we can take a couple of rounds.
1:03:20 I,
1:03:20 I think we'll do what,
1:03:22 uh,
1:03:22 previous sessions have done,
1:03:23 which is to ask,
1:03:24 um,
1:03:25 maybe take 3 or 4 questions,
1:03:27 I guess since we have 4 panelists,
1:03:28 maybe 4.
1:03:29 Uh,
1:03:30 please keep it short,
1:03:31 keep it as a question if you don't mind.
1:03:34 Um,
1:03:35 so,
1:03:35 OK,
1:03:36 raise your hands again.
1:03:38 And then,
1:03:39 so,
1:03:39 OK,
1:03:39 we have one there,
1:03:40 although you've asked a couple of questions,
1:03:42 so I'll,
1:03:42 I'll come back to you,
1:03:43 but I'll give somebody else the first one.
1:03:45 So over here,
1:03:46 um,
1:03:47 and then we'll go over there
1:03:49 and then maybe we'll go there and then one more this side over here,
1:03:52 OK.
1:03:54 Yeah,
1:03:54 I'd like to ask Dev to comment on how this,
1:03:59 this method uh
1:04:00 compares to the HLO,
1:04:02 the harmonized learning outcomes approach,
1:04:04 which has been used by the bank and
1:04:05 underlines human capital index and was also mentioned by
1:04:08 Lee uh in his presentation.
1:04:21 Thank you.
1:04:21 I have a question uh for Gabriela.
1:04:24 I'm
1:04:24 Nasheen also from the Tech Development Impact Team.
1:04:27 So I wanted to ask what you thought of
1:04:30 practices within the self-reported surveys,
1:04:34 like,
1:04:34 you know,
1:04:34 list experiments,
1:04:36 changing the order of the questions,
1:04:38 and
1:04:38 other techniques you can use to
1:04:41 reduce the social disre.
1:04:43 desirability bias
1:04:44 as opposed to using the computer techniques and
1:04:48 um whether you have looked into it and what
1:04:50 have you seen differences in
1:04:52 like self-reported surveys that use these techniques versus ones that do not.
1:04:58 OK,
1:04:59 we had another question.
1:05:04 Hi,
1:05:04 I think my question is for Gabriela.
1:05:06 I think stand up
1:05:08 stand up,
1:05:08 sorry.
1:05:10 Hi,
1:05:10 uh,
1:05:11 my question is for Gabriela,
1:05:12 and I'm gonna go quick,
1:05:14 uh,
1:05:14 regarding violence,
1:05:15 such an important topic,
1:05:16 especially
1:05:18 within our community,
1:05:19 the African community.
1:05:20 So I wanted to ask if I missed it,
1:05:22 uh,
1:05:22 I'm so sorry,
1:05:24 but I wanted to check on the corporal
1:05:26 punishment if it was only on teachers,
1:05:30 uh,
1:05:30 but not family members,
1:05:32 and if it wasn't for a family from family members,
1:05:35 worse parents,
1:05:37 do you have,
1:05:39 um.
1:05:40 Like,
1:05:41 is there an opportunity to extend it
1:05:44 to look into family members' violence
1:05:47 that maybe could trigger the bullying that we see
1:05:51 at school,
1:05:53 maybe not for corporate punishment or sexual violence,
1:05:56 and also to tap into
1:06:00 their.
1:06:01 Presentation,
1:06:02 do you think that the household
1:06:04 income,
1:06:05 or maybe the geographical context also impact
1:06:09 the level of violence that we see in school?
1:06:12 Thank you.
1:06:14 Thanks.
1:06:14 And then,
1:06:15 yeah,
1:06:15 invitation.
1:06:18 Um,
1:06:19 yes,
1:06:19 I have a question for,
1:06:20 uh,
1:06:21 Deb,
1:06:21 which is,
1:06:22 um,
1:06:23 Just where do Bihar and Florida fall in the
1:06:26 relative distributions of seals for the two countries,
1:06:29 um.
1:06:31 And if you'll permit me organizers privilege
1:06:33 and squeezing in a second for Christina,
1:06:35 um,
1:06:36 just thinking about cognitive endurance,
1:06:37 just,
1:06:38 um,
1:06:39 you know,
1:06:39 when I think about,
1:06:39 so I grew up in Lucknow actually I spent 7 years of my childhood in Lucknow,
1:06:43 went to a small private school there.
1:06:45 When I think about,
1:06:45 um,
1:06:47 Some of my friends' lives,
1:06:48 ah,
1:06:48 in particular,
1:06:49 compare that with,
1:06:50 you know,
1:06:50 my seven year old son's life here,
1:06:52 um,
1:06:53 their life was much harder in many ways.
1:06:55 Is that something we should be thinking of as
1:06:57 amplifying or competing with the sort of cognitive endurance you talked about?
1:07:04 OK,
1:07:04 maybe we start with Gabriella and work back this way.
1:07:08 Sounds good.
1:07:09 Uh,
1:07:09 thank you for the two questions.
1:07:11 So,
1:07:11 on the first question,
1:07:13 definitely there are other methods that can be considered,
1:07:16 uh,
1:07:16 when we are trying to collect,
1:07:18 uh,
1:07:19 this type of data.
1:07:20 For this specific study,
1:07:21 we also consider whether list experiments would be an option,
1:07:25 um,
1:07:27 considering that with this experiment,
1:07:29 experiments there is also a
1:07:31 Um,
1:07:32 they are more difficult to explain and sometimes to understand,
1:07:35 particularly for young children,
1:07:37 we decided not to test these,
1:07:39 but it's definitely something to consider,
1:07:42 and overall I would say that when thinking of
1:07:45 which methods to use to collect this type of data,
1:07:48 it really depends on the
1:07:50 the population you are working with,
1:07:52 but
1:07:53 There is also other good research comparing this experiments with face to face,
1:07:58 particularly on the topic of
1:07:59 domestic violence and also showing some,
1:08:02 some mixed results,
1:08:03 but something worth to to
1:08:04 to explore if thinking of that
1:08:07 um.
1:08:08 And on the other point,
1:08:10 we only focus on corporal punishment from teachers.
1:08:13 So this study is
1:08:14 mainly focusing on forms of violence that happen in the school environment,
1:08:19 but definitely
1:08:20 something important to also study and also study.
1:08:23 You explain these connections between what happens
1:08:26 in the school environment and how that
1:08:28 might result in changes in bullying in the
1:08:30 school environment or differences in the likelihood of experience
1:08:34 sexual abuse.
1:08:35 So another point that is,
1:08:37 uh,
1:08:38 important to study
1:08:39 and you had another question at the end
1:08:42 that I am.
1:08:44 Definitely,
1:08:45 thank you.
1:08:46 Uh,
1:08:47 there are studies showing that these are drivers that help
1:08:50 to that explain uh likelihood of being a victim of bullying
1:08:54 and also sexual violence.
1:08:55 So there are already some studies making that connection,
1:08:58 uh,
1:08:59 as these being drivers that lead to
1:09:02 might contribute to more exposure of violence.
1:09:05 Thank you.
1:09:08 OK,
1:09:08 uh,
1:09:09 Christina.
1:09:11 Thanks so much for your question.
1:09:13 Um,
1:09:14 this is definitely something that we had in
1:09:15 mind when we started doing the experiment that,
1:09:17 you know,
1:09:17 the types of day to day lives that these children have,
1:09:21 it's,
1:09:21 it's very distracting environments compared to a lot of more students
1:09:24 in primary school in high-income settings where you'd actually have time
1:09:28 outside of school to focus for extended periods
1:09:30 of time on homework or other assignments.
1:09:32 These kids just weren't getting that type of of practice
1:09:35 when they're at home and and then in the classroom,
1:09:37 there's getting,
1:09:37 you know,
1:09:38 a few minutes per lesson
1:09:40 of kind of a focused individualized practice.
1:09:42 So it feels like the baseline level of of time
1:09:45 on focused practice is very low and there's
1:09:47 tremendous amount of distraction that's preventing this,
1:09:50 this kind of extended focused practice.
1:09:53 So I think the baseline level is low,
1:09:55 which definitely contributes to lower overall cognitive endurance.
1:09:59 And part of what we had in mind when we were
1:10:01 designing the intervention was kind of keeping in mind that,
1:10:03 you know,
1:10:04 their,
1:10:04 their time is scarce,
1:10:05 right?
1:10:05 There's already a lot of other things on teachers' plates in schools
1:10:08 on students' time once they're out of school
1:10:10 of supporting their family or doing other things.
1:10:13 And so we were hoping to basically say,
1:10:14 can we
1:10:15 take an existing class,
1:10:16 an existing lesson.
1:10:17 And kind of tweak it in some way so that,
1:10:20 you know,
1:10:20 for example,
1:10:21 what the teacher is doing,
1:10:22 they still get to teach the same content,
1:10:23 but we can change something about the pedagogy so
1:10:26 that they're also able to build this other skill.
1:10:28 Um,
1:10:29 and I think that's something hopefully for schools to think about as well too
1:10:31 is what are the types of teaching practices that allow you to not only
1:10:35 increase literacy and numeracy skills,
1:10:37 but also build these fundamental socio-emotional scores.
1:10:40 It doesn't have to be one or the other necessarily.
1:10:46 Great,
1:10:46 uh thank you so much for those questions.
1:10:48 Um,
1:10:49 so those of you don't know,
1:10:50 the HLO is a very impressive data product
1:10:52 that's released by the World Bank and covers a
1:10:54 much larger set of countries than um what Justin and I have covered in our paper.
1:10:59 Um,
1:10:59 so I think if you were to directly compare,
1:11:01 for example,
1:11:01 the average.
1:11:02 Test scores in
1:11:03 our sample to those
1:11:05 you would see that in general we seem to find is that the low
1:11:07 income countries tend to be doing worse than than is estimated by the HLO.
1:11:11 So
1:11:12 for example,
1:11:12 I think Chad is scoring about 0.2 standard deviations lower
1:11:15 in our measure than what the HLO is estimating.
1:11:19 But the second thing I'll say is that one of the key
1:11:22 advantages of our approach is that it really lends itself to
1:11:24 looking at the microdata and so we can really examine,
1:11:27 you know,
1:11:27 not just country averages or certain moments of the distribution,
1:11:30 but actually looking at,
1:11:31 you know,
1:11:31 how private schools or
1:11:33 gender or these other things are really interacting with test scores.
1:11:36 And the third thing I'll say is that
1:11:38 I think
1:11:39 what you know,
1:11:40 one thing that we emphasize in this talk is
1:11:42 that there's a big learning crisis going on.
1:11:44 I think
1:11:44 you didn't need me to tell you that.
1:11:46 I think you all knew that beforehand.
1:11:48 We didn't really need to have that task score data to show that,
1:11:51 but there are some other policy
1:11:52 implications,
1:11:53 I think,
1:11:53 of our work and in particular one thing.
1:11:55 And that I think was really salient to me from the meta-analysis that Lee was showing
1:11:59 is that,
1:12:00 you know,
1:12:01 Abhijit Singhman famously quips,
1:12:03 there's nothing standard about a standard deviation,
1:12:05 and there is,
1:12:06 I think,
1:12:06 a challenge that we have when we're comparing studies across places if we're
1:12:10 all measuring test scores in different places
1:12:12 or running our own interventions or things
1:12:14 that,
1:12:14 you know,
1:12:15 Christina does an amazing intervention in
1:12:17 improving cognitive endurance in India,
1:12:19 but we want to compare the test score effects to that.
1:12:21 To a different intervention to deworming,
1:12:22 for example,
1:12:23 we can't really,
1:12:24 there's nothing sensible about just comparing the standard deviation units.
1:12:28 And so we can actually use this type
1:12:29 of methodology that Justin and I have talked about
1:12:32 to try and link all these test scores on a common scale.
1:12:34 So all you have to do is next time you go out
1:12:36 and run your intervention,
1:12:38 include a few questions from one of these tests,
1:12:40 and suddenly we can actually support all of our interventions on
1:12:44 Tim's units,
1:12:45 and then we can actually have these on the same scale.
1:12:47 And then I think that's kind of related to your question,
1:12:49 Sean,
1:12:49 as well.
1:12:49 So,
1:12:50 um,
1:12:51 one thing that we're kind of limited by in the existing data of these
1:12:54 sampling frames of these tests is that
1:12:56 they're concentrated essentially among specific grades and specific
1:13:00 year in countries.
1:13:01 And so when we went to,
1:13:03 you know,
1:13:03 Tims,
1:13:03 for example,
1:13:04 was 4th grader.
1:13:04 In 2011,
1:13:06 so we're going back to Florida and doing all of middle school or you know,
1:13:10 a bunch of grades in Bihar.
1:13:11 So it's not quite obvious how to exactly compare those,
1:13:14 but you do see a huge kind of range and actually quite a bit of overlap
1:13:18 in general it seems that like Bihar in particular
1:13:20 of course at the lower end of the distribution.
1:13:22 And
1:13:23 I think one
1:13:24 big thing that emerges from Justin and I's work is that there's a big,
1:13:27 uh,
1:13:27 empty spot on that map for India and China,
1:13:30 and,
1:13:30 um,
1:13:31 lack of standardized tests for those countries is a huge kind of policy problem for,
1:13:35 for all of us who care about kids learning.
1:13:38 Thanks.
1:13:39 So I'm gonna go down and have,
1:13:40 I sort of,
1:13:41 I have a few questions myself,
1:13:42 so I'm gonna take the chance to insert them.
1:13:45 Um,
1:13:45 so I actually had the same question as Gabriel for you,
1:13:47 obviously coming from the World Bank,
1:13:48 but I had another question too,
1:13:49 which is,
1:13:51 um,
1:13:52 I mean,
1:13:52 you inferred you didn't say,
1:13:53 uh,
1:13:54 that the lack of,
1:13:56 would you agree that the lack of association between national income
1:14:00 and um
1:14:01 And the test scores is because of the
1:14:03 quality of education service delivery across the country.
1:14:06 I mean,
1:14:06 that's sort of the obvious inference,
1:14:08 but I don't know if that's true or not.
1:14:09 And then the other question for you,
1:14:10 Dev is
1:14:12 on the inequality result,
1:14:13 is it,
1:14:13 is it,
1:14:13 could it just be that the
1:14:15 income different when the genie is higher,
1:14:17 the income differential is just higher between the top and the bottom
1:14:20 and so you get this coefficient,
1:14:23 um,
1:14:24 that's a question.
1:14:25 Um,
1:14:26 to Christina,
1:14:26 my question is,
1:14:27 um,
1:14:29 As you were presenting this,
1:14:30 I say,
1:14:31 I was thinking to myself,
1:14:32 wow,
1:14:32 you're just undermining the big five interpretation of perseverance
1:14:36 as a trait and you're saying as a skill,
1:14:38 but then at the end you sort of
1:14:40 treat perseverance as different than what you were doing,
1:14:43 and then you also also refer to sustained attention.
1:14:45 So I guess,
1:14:47 would you mind just elaborate a little bit on
1:14:49 how these are different or similar dimensions of,
1:14:51 of,
1:14:52 of,
1:14:52 of,
1:14:53 of,
1:14:53 of,
1:14:55 of
1:14:55 The same or different things.
1:14:57 To leave,
1:14:58 um,
1:14:59 this,
1:14:59 I know this is completely unfair because if you could have done this,
1:15:02 I know that you would have done this,
1:15:03 which is the obvious cost-effectiveness question.
1:15:06 Um,
1:15:07 and so
1:15:10 What are your thoughts
1:15:11 on cost-effectiveness of,
1:15:12 of,
1:15:12 of abatement and how that would relate to your
1:15:14 work,
1:15:15 but maybe the fairer question
1:15:16 is,
1:15:19 You,
1:15:19 you project this onto sort of HLO numbers which are pegged to either kind
1:15:23 of end of primary school or somewhere sort of late in secondary school,
1:15:28 and then that got me thinking about,
1:15:29 well,
1:15:32 where,
1:15:32 what's the timing of exposure and what grades are these measures
1:15:37 actually being measured and is that a fair projection on,
1:15:40 you know,
1:15:40 from the studies,
1:15:41 which I don't know,
1:15:42 but you do,
1:15:43 um,
1:15:44 to how you're trying to quantify the impact is that.
1:15:48 Uh,
1:15:49 is that fair?
1:15:50 And then to Gabriela,
1:15:50 I guess.
1:15:52 Oh,
1:15:53 you,
1:15:54 you're comparing a lot of survey methods.
1:15:56 I guess I'm,
1:15:57 I'm curious about,
1:15:58 I mean,
1:15:59 you probably did this in the piloting,
1:16:00 which is,
1:16:01 what about non-survey methods and social workers and how they,
1:16:04 you know,
1:16:04 how do,
1:16:04 how do,
1:16:05 how do these surveys compare to
1:16:06 kind of what you get out of a non-survey type
1:16:10 um
1:16:11 exchange with the,
1:16:12 with the,
1:16:12 with the respondent.
1:16:14 So,
1:16:14 I don't know,
1:16:14 should we start on this end this time?
1:16:19 And then
1:16:20 hold,
1:16:20 keep your questions because we'll do another round in the in the room.
1:16:24 Great.
1:16:24 So,
1:16:24 um,
1:16:25 Justin and I are big fans of cross-country regression.
1:16:28 So,
1:16:28 um,
1:16:28 we did do the
1:16:30 regression of test scores on education spending per capita,
1:16:34 and,
1:16:34 and there's a positive relationship there.
1:16:36 Um,
1:16:36 so,
1:16:37 to whatever extent I think
1:16:39 that,
1:16:39 yeah,
1:16:40 so I'll maybe leave the interpretation up to the rest of you,
1:16:43 but,
1:16:43 um,
1:16:43 uh,
1:16:44 we do find that positive correlation.
1:16:45 And then on your second point,
1:16:47 um,
1:16:47 so yeah,
1:16:47 I think one of the kind of key advantages
1:16:49 of
1:16:50 that.
1:16:50 regression that we're allowed to run is
1:16:53 we can put everything in the same unit,
1:16:54 so it's household income,
1:16:56 you know,
1:16:57 in PPP dollars and it's test scores in TIS units.
1:17:00 And so
1:17:01 the kind of interpretation of that coefficient in that geni plot is
1:17:04 for,
1:17:05 you know,
1:17:06 a certain log income increase.
1:17:08 What is the increase in test scores?
1:17:09 And so I think
1:17:11 differences in the overall distribution won't really be,
1:17:13 won't really kind of matter there.
1:17:14 It's going to all be
1:17:16 per the same dollar income.
1:17:20 Great,
1:17:21 really great question.
1:17:22 I,
1:17:22 I'm not a psychologist,
1:17:23 so I will butcher some of this,
1:17:25 but um the way I kind of think of separating these different concepts.
1:17:28 So in terms of the kind of big five personality traits,
1:17:31 I don't know how much evidence we have of the extent
1:17:33 to which this can be kind of developed through training otherwise,
1:17:36 but in the category of kind of executive functioning skills,
1:17:39 so working memory,
1:17:41 sustained attention,
1:17:42 which
1:17:42 I kind of say it is very similar to cognitive endurance.
1:17:45 I think that psychologists measure sustained attention a little
1:17:48 bit different than how we're measuring cognitive endurance,
1:17:50 but I think of them as very fundamentally,
1:17:53 ah,
1:17:53 very similar concept.
1:17:54 So,
1:17:55 you know,
1:17:55 I'm,
1:17:55 I'm happy to
1:17:57 have you use them in in exchange,
1:17:59 um,
1:18:00 and then perseverance.
1:18:01 I also put in the same kind of category of other executive functioning skills.
1:18:04 That
1:18:05 there is some evidence for
1:18:06 for working memory perseverance and,
1:18:09 um,
1:18:09 and now I sustained attention or cognitive endurance
1:18:12 that this can be developed through practice.
1:18:14 Um,
1:18:15 so it does seem like this is kind of an area that's
1:18:18 there's scope for schools to potentially influence or um parenting,
1:18:22 for example,
1:18:23 and in particular in our study,
1:18:25 the way we try to separate this is that
1:18:27 We have another kind of a variation that we introduced.
1:18:30 So for these tests,
1:18:32 we also have some where kids get like a toy prize um based on their performance
1:18:37 and you see this really interesting pattern that when you have an incentive,
1:18:40 um,
1:18:41 kids work harder at the beginning of the test.
1:18:43 They do better at the first few questions.
1:18:45 So you know,
1:18:45 incentives work.
1:18:46 Yay,
1:18:47 economics.
1:18:48 Um,
1:18:49 but you actually see them decline even faster over the length of the test,
1:18:53 right?
1:18:53 They,
1:18:53 they start to really flag very quickly.
1:18:56 They're really motivated for the first few minutes,
1:18:58 but then they do quite poorly
1:18:59 at the end of the test,
1:19:01 um,
1:19:01 even when there's this incentive prize,
1:19:03 and this doesn't crowd out any of the effect of our intervention.
1:19:07 So it seems.
1:19:08 Almost like this is kind of a separate mechanism.
1:19:10 There's the kind of motivation channel
1:19:12 coming from the incentives,
1:19:13 and this is separate from
1:19:15 um the,
1:19:16 the cognitive endurance or sustained attention muscle,
1:19:19 and I think of it similar to like,
1:19:20 you know,
1:19:21 you could give me a million dollars to,
1:19:23 you know,
1:19:24 I run a 4-minute mile,
1:19:27 like,
1:19:27 you know.
1:19:28 I'm going to have a lot of motivation.
1:19:30 It doesn't mean I have the kind of musculature to be able to perform that.
1:19:33 We think of that similarly with sustained attention,
1:19:35 that you can have the incentives to want to complete some tasks,
1:19:38 but you need the underlying kind of attention muscle to
1:19:41 be able to then capitalize that and focus over time.
1:19:44 So that's kind of how we think of separating them.
1:19:49 Um,
1:19:50 thanks.
1:19:50 So the,
1:19:50 the first question was about cost-effectiveness,
1:19:52 and there I think
1:19:53 we,
1:19:54 we,
1:19:54 we'd looked,
1:19:54 we'd looked for all the studies we could find on,
1:19:57 you know,
1:19:57 what works to address lead exposure and this is,
1:19:59 there's hardly any,
1:20:00 so it's difficult to draw
1:20:02 firm conclusions,
1:20:03 um.
1:20:04 There are some fantastically cost effective stories out there.
1:20:07 So the story of
1:20:08 the banning of leaded petrol and gasoline globally,
1:20:11 the World Bank played a big role in that.
1:20:12 It happened in the US and in other high income countries.
1:20:15 There was a big conference organized in Africa
1:20:18 and essentially they all agreed that,
1:20:20 oh yeah,
1:20:20 we should do this with,
1:20:22 you know,
1:20:23 support from the UN
1:20:24 and the World Bank,
1:20:25 and they all just did it in the space of a few years
1:20:27 and it was just that cost very little and had a huge,
1:20:30 huge impact.
1:20:32 Um,
1:20:32 similarly,
1:20:33 more recently,
1:20:34 the NGOs working on lead paint regulation.
1:20:36 It's not that there's a really big powerful pro-lead paint lobby.
1:20:40 It's like they just don't realize that it's that big of
1:20:42 a problem and it costs a bit to reformulate the paint,
1:20:45 or there might be one or two holdout companies,
1:20:47 but it's just,
1:20:48 it's not that big of a,
1:20:50 it's an amazing like rare occurrence where actually a bit of data
1:20:53 and evidence really can swing policymakers because it's just they don't,
1:20:56 haven't thought about it and don't have a strong prior.
1:21:00 And then in Bangladesh,
1:21:02 where
1:21:03 it costs next to nothing for one PhD student to dig
1:21:05 around and try and find what the key source was,
1:21:07 and she discovered
1:21:09 they were adding lead to turmeric,
1:21:11 and so they just went down to the market and they confiscated it
1:21:14 all and you know the government was very didn't know about this,
1:21:17 was very happy to enforce the existing regulations
1:21:20 and,
1:21:21 you know,
1:21:22 the turmeric producers were very happy to stop
1:21:24 doing it because they didn't realize they were poisoning
1:21:26 all of their customers' children.
1:21:28 So there are these like phenomenal opportunities out there.
1:21:32 Some of this is,
1:21:34 you know,
1:21:34 cleaning up
1:21:36 a decades old mining site where there's just soil across the whole town as there is in
1:21:40 Kabwe in Zambia.
1:21:42 I mean,
1:21:42 currently they're trying to sue
1:21:44 Anglo American and the London court to try and get
1:21:48 them to pay for that clean up,
1:21:49 but that's that's expensive,
1:21:50 so
1:21:51 the worst sites,
1:21:51 it's not always going to be super cost effective,
1:21:53 but a lot of the regulation is
1:21:56 probably as
1:21:57 And then on the age of the kids,
1:21:59 I think
1:21:59 this is lifelong
1:22:01 impairment to your brain,
1:22:02 and so we see effects on teenagers with crime and
1:22:06 violence
1:22:07 and so I think it's you know these studies
1:22:10 do show a range of ages of the children.
1:22:15 I think there's,
1:22:16 we don't know exactly what the cutoff point is.
1:22:18 We think it matters most under 6
1:22:21 for children as your brain is still developing most rapidly,
1:22:24 but you know,
1:22:25 we don't know if
1:22:27 if you remove lead exposure for 10-year-olds,
1:22:29 is that going to help them?
1:22:30 It probably will,
1:22:31 but
1:22:32 there's a bit of a lack of clarity there.
1:22:34 Thanks.
1:22:37 Thanks,
1:22:37 that's a great question.
1:22:39 So in our research,
1:22:40 something that we did um
1:22:43 is that
1:22:45 Our team included not only enumerators but social workers as well.
1:22:49 So this happened
1:22:51 during data collection that
1:22:52 every child had the opportunity to speak and was
1:22:55 offered the opportunity to speak to a social worker,
1:22:58 and that gives us
1:23:00 quite unique information and in total of the 6000 children that were surveyed,
1:23:05 25 agreed to speak to a counselor.
1:23:09 Um,
1:23:09 and that gives us unique information in the sense that
1:23:13 through this one on one conversations where you could build more trust,
1:23:16 children also open up about other experiences
1:23:19 of violence that happened outside the school,
1:23:22 including domestic violence.
1:23:23 We are also able to identify if
1:23:26 children were misunderstanding some of the questions,
1:23:29 so
1:23:29 that's why.
1:23:30 At the end of the presentation,
1:23:31 I mentioned that we
1:23:32 did some also analysis for correcting for those
1:23:36 children that misunderstood some of the questions.
1:23:39 Um,
1:23:40 so these other alternative methods of trying to do more
1:23:45 one on one conversations with social workers are definitely a good option uh to.
1:23:51 Try to understand how to get this measurement right,
1:23:54 how to ask these questions
1:23:56 in,
1:23:57 in the best possible way to,
1:23:59 to ensure that children are understanding.
1:24:01 And this makes me remember the paper by Chris Blackman
1:24:04 and others in Liberia that I think they also did
1:24:07 use a similar method to,
1:24:10 to do this.
1:24:10 So something that we can consider as well for
1:24:13 this type of studies with children.
1:24:17 Thanks.
1:24:18 OK,
1:24:20 so let's do one more round in the room,
1:24:22 uh,
1:24:22 maybe I know you had a question,
1:24:24 um,
1:24:24 maybe 3 questions total,
1:24:26 and then we'll do,
1:24:27 OK,
1:24:27 2 questions I'm told,
1:24:29 um,
1:24:29 one here and then maybe 1 on the side,
1:24:32 but keep it short,
1:24:32 keep it a question and then
1:24:34 we'll have a response.
1:24:35 Thank you very much for the opportunity and thanks for brilliant.
1:24:40 Um,
1:24:40 Dave,
1:24:40 um,
1:24:41 you indicated that um
1:24:43 You know,
1:24:43 girls outperform boys on
1:24:48 across all
1:24:49 income levels,
1:24:51 but then when it comes to maths,
1:24:52 they only outperform the boys at the lower income level.
1:24:56 I just wanted some explanation about this finding.
1:24:58 Thank you.
1:25:01 OK,
1:25:01 any question on this side?
1:25:03 OK,
1:25:03 over there.
1:25:06 I'm curious with the new wave of education,
1:25:08 especially when it comes to AI and
1:25:10 um
1:25:11 basically you have to review test taking
1:25:14 as a whole intrinsically,
1:25:15 um,
1:25:16 what do you think that is going,
1:25:18 how,
1:25:18 how is that going to affect standardized testing
1:25:20 and how are the results that we're seeing across different countries
1:25:24 maybe they should be different because the skills
1:25:27 that you need in places are different.
1:25:30 Um,
1:25:30 is that something that
1:25:32 maybe we should take into consideration when comparing across countries?
1:25:39 I guess both of those were kind of
1:25:41 sent your way,
1:25:42 so why don't you take a shot?
1:25:44 Great.
1:25:44 Um,
1:25:45 so yeah,
1:25:45 I guess,
1:25:45 so this first question,
1:25:47 it's a great question,
1:25:48 and I don't have any good real answers.
1:25:50 Maybe I'll just leave you with one
1:25:51 striking statistic that stuck in my mind,
1:25:53 which is,
1:25:54 um,
1:25:54 guess what country in the world has the largest test score gap between boys and girls
1:25:59 for 4th graders.
1:26:00 Uh,
1:26:01 it's actually Saudi Arabia,
1:26:03 and girls are way outperforming boys there.
1:26:05 So girls are scoring about 0.4 standard
1:26:07 deviations higher than boys in Saudi Arabia.
1:26:10 Um,
1:26:10 and I've never actually quite understood that statistic.
1:26:13 They don't wear the origin of that statistic either.
1:26:15 Um,
1:26:15 so I think there's lots of open questions here that would be great to answer.
1:26:18 And I'm sorry,
1:26:19 could you guys maybe was saying your question one more time.
1:26:28 Yeah,
1:26:28 it's a great question,
1:26:29 um,
1:26:30 and again,
1:26:30 yeah,
1:26:30 I don't think I have any really good answers to the predicting the future about,
1:26:34 uh,
1:26:34 especially on the AI side.
1:26:35 Um,
1:26:36 I think,
1:26:37 you know,
1:26:37 one thing that's really emerging from
1:26:39 what we're seeing is that there's a huge amount
1:26:40 of inequality that does seem to be somewhat systematic,
1:26:43 and this is kind of what really to what you were saying earlier
1:26:45 in ways that we should be able to predict and so.
1:26:48 Um,
1:26:48 you know,
1:26:48 we,
1:26:48 we should be able to kind of look
1:26:49 at those different gradients in which countries are doing
1:26:52 better and worse and try to understand a
1:26:53 little bit better about what features of the,
1:26:55 of the education systems are worse,
1:26:57 and I know,
1:26:58 for example,
1:26:58 Jishnu has been doing some really amazing work on
1:27:00 thinking about
1:27:02 education problems at a systems level,
1:27:04 and I think
1:27:05 to me that's actually certainly the most exciting types of work out here.
1:27:08 OK,
1:27:09 well,
1:27:09 so I'm gonna weigh in a little bit.
1:27:10 I'm not going to weigh in on the AI.
1:27:11 We had a whole session yesterday.
1:27:13 Uh,
1:27:14 on the other one,
1:27:14 I mean,
1:27:15 Lee,
1:27:15 Lee showed the,
1:27:16 the,
1:27:16 the chart where
1:27:17 was it 29% of kids aged 10
1:27:20 couldn't understand
1:27:21 a basic sentence,
1:27:23 right?
1:27:23 They couldn't,
1:27:24 so.
1:27:25 That's a pretty foundational skill and
1:27:27 one that's required pretty much everywhere.
1:27:30 So
1:27:31 parts of this are just
1:27:32 um apply everywhere I would argue.
1:27:35 Um,
1:27:35 yes,
1:27:36 there might be more subtle skills that,
1:27:37 that,
1:27:37 that
1:27:38 vary across locations,
1:27:39 but I don't think that's really what's driving a lot of what was here.
1:27:42 Anyway,
1:27:43 sorry,
1:27:44 you
1:27:44 didn't want to hear me pontificate,
1:27:46 um.
1:27:47 Please join me in thanking the panel.
1:27:49 Uh,
1:27:50 that was great.
1:27:51 Thank you very much.
1:27:53 And I don't know if you have any.
1:27:55 No more announcements.
1:27:56 I think there are drinks somewhere here,
1:27:59 so
1:27:59 see you there.
- add-style
- lp-body-content
A video recording of Session 7 on Day 3 of The Annual Bank Conference on Development Economics 2025 "Development in the Age of Populism." This session discusses "Education Policy ".
CHAIR: Deon Filmer, Director of the Development Research Group, World Bank
PANELISTS:
Paper 1: The Global Distribution of Skills (Dev Patel, Harvard University)
➜ Presentation | ➜ Paper
Paper 2: Cognitive Endurance as Human Capital (Christina Brown, University of Chicago)
➜ Presentation | ➜ Paper
Paper 3: Lead Exposure and Learning Outcome Stagnation (Lee Crawfurd, Center for Global Development)
➜ Presentation | ➜ Paper
Paper 4: Advancing the Measurement of Violence in and Around Schools (Gabriela Smarrelli, Center for Global Development)