00:00 OK,
00:00 we can get started.
00:01 Um,
00:01 so welcome everyone to the,
00:04 the seminar of the Strategic Impact Evaluation Fund.
00:07 We are
00:08 very excited to have this presentation today.
00:11 Um,
00:11 this is,
00:12 uh,
00:12 I'm just gonna give you a brief introduction before passing it on.
00:16 This is one of my favorite
00:18 evaluations in our portfolio for,
00:20 for many reasons.
00:21 Um,
00:21 this is was part of our COVID recovery window
00:24 where,
00:25 um,
00:25 we had
00:26 tests of what could be done using digit.
00:28 Digital technology to recover learning and the loss of skills after the
00:33 the pandemic,
00:34 um,
00:34 and I really like this evaluation for a number of reasons.
00:37 One,
00:38 it's in a part of Pakistan that researchers normally don't go to.
00:42 Two,
00:42 it's taking,
00:44 um,
00:44 the idea of teaching at the right
00:45 level and asking whether teachers actually need some
00:49 sort of assistance,
00:51 um,
00:51 in order to pull this off,
00:52 not taking,
00:53 not,
00:54 you know,
00:54 assuming that the teacher is going to be able to do everything,
00:56 um,
00:57 on.
05:57 Uh,
05:58 so I've heard also that some of the,
05:59 this research,
06:00 uh,
06:01 might be used for,
06:02 for the work that is being in one of the,
06:04 uh,
06:05 projects in,
06:05 in Punjab now.
06:06 So,
06:07 so I think we see already like interest on,
06:10 on,
06:10 on how to do this,
06:11 uh,
06:11 TR,
06:12 uh,
06:12 with technology,
06:13 not technology.
06:14 Uh,
06:15 with that,
06:15 I'll hand it over to Tahir who we will,
06:17 uh,
06:18 present,
06:18 um,
06:19 and thank you to,
06:20 uh,
06:21 Tahir who has been like,
06:22 uh,
06:23 really great to work with and with Zenab and Maliha and,
06:25 and,
06:26 um,
06:27 and.
07:27 And I'll be happy to talk more about it.
07:29 OK,
07:30 so here's the story.
07:31 Uh,
07:32 so I'll,
07:32 I'll start with motivation.
07:33 This crowd doesn't need a lot of motivation on this issue,
07:36 but,
07:36 uh,
07:37 this is what we talk about,
07:38 OK?
07:39 So this is the learning crisis.
07:40 This is a slide that we have used in many papers.
07:42 This is very common.
07:44 There is a learning crisis
07:46 in countries like Pakistan.
07:47 There's been massive improvements in school attendance,
07:49 and massive is the right word
07:51 at all levels,
07:52 but
07:53 genetically
07:54 this is true around the world that learning levels
07:58 are really low.
07:59 So the bank has talked a lot about the learning crisis.
08:02 So that's very clear as to where we are coming from.
08:05 So this is Pakistan.
08:06 This is 2022,
08:08 uh,
08:08 when we did our baseline testing for this project.
08:12 Uh,
08:12 this was a lot of kids.
08:13 This is about 200,000 kids,
08:15 uh,
08:16 for which this data I'm showing.
08:18 Focus on the fourth grade,
08:20 uh,
08:21 numbers,
08:21 and then you can see,
08:23 you know,
08:23 this English,
08:24 math,
08:24 and Urdu,
08:25 uh,
08:25 you can see in all of them,
08:27 uh,
08:28 kids need a lot of remediation,
08:30 you know,
08:31 for example,
08:32 Even in Urdu,
08:33 about 25% of the kids are,
08:35 are,
08:35 are 2 to 3 years behind,
08:37 uh,
08:38 easily,
08:38 and,
08:39 and,
08:39 and the gray bar at the top
08:41 are kids that do not need remediation,
08:43 right?
08:44 So,
08:45 so the,
08:46 the,
08:46 the issue was very stark.
08:48 Uh,
08:49 this is coming out of COVID,
08:51 uh,
08:51 a year after COVID,
08:53 so,
08:53 so maybe starker than usual,
08:55 but,
08:56 but I think this is,
08:57 we are,
08:57 we're testing some stuff in LEAPS right now.
09:00 Uh,
09:00 in Punjab and,
09:01 you know,
09:02 similar kinds of results,
09:03 uh,
09:04 uh,
09:04 happened there.
09:06 So what do we do?
09:06 So this is something that people don't usually talk about or don't present data.
09:12 Even though this is a problem that has been mentioned,
09:15 particularly in to type situations,
09:17 which is this learning variation within the classroom,
09:20 OK?
09:21 So each bar is,
09:23 is a classroom.
09:24 These are 4th graders.
09:25 So most of them is a single classroom.
09:27 Some have more sections,
09:27 but generally it's one.
09:29 So what you can see is that the dotted line,
09:31 which is going from left to right,
09:33 so these are put according to their
09:35 school average learning level.
09:36 So you can see the gap between across schools,
09:39 right,
09:39 which is,
09:40 which is between -2
09:42 on the lowest end
09:43 to about 1 standard deviation above the mean,
09:47 uh,
09:48 which is across,
09:49 but each,
09:49 the length of each bar is within the,
09:51 within the classroom variation.
09:54 And you can see that within the classroom variation,
09:56 if anything,
09:57 is much greater than across school variation.
10:00 And it's quite large.
10:02 So,
10:02 I think these are the two problems,
10:04 right?
10:04 Low learning levels,
10:05 high within classroom variation in learning.
10:08 I mean,
10:08 it's,
10:09 I don't know if you have seen graphs like this,
10:10 but I would really suggest
10:12 we want to focus on this,
10:13 uh,
10:14 more,
10:14 much more so moving forward explicitly.
10:17 OK.
10:18 So what's the solution?
10:19 The solution is very clear
10:21 for learning level variation within the classroom,
10:24 you know,
10:24 targeted instruction type programs,
10:27 uh,
10:27 are designed for particularly that targeted
10:30 help for students who are falling behind
10:32 and sorting students into groups based on their learning levels rather than age
10:36 for part or all of the day,
10:38 right?
10:39 So this is,
10:39 I'm,
10:39 I'm a member of the GEP panel,
10:41 and you know,
10:42 the GEP has really highlighted this as.
10:44 As,
10:44 as a best buy,
10:46 uh,
10:46 Pratham does a lot in India,
10:48 Rukmini Banerjee,
10:49 and,
10:50 and,
10:50 and there's a bunch of papers on it,
10:52 uh,
10:52 around the world,
10:53 uh,
10:54 of,
10:54 of,
10:54 of,
10:54 of these
10:55 things being used around the world.
10:57 The second is structured pedagogy.
10:59 Structured pedagogy is also a,
11:00 a the best buy,
11:02 uh,
11:02 which is a bundled intervention that provides
11:04 teachers with necessary support and all that.
11:07 So what we do is,
11:08 is really combine these two things,
11:10 right?
11:10 So we explore this question.
11:12 In a representative sample
11:15 of fourth grade teachers,
11:16 the program was done for all grades 1 to 5,
11:19 but this is the study sample,
11:21 uh,
11:21 about 2 24,000 or so fourth grade students,
11:25 19916 public primary schools,
11:27 they were
11:28 randomly selected in these two districts,
11:30 Muran and Peshawar in the KP province of Pakistan.
11:33 This is the map,
11:34 and you can see that,
11:36 uh,
11:36 these are the schools in these two districts.
11:39 These are the two.
11:40 Peshawar is the largest district in KP and Milan is,
11:43 is,
11:43 is pretty large.
11:44 What did we do?
11:45 They were randomly assigned
11:47 to implement a 40 day,
11:48 these schools foundational learning program
11:51 called TEP targeted instruction implemented during school hours by
11:55 regular teachers.
11:56 Uh,
11:57 these are the summary statistics on these schools.
11:59 There are more,
12:00 have more data,
12:00 but you can just see,
12:02 uh,
12:02 to highlight,
12:03 uh,
12:04 median school primary enrollment is about 152.
12:08 Uh,
12:10 slightly more male schools than female.
12:13 Uh,
12:13 most of the sample is pretty rural.
12:16 Uh,
12:17 very few urban schools in these districts.
12:19 You can see the median number of teachers in a,
12:21 in a school is 5.
12:23 So it's they're pretty small schools,
12:25 uh,
12:26 more male than this.
12:28 Median teacher age is 37 and,
12:30 and with a fairly
12:31 large standard deviation.
12:33 Uh,
12:33 and,
12:34 uh,
12:34 I want to highlight this proportion teacher new hired.
12:37 There was a,
12:38 a new hiring scheme that was started from 2013 onwards,
12:41 which changed teacher hiring,
12:42 uh,
12:43 rules,
12:44 uh,
12:44 and that is in there.
12:46 Uh,
12:46 there is more emphasis on,
12:48 uh,
12:48 on,
12:49 uh,
12:49 after this new hiring scheme on
12:51 getting teachers outside that traditional education degree.
12:55 So there's a lot more teachers with STEM degree,
12:57 lots of teachers below 40 years old,
13:00 and so on.
13:02 OK.
13:05 OK,
13:05 so this is Zena is here.
13:06 She's the one who really should be talking about the design of the,
13:09 of the actual education program,
13:12 but uh here's,
13:13 here's my attempt at it.
13:14 This is a typical
13:15 program.
13:16 We start off with diagnose diagnosing learning.
13:20 Uh,
13:20 it was quite an achievement to test these 200,000 kids within the span of a month.
13:25 And,
13:25 and then we mapped the learning gaps against the foundational learning SLOs
13:30 which were derived from the curriculum.
13:32 Then start sort students across classrooms.
13:34 We have also done another tip,
13:36 uh,
13:36 intervention in Islamabad,
13:38 ICT
13:40 where we keep the students within class.
13:41 So there's a different,
13:42 uh,
13:43 we are,
13:43 we are playing around with different models.
13:45 But here,
13:46 this is a more classic program where students are sorted according to ability
13:49 and put in separate groups,
13:51 uh,
13:52 for part of the day,
13:53 and these are subject-wise sorting.
13:54 So you could be in a different group in,
13:56 uh,
13:57 in English or you could be in a different group in math,
13:59 the same kid.
14:00 And then
14:01 targeted instruction,
14:02 I'll,
14:02 I'll go over it,
14:03 40 act
14:04 activity-based lesson plans.
14:06 Uh,
14:06 so this is about an 8-week program for each remedial subject and level.
14:10 So there are 4 remedial subjects,
14:11 3 levels,
14:13 right?
14:13 So it's 40
14:14 lesson plans for each one of those.
14:17 And then a lot of quizzes to determine if students have mastered the content.
14:21 And then there is ongoing support.
14:23 I'll,
14:23 I'll talk about that with a technology tool,
14:25 and then there's some mentorship which comes into play,
14:28 uh,
14:28 later on.
14:30 OK,
14:30 so this is the details of the program.
14:32 I'm not going to go over it.
14:33 I don't have that much time,
14:35 but in Q&A,
14:35 if people are interested in knowing more about the program,
14:38 uh,
14:38 you know,
14:38 I can talk about it or Zen I can talk about it.
14:41 Uh,
14:42 the,
14:42 the good thing which Juan mentioned is this,
14:44 it was designed in collaboration with the KP Department of Education.
14:47 So both with their teacher training wing
14:49 and with their curriculum wing.
14:50 So it had,
14:51 it had approvals at all different levels,
14:53 and we also worked with the district,
14:54 distinct staff.
14:56 One of the things that we wanted to play around with is,
14:58 is not get additional human resources,
15:00 which is teaching staff.
15:01 Volunteers
15:02 because we were thinking right from the beginning in terms of scalability,
15:05 uh,
15:06 so,
15:06 and
15:07 one of the things that came out in a discussion with the teachers is
15:10 that they did not want to teach outside the classroom.
15:13 So this was done within the regular school hours.
15:15 Uh,
15:16 it was integrated with the,
15:17 with the,
15:17 with the foundational learning SLOs of the regular school curriculum and so on.
15:22 Teachers were provided with all kinds of things,
15:24 uh,
15:24 structured lesson plans.
15:26 For
15:27 learning materials,
15:28 and
15:29 so.
15:30 OK.
15:32 Among
15:34 schools assigned to complete the program,
15:36 some were randomly given access to a specially designed low-cost technology tool
15:42 designed to provide
15:44 assistance,
15:44 reduce administrative burdens.
15:46 The tech tool allows teachers to enter student test scores,
15:49 track student and classroom,
15:51 source students into ability-based learning groups.
15:53 Part of one of the reasons why we thought about this technology
15:56 is we did a very extensive focus group with the teachers beforehand.
16:00 And one of the things was that,
16:02 uh,
16:03 you know,
16:03 tracking students,
16:04 getting quizzes,
16:06 figuring out their ability level was,
16:08 was,
16:09 was one of the constraints,
16:10 a friction that,
16:11 that was stopping them from implementing these programs themselves.
16:15 So that was one of this.
16:17 Teachers could use their
16:18 phones,
16:18 tools on any of these things.
16:20 Let me give you the access.
16:21 So this is,
16:22 this is from the baseline.
16:24 Um,
16:25 this is access to technology.
16:26 This is,
16:27 uh,
16:28 from the baseline survey.
16:31 And you can see that the red
16:33 is,
16:33 is the smartphone.
16:34 So smartphone is,
16:36 is,
16:36 is by and large.
16:38 The technology of choice,
16:40 OK.
16:41 For younger people,
16:43 so look at,
16:44 right,
16:44 I mean,
16:44 65% is,
16:45 is below 40%,
16:46 right?
16:47 The teaching staff.
16:48 So for younger people,
16:49 both males and females,
16:51 the smartphone access is,
16:52 is,
16:53 is roughly the same.
16:55 It,
16:55 it drops.
16:56 For older people,
16:56 we have very few teachers above the age of 50.
16:59 We do have some,
17:00 but,
17:00 uh,
17:01 but you can see
17:03 that.
17:04 The idea was to really develop a smartphone
17:08 application
17:08 because that's what they really use,
17:10 you know,
17:11 you look at tablets,
17:12 computers,
17:13 uh,
17:13 especially,
17:14 you know,
17:14 uh,
17:15 among the older people
17:17 are just not there.
17:18 The basic phone.
17:19 It's only people,
17:20 you know,
17:21 older guys,
17:22 uh,
17:23 close to retirement age who still use basic phones,
17:26 but smartphones seems to be the access of choice.
17:28 I'll just give you some screenshots.
17:30 This is what the tech tool looked like,
17:32 the kinds of things that we can do,
17:34 uh,
17:34 and,
17:35 and it allowed people to sort,
17:37 right?
17:38 So this was the important thing,
17:39 right?
17:40 So if you entered the quiz score,
17:41 they would tell you by subject whether this kid was struggling,
17:45 whether the previous quiz things worked and all that.
17:47 So that was part of the,
17:49 the,
17:50 the idea,
17:50 right,
17:51 uh,
17:52 here.
17:53 So then one of the things that we did was,
17:55 you know,
17:56 well,
17:56 how should we introduce this technology?
17:59 You know,
17:59 prior research has found
18:00 varying returns to technology in the GE panel,
18:03 right?
18:04 Technology is something that we say,
18:06 you know,
18:07 Uh,
18:07 can be a bad buy,
18:09 but generally we need a lot more information on
18:12 the context in which technology works.
18:14 Sometimes it works well,
18:15 like this Kartik paper,
18:17 uh,
18:17 other places,
18:18 you know,
18:18 big Sabine's paper.
18:20 It,
18:21 uh,
18:21 it talks about the conditions under which it works well.
18:23 So it,
18:24 it doesn't work well.
18:26 As a matter of course
18:28 and in our case,
18:29 the interesting thing that we thought was that policies,
18:32 this is generally true for education policies,
18:33 but technology policies
18:35 are almost uniformly used when they are introduced.
18:38 That's a government,
18:39 right?
18:39 Everybody has to do this thing.
18:41 Now we thought that in our school system,
18:43 which is a large system,
18:45 and that's true for many of the low and middle income countries,
18:48 there are considerable differences in student learning levels.
18:51 Schools are spread over rural and urban areas.
18:53 Teacher demographics as you saw in educational attainment vary as well.
18:58 Now we can observe many dimensions of this heterogeneity,
19:00 but I think there are significant unobservable aspects,
19:03 you know,
19:04 of a particular school environment,
19:05 teacher capacity and
19:07 comfort with technology remain.
19:09 So we thought that,
19:10 you know,
19:10 uniformly imposing
19:12 may,
19:13 may,
19:13 may create some issues.
19:15 And so that's why we wanted to really think about the experiment.
19:18 So this is the experiment.
19:21 So we have control.
19:22 So this allows us to test
19:24 TIP program as a whole,
19:25 right?
19:26 So this is,
19:26 this is business as usual.
19:28 And then what we call paper,
19:30 which is
19:30 teachers deliver the program with paper tracking tools in view of the tech tools.
19:34 So the structured pedagogy,
19:35 the training,
19:36 the lesson plans,
19:36 the sorting
19:38 is,
19:38 is all of this is common to all three of them,
19:41 right?
19:41 So these are all randomized,
19:42 but,
19:43 but the,
19:43 the whole tip structure pedagogy program
19:46 in all three treatments is exactly the same.
19:48 But here they were not given the tech tool
19:51 and mandatory teachers were given access to the tech tool and were
19:55 required by the education department to use the tech tool regularly,
19:58 and they were not given the paper resources.
20:00 Optional is teachers that deliver the program with access to the tool
20:04 but were informed that each teacher could decide
20:06 whether to use it or use paper tracking tool.
20:09 So that's really the thing.
20:10 Everybody got 4 days of initial training,
20:12 pretty intense,
20:13 and I was involved in this,
20:15 and then 1 day of refresher training on the program and tech tool.
20:18 We had mentors which were largely tech mentors
20:22 who provided additional support and so on,
20:24 OK.
20:27 The students are tested in English,
20:28 math,
20:29 and Urdu covering content from KG level up to
20:31 grade 4 with questions increasing in degree of difficulty.
20:34 These are the types of questions that we ask
20:37 in English and math.
20:38 This is a more.
20:40 I'd say an extensive test than the user test.
20:42 This is more like the LEAS test,
20:43 so there are like 30 questions in each one of them,
20:45 but they start right from the beginning,
20:47 from counting all the way to word problems,
20:49 for instance,
20:49 in math,
20:50 from matching the picture to the correct word,
20:52 from reading the paragraph,
20:53 and so on.
20:54 So the idea is,
20:55 is that we can get students at any
20:59 level of the foundational learning.
21:02 OK,
21:02 um,
21:03 there's a small technical note here that,
21:05 uh,
21:06 we were,
21:07 we created 4 versions of the test exactly with the same structure,
21:10 uh,
21:11 at,
21:11 uh,
21:12 at the request of,
21:13 uh,
21:13 the government because they were concerned about cheating and all that,
21:17 and,
21:17 uh,
21:18 so we then created a common,
21:19 uh,
21:20 a common,
21:21 uh,
21:22 template
21:23 using IRT,
21:24 this kind of Rosetta Stone methodology.
21:25 It works pretty much
21:27 with everything,
21:27 but this is what we are going to be using.
21:29 Uh,
21:30 the,
21:31 the randomization balance works
21:33 pretty well for all treatments.
21:35 OK,
21:36 so here are the results.
21:38 OK,
21:38 so this is all treatments combined since randomized.
21:40 I can combine all of them.
21:41 This is relative to the control.
21:43 The TIPP program works with a learning gain of about 0.14 standard deviation.
21:49 It's not that high,
21:51 but it's not that low.
21:52 It's over the bar which JPL has set for effective interventions,
21:56 but it,
21:56 it,
21:57 it,
21:57 it,
21:57 it falls smack in the middle of of these
22:00 TIP interventions,
22:01 tipP type interventions done
22:03 around the world.
22:04 I sometimes joke about it,
22:05 you know,
22:06 that's what you need to write
22:07 on my tombstone.
22:08 0.14 seems to be
22:10 where I end up most of the time.
22:12 Uh,
22:13 the right hand figure shows this is the lowest graph,
22:15 and I can do the margins plot with controls also,
22:18 but you can see this is the distribution of baseline learning,
22:22 the,
22:22 the graph,
22:23 the histogram,
22:23 and the background.
22:25 And what you see is
22:26 that,
22:27 well,
22:27 yes,
22:27 it's the TIPP program works for students who are doing pretty well,
22:31 but it works at all levels.
22:33 It,
22:33 it certainly doesn't do worse at any level,
22:35 but it,
22:36 uh,
22:36 the,
22:37 the distributional impacts
22:38 uh seem to be going uh across the ability spectrum,
22:43 OK.
22:44 Uh
22:46 Now,
22:46 here,
22:47 so the big
22:48 punchline of this paper is how we introduce technology matters,
22:51 OK?
22:52 They're not all the same.
22:54 So this is,
22:54 you can see the three bar graphs
22:57 uh with the paper optional and mandated.
23:01 The optional treatment works the best.
23:03 OK,
23:04 uh,
23:04 the point estimate is certainly much greater than paper also,
23:08 but,
23:08 uh,
23:09 the FSTAT on significance is between paper and mandated.
23:13 So we can say for sure
23:15 that the,
23:16 the optional treatment
23:18 definitely works better than mandated,
23:20 and it has the highest.
23:22 So 0.19 standard deviation is pretty good
23:25 in terms of getting these,
23:26 these treatment effects.
23:28 What is important to say is that this treatment effect.
23:31 You know,
23:32 one way of looking at it is saying that
23:33 the treatment effect is not monotonic in tech usage,
23:37 right?
23:37 The take-up rate,
23:38 uh,
23:39 roughly for people who opened accounts and graded quizzes and all
23:42 is about 65% of teachers in optional,
23:45 OK,
23:46 as opposed to the mandated one.
23:49 So what you see is when you go from 0 to 65,
23:52 you see an improvement,
23:53 but when you go from 65
23:55 to over 90% in mandated,
23:58 it falls,
23:59 right?
24:00 So I think that,
24:00 you know,
24:00 we all think that more is better,
24:03 that more tech usage is better than less tech usage,
24:05 but it's not obvious
24:07 here.
24:07 It's not,
24:08 it's actually clear completely that
24:10 that's not the way to go.
24:12 And then when you see the distribution of learning,
24:14 this to me is the most interesting part of this graph
24:17 of this,
24:18 this,
24:18 this study,
24:19 then what you see is that,
24:21 you know,
24:21 I don't know if you people can see it,
24:23 but the,
24:24 the,
24:24 this bright t line is the optional,
24:27 and what you see is,
24:28 uh,
24:28 is,
24:29 is the,
24:30 that at the bottom of the learning distribution,
24:34 the optional.
24:35 Works really well,
24:36 right?
24:36 I mean,
24:36 you can see,
24:37 you know,
24:38 control is that blue line which is right at the end.
24:41 So optional is better than control
24:43 right at the top,
24:45 mandated and paperwork pretty well,
24:47 more or less the same at the top as optional,
24:50 but it's at the bottom end of the distribution,
24:52 which arguably is really why we are doing this,
24:55 right?
24:56 We are not doing these programs.
24:58 You know,
24:58 certainly we want to help all kids,
25:00 but for kids at the top of the distribution,
25:03 you know,
25:03 even in control are doing pretty well.
25:06 It's really at the bottom
25:08 part
25:09 that optional really comes through,
25:13 and you can see that paper and mandated both
25:16 really helps students only at the top.
25:19 So I think that's really
25:20 what this optional allows us,
25:22 uh,
25:23 is,
25:23 is not only gives us
25:25 on the average
25:26 higher learning,
25:28 uh,
25:28 gains,
25:29 uh,
25:30 than mandated for sure and,
25:32 and,
25:32 and,
25:33 and paper,
25:34 uh,
25:34 but at the bottom as well,
25:36 if I do a margins plot,
25:38 I don't know,
25:38 this doesn't look like a very nice graph,
25:40 but this is controlling for covariates and all that in the heterogeneity,
25:44 uh,
25:45 with the,
25:45 with the baseline scores.
25:47 You can see that the optional is,
25:49 is,
25:50 is,
25:50 is doing better as well,
25:52 OK?
25:54 Now,
25:54 why optional matters?
25:56 Hey,
25:56 listen,
25:56 this is work in progress,
25:57 so,
25:58 you know,
25:58 I'm planning to come out with the final version of this paper somewhere in,
26:01 in,
26:02 in,
26:02 in,
26:02 in,
26:03 at the end of the summer.
26:04 I'm presenting it in the UK,
26:05 so by then,
26:06 hopefully everything will be there.
26:08 But these are some possible reasons why optional matters.
26:12 So as you must be thinking,
26:13 the most obvious reason is some sort of self-selection.
26:16 Which is self teaches self-selecting technology tools based on
26:19 private information such as prior knowledge of the schools,
26:22 the work environment,
26:23 and inherent confidence in technology,
26:25 etc.
26:26 So people who are comfortable with technology use it,
26:28 people who don't.
26:29 That's a self-selection argument.
26:30 But there are a couple of other arguments which I think are,
26:32 are also important to keep into,
26:34 keep into,
26:35 keep in mind.
26:36 I was talking to Vij Banerje,
26:37 and he thought the second argument was really,
26:39 uh,
26:40 quite interesting.
26:41 Which is,
26:42 listen,
26:42 the people in KP and in Pakistan,
26:44 the teachers have been reformed like hell,
26:46 right?
26:46 I mean,
26:47 they have,
26:47 every time
26:48 there's a new mandate that is imposed on them,
26:50 they used to be
26:51 governed by mandates where nobody asks them what needs to happen
26:55 and somebody tells them,
26:56 hey,
26:56 there's a new policy,
26:57 you've got to implement it.
26:59 And people don't like it.
27:00 We have talked to teachers about it.
27:02 They don't like it,
27:03 and I think that they feel that it,
27:04 it,
27:05 it is,
27:05 it,
27:06 it feels it leads to lower empowerment.
27:08 So this is not a self-selection argument.
27:10 This means all teachers
27:11 in the mandated pool would feel a little bit more discouraged.
27:15 Optional gives them a sense of ownership over the reform,
27:18 and I think that's an interesting idea too.
27:21 The second part,
27:22 which I think is very interesting,
27:24 is that when we gave teachers access to
27:26 both the technology tool and paper resources,
27:29 they inferred also that this,
27:31 they,
27:31 they,
27:31 that they had flexibility.
27:33 They could use technology when they felt like it,
27:35 they could use paper when they felt like it,
27:37 and they could come up with their own kind of optimal mix and match.
27:41 So it leads to a lot more flexibility.
27:44 We are working with our online survey to really kind of uh
27:47 see if we can separate the two and test them.
27:50 Uh,
27:51 but it's,
27:51 it's hard to get kind of more precise estimates,
27:53 OK.
27:54 But anyway,
27:55 this is something we are working on
27:56 to be really able to parse one of these,
27:58 but,
27:58 but think about some of these things
28:00 as,
28:00 as,
28:01 and we can talk about them in Q&A.
28:03 OK.
28:04 Heterogeneity,
28:05 in our pre-analysis plan,
28:06 the registered pre-analysis plan,
28:08 we thought we would look at heterogeneity for sure,
28:10 so it's not like I'm fishing for anything.
28:12 These are things that we said that we would look for.
28:15 And this is baseline scores,
28:17 OK?
28:17 And this is a multiple regression,
28:19 just,
28:19 just descriptive saying,
28:21 how do scores respond to,
28:24 to heterogeneity.
28:26 Uh,
28:26 very clearly,
28:27 males perform,
28:29 this is baseline score,
28:30 males perform worse.
28:31 OK,
28:31 this is now coming out in leaps even more.
28:34 Uh,
28:34 which we just finished two rounds of testing in 2024,
28:38 which is in Pakistan now.
28:40 Males are performing worse.
28:41 In this data,
28:42 males are performing.
28:43 This is the average score.
28:45 Males are,
28:45 are way behind in English and Urdu.
28:48 Math,
28:49 they're a little bit ahead,
28:51 but at least it's not so.
28:52 So very clearly this,
28:53 this stuff is now coming out consistently over lots of kids testing.
28:58 Larger schools on average do better,
29:00 urban schools do better.
29:02 Uh,
29:03 nothing else really comes through
29:05 except when you come to like this grit,
29:07 right above this,
29:08 you know,
29:09 teachers who had above median grit score,
29:11 this is,
29:11 this is,
29:12 this is before the intervention,
29:13 right?
29:14 Their kids were doing really better.
29:16 So I'll,
29:16 I'll,
29:16 I'll talk about that a little bit.
29:18 So we,
29:19 we analyzed the heterogeneous impact of the
29:21 TIPP program based on the pre-registered plan.
29:24 Nothing really seems to matter,
29:26 you know,
29:26 this big five personality.
29:27 I've done it a number of times.
29:29 I really don't find much action in it,
29:31 uh,
29:31 but other people have used it.
29:33 This is validated for Pakistan,
29:35 uh,
29:35 and they think it is.
29:36 Some of the things that you think about might matter,
29:40 uh,
29:40 age,
29:40 for instance,
29:41 you know,
29:42 it doesn't have a big heterogeneity.
29:45 Element with uh with any of the treatment.
29:47 This is hetero
29:48 looking at these variables and interacting them with all the treatments.
29:51 That's what I'm doing,
29:52 right?
29:53 I don't see anything much.
29:54 Part of it is these things are very correlated with each other,
29:57 right?
29:57 Techability and age is very correlated
30:00 with each other
30:01 and school characteristics.
30:03 Once you do
30:04 the,
30:05 the treatment of fact interaction,
30:07 uh,
30:08 none of these things mattered.
30:09 There is some very interesting heterogeneity with gender.
30:14 Uh,
30:15 which is
30:16 that
30:18 Male schools benefited more from the mandated
30:21 treatment for math than female schools,
30:23 so math operates slightly differently than than English and Urdu.
30:28 So I'm,
30:28 I'm,
30:29 I'm saying this because I think that if you want to take education seriously,
30:33 education research,
30:34 and not just worry about causal effects and econ,
30:37 I think there's something going on with math versus language.
30:40 On male versus females
30:41 that we really need to dig into
30:43 a lot more,
30:45 uh,
30:45 and the mandated treatment,
30:47 the reason why
30:49 on the average,
30:50 uh,
30:50 test score,
30:51 the mandate does well,
30:53 it's,
30:54 it's largely coming from math,
30:56 OK?
30:56 So that's something that I need to dig more into.
30:59 Uh,
31:00 the interesting,
31:01 within male schools,
31:03 optional paper and uh mandated work kind of equally well,
31:07 while within
31:09 within female schools
31:11 optional performed better than the other treatments.
31:13 So part of the reason on average optional is better
31:17 than the other two treatments
31:19 is largely coming from uh
31:22 from females.
31:23 So we need to think a little bit about it.
31:25 We have some hypothesis,
31:26 conjectures
31:27 as to in terms of collaboration,
31:29 in terms of people learning with each other,
31:31 uh,
31:32 from each other.
31:32 So that's,
31:33 uh,
31:34 that's something that uh.
31:37 That we can,
31:37 we can think about
31:39 now.
31:39 I,
31:39 now I want to say on this male versus female,
31:42 uh,
31:43 Maya,
31:43 we should put this in the slide right in the beginning,
31:46 uh,
31:46 these schools are gender segregated.
31:48 So we cannot separate a boy girl issue
31:51 from a male female teacher issue.
31:53 Female teachers teach.
31:56 Girls
31:57 and male teachers teach
31:59 boys,
31:59 OK?
32:00 So there,
32:00 it's,
32:00 it's a pair,
32:01 right?
32:02 So when I say female schools and,
32:03 and male schools,
32:05 I'm not saying male versus female teacher,
32:07 it's the,
32:07 it's the dyad
32:09 together
32:09 that I am,
32:10 I am talking about.
32:12 Students taught by grittier teachers benefit more,
32:15 uh,
32:16 you know,
32:16 for sure,
32:18 uh,
32:19 well.
32:20 In in control,
32:21 the benefit a lot more.
32:23 It's kind of stunning that in control,
32:26 uh,
32:26 there's a huge difference between teachers with more grit,
32:29 uh,
32:30 and,
32:31 uh.
32:34 So these are,
32:34 this is,
32:35 I,
32:35 I don't know what you want to think about it,
32:36 OK?
32:37 There's a very interesting work.
32:38 Suley Allen has done a randomized study.
32:40 We should really do something proper on it,
32:43 uh,
32:43 but,
32:44 uh,
32:44 this is,
32:44 these are the great questions,
32:45 right?
32:46 This is,
32:46 this is also,
32:47 we spend a fair amount of time,
32:49 uh,
32:49 piloting this,
32:50 but,
32:51 uh,
32:51 these are types of questions that people use.
32:53 I'm not a,
32:54 I'm not a,
32:54 a specialist in this,
32:56 but,
32:56 uh,
32:56 one of our researchers actually really Agu,
32:58 who was very obsessed with it.
33:00 Uh,
33:00 who,
33:00 who played around with it quite a bit and he's working with the World Bank now,
33:04 I think,
33:05 uh,
33:06 so.
33:08 So what you do is
33:09 in control group,
33:11 student test scores
33:12 are higher
33:13 in the endline test for above median,
33:15 very big difference
33:16 compared to below median grade teachers.
33:19 And the,
33:19 the,
33:20 the,
33:22 in the treatment group,
33:23 uh,
33:23 the grade premium vanishes
33:25 completely,
33:26 right?
33:26 So basically,
33:27 what it looks like the treatment is doing
33:29 is pulling up
33:31 the below grade teachers
33:32 up to the same level as the above grade teachers in the control,
33:36 OK?
33:37 Uh,
33:37 and I think this might be effective.
33:40 So one of the things is,
33:41 you know,
33:41 when you think about grit,
33:42 Angela Beckworth at Penn has done a lot of work on it.
33:44 She's a professor of psychology,
33:46 and she says grit,
33:48 grittier people
33:49 are,
33:50 are,
33:50 are work better in these environments
33:53 in which there is a lot of support around them and so on.
33:56 So that might be something more to think about.
33:58 This is again,
33:59 I think,
33:59 preliminary,
34:00 but we,
34:01 we can think more about it.
34:03 OK.
34:04 Choice of technology.
34:05 The problem for me is you ask about what works in optional.
34:09 We do have data on which teachers use technology and which don't.
34:14 But the problem is,
34:15 it's endogenous choice.
34:16 This kind of OLS correlation on teachers that use
34:20 on takeoff,
34:20 I can't use it as an instrument because all
34:23 of them have a direct effect on landline outcomes,
34:25 right?
34:25 So,
34:26 it's a really difficult problem on this endogenous.
34:29 Choice of technology and OLS will take us only this far.
34:33 I don't think it can really help us too much.
34:35 This is what I did.
34:36 So this is the second part,
34:37 OK,
34:37 on this choice of technology.
34:39 I want to give you some detail on this choice of technology.
34:42 So this is,
34:42 this is what happens.
34:43 This is really curious,
34:45 OK?
34:45 And I'm going to Pakistan in uh,
34:46 in next week,
34:48 and I'm going to visit these schools again to get better at it.
34:51 I mean,
34:51 there was an,
34:52 you know,
34:53 it almost looks like it has a
34:54 school-level component as opposed to individual choice,
34:58 right?
34:58 I mean,
34:59 what you see is
35:01 There are some schools in which no teacher uses tech
35:05 in the optional,
35:06 and there are some schools in which all teachers use tech.
35:10 And then there are schools in which some teachers use tech and others don't.
35:15 And I think that this means that our original thinking in terms of
35:19 thinking of school or tech choice as an individual
35:23 level variable and thinking of these econ type,
35:26 type,
35:27 uh,
35:27 reasoning,
35:28 right,
35:28 which I said like self-selection,
35:29 that's an individual.
35:30 Reasoning,
35:31 it seems to be working in some places in the sense of about 25% of the schools,
35:36 but in others,
35:37 it looks like there is some coordination that
35:39 is happening or some decision that is happening.
35:41 By the way,
35:42 uh,
35:43 in KP and in Punjab,
35:44 also in Pakistan primary school,
35:46 the head teacher is really not an empowered,
35:49 uh,
35:49 principal-like character.
35:51 It,
35:51 it's just one of these,
35:52 uh,
35:52 people who are,
35:53 uh,
35:55 just a teacher with a,
35:55 with a designation.
35:57 So the head teacher doesn't really come into play that much here.
36:00 And this is what I do.
36:01 It's not perfect,
36:02 but what I do is
36:04 I,
36:04 I look at the fact of
36:06 whether a teacher has a check account and I put in school fixed effects.
36:09 When I put in school fixed effects,
36:11 it basically,
36:12 you know,
36:13 schools in which
36:14 everybody takes technology or school in which nobody takes technology,
36:17 they drop out,
36:17 right,
36:17 because they have a correlated with the school fixed effect.
36:20 So I take away all the,
36:22 the school level variation,
36:24 and I'm only left with schools
36:26 in which some people choose technology
36:28 and others don't.
36:29 And there I see,
36:31 you know,
36:31 some math fact,
36:32 right?
36:33 There's a Urdu effect,
36:34 but it's not significant.
36:35 English is pretty much zero,
36:37 so I do see.
36:38 Uh,
36:39 some math effect,
36:39 by the way,
36:40 I use
36:41 separately for
36:42 English,
36:42 math,
36:42 and Urdu
36:43 whenever I'm using teacher level,
36:45 uh,
36:46 variables
36:47 because,
36:48 uh,
36:48 each student is matched with different teachers for different subjects,
36:52 OK.
36:53 And I can show you the table.
36:55 Uh,
36:55 this is very possibly interesting for,
36:57 for the economists in the crowd.
36:59 There's a whole instrumental variable strategy coming from,
37:02 uh,
37:02 criminology and sentencing,
37:04 which is called the judge IV.
37:05 Which is,
37:06 it looks like that certain judges have have traits that
37:11 make them sentenced more,
37:12 which is uncorrelated.
37:14 Our mentors were very different types,
37:16 right?
37:17 Mentors were
37:18 not randomly but kind of arbitrarily assigned to these schools.
37:21 Male mentors went to male schools,
37:23 female to female schools.
37:24 It looked like.
37:26 You know,
37:26 which mentor went to which school
37:29 had a big effect in choice of technology,
37:31 right,
37:32 as uh compared with the with the
37:35 kind of mentor identity.
37:37 OK,
37:38 and each mentor went to about 8 to 10 schools
37:41 on average.
37:42 So the idea is you can use mentor identity arguably as an exogenous
37:48 effect
37:49 on variation in in adoption of technology,
37:52 and you can use it as an instrument.
37:54 That's what people do in this kind of judge IV.
37:56 So we are planning to do this.
37:58 So maybe the idea is that you are random,
38:01 you are essentially as if randomly assigned to a mentor,
38:04 you just get randomly assigned to a good mentor,
38:07 and then you're more likely to choose technology as
38:09 opposed to a more or less convincing mentor,
38:11 and you are less likely to choose technology
38:13 and so on.
38:14 And that gives,
38:15 gives us some exogenous variation and adoption of technology,
38:19 and then we can see the effect.
38:20 So that's our other thing that we want to do.
38:23 Finally,
38:24 we want to do one more thing.
38:26 This is all happening right now.
38:27 I have some results on this,
38:28 but I don't want to present them right now.
38:30 What I want to do is to use the machine learning techniques
38:34 to predict
38:35 what types of teacher and school characteristics allow,
38:39 uh,
38:40 predict choice of technology in the optional group.
38:43 And use the optional group as training data
38:46 and then try to simulate the counterfactual on based on this training data,
38:51 what schools in the mandated group
38:53 would have used technology based on the training data in the optional group
38:58 and then we can see
38:59 what the results are
39:01 of of.
39:04 Of uh
39:06 Uh,
39:07 the simulated choice in the mandated group is,
39:09 so we are well on our way.
39:10 We are trying to do it.
39:12 The problem with that is a little bit less
39:15 precision
39:16 than one would like because of this school level choice.
39:20 There are lots of,
39:20 you know,
39:21 there's variation in teacher characteristics within the school,
39:24 but then when everybody chooses
39:26 technology or nobody chooses technology that weakens
39:28 kind of the power of individual teacher characteristics
39:31 in terms of choice.
39:32 So we are kind of wrestling with that part.
39:34 OK.
39:36 So in conclusion,
39:37 The TIPP program,
39:38 which combined targeted instruction and structured pedagogy approach,
39:42 works across different forms of implementation.
39:45 The program yielded the best results by
39:47 giving teachers the option to use the technology.
39:50 The option treatment arm more than others
39:52 also helps students across the ability spectrum.
39:55 Mandated in technology with significant teacher heterogeneity in
39:58 the system proves to be less effective.
40:01 Bottom line,
40:01 with technology,
40:02 more
40:03 may not necessarily be better.
40:06 Hey,
40:06 I'll take one more minute to say something else which is very
40:09 preliminary and speculative.
40:10 It's not part of the pre-analysis plan,
40:13 so I don't want to present it,
40:14 but which is I also do stuff like this,
40:16 OK?
40:17 This is very interesting,
40:18 and I'm the reason I'm saying this to the World Bank crowd
40:20 is this is something that we might want to do some more studies on,
40:23 do more experiments with,
40:25 uh,
40:25 explicitly.
40:26 This is not something we had explicitly in mind,
40:28 but I started playing around with it.
40:30 So here's what it is.
40:32 So this is baseline within
40:34 class.
40:35 Are learning variation.
40:36 Look at this difference
40:38 between
40:39 the treatment.
40:40 And the control,
40:43 the control when the student learning variation baseline is high,
40:46 the control test scores really collapse,
40:48 right?
40:49 I mean,
40:49 they,
40:49 they just go,
40:50 they are the worst,
40:50 right,
40:51 in,
40:51 in school
40:52 with high learning variation,
40:54 and we can control for uh mean learning baseline levels in a regression margin plot,
40:58 and we can still see this,
40:59 uh,
41:00 here's what it is,
41:01 right?
41:01 In the margins plot,
41:02 this is controlling for it.
41:04 It's less precise in margins plot,
41:05 but it's,
41:06 uh,
41:06 it's there.
41:08 And then when we,
41:09 when we look at all treatments,
41:10 so optional really performs better than control
41:14 even when the baseline learning level variation is very high,
41:17 OK?
41:18 So I think this is a very interesting.
41:20 I want to do this.
41:21 I really want to do some more studies
41:23 where I explicitly target the baseline learning variation.
41:26 OK,
41:26 I'm done.
41:31 Thanks so much,
41:31 Tahir.
41:32 This is really,
41:32 really fascinating,
41:33 and thanks for presenting even that,
41:35 that latest,
41:37 the latest results.
41:38 We have time for questions,
41:40 um,
41:40 and so I think you can put them in the chat or
41:43 you can just raise your hand the old fashioned way and,
41:45 and,
41:46 and ask them,
41:47 um,
41:48 but the,
41:48 the floor is open
41:49 for Q&A.
41:58 Like there are some in the chat already.
42:01 I believe.
42:14 Yeah,
42:14 I think some of them are just uh
42:17 sharing
42:19 comments and then clarification questions,
42:21 but if anyone
42:22 um has a question,
42:24 also Amer has asked
42:26 um what,
42:27 how is the,
42:28 what's been the government response to these results and are they planning to,
42:32 I guess,
42:32 adjust
42:33 after this?
42:34 Oh,
42:34 Amer,
42:34 you can ask if you can.
42:39 Hi,
42:40 sure.
42:40 Can you guys hear me?
42:41 Yes.
42:42 Hi,
42:43 thank you so much.
42:43 I um just wanted to understand a little bit about the
42:46 dynamics of uh how the conversation with the government has gone
42:49 and how they're planning on
42:51 using this information either in the ongoing operations or in
42:54 any subsequent conversation that the bank is having with the,
42:57 the KP government.
42:58 Thank you.
42:59 OK,
42:59 this is great,
43:00 right?
43:00 So,
43:01 very interesting,
43:02 right?
43:02 I mean,
43:03 my sense is that,
43:04 uh,
43:05 the KP government was kind of,
43:06 uh,
43:08 was very supportive throughout this whole process.
43:11 Uh,
43:12 we spent a lot of time,
43:13 Malia in particular,
43:14 spending,
43:14 you know,
43:14 I,
43:15 I am now a kind of a regular fixture in Peshawar
43:18 also,
43:18 uh,
43:20 so they were quite supportive.
43:21 In general,
43:22 uh,
43:22 for sure,
43:23 I think that moving forward,
43:25 uh,
43:27 you know,
43:28 secretaries come and go,
43:29 uh,
43:30 but,
43:31 uh,
43:31 I think there was a lot of interest
43:33 in pushing this in,
43:35 in what is called the backward districts.
43:37 So they were very worried about learning gaps between
43:40 the more settled districts
43:42 and backward districts.
43:43 So there was,
43:44 uh,
43:44 there was some discussion.
43:47 Uh,
43:48 on doing it.
43:49 We're having a discussion with them right now
43:52 also in trying to,
43:54 uh,
43:55 Uh,
43:56 regularize some sort of a boot camp,
43:59 uh,
44:00 after the summer.
44:02 Uh,
44:02 we have a project
44:04 where we are trying to now think about persistence and diffusion,
44:07 right?
44:07 I mean,
44:08 does this innovation persist?
44:10 And there are lots of very interesting elements,
44:13 right?
44:13 I mean,
44:14 we trained these teachers pretty extensively.
44:16 We are,
44:16 we are interviewing them right now,
44:18 uh,
44:18 qualitatively,
44:20 uh,
44:20 as we speak,
44:22 in terms of,
44:22 you know,
44:23 2 years later,
44:23 what do they think about TIP and all that.
44:26 And the interesting question is,
44:28 is,
44:28 and,
44:29 and,
44:29 and the government is quite interested in this.
44:31 The interesting question is,
44:32 you know,
44:32 with these trainings and the program that happened with existing teachers,
44:36 you know,
44:36 the interesting thing is Pakistan just works by notification,
44:39 right?
44:39 I mean,
44:39 the program is running,
44:41 everybody's great.
44:42 That notification ends,
44:43 and,
44:43 you know,
44:43 it's back to business as usual.
44:45 Uh,
44:46 and,
44:46 and the effects of these programs do not last.
44:48 So we are very interested in looking at persistence as well.
44:52 Uh,
44:52 and you know,
44:53 we have two hypotheses.
44:54 One is that,
44:55 you know,
44:55 persistence does not happen because there's decay of capability.
44:58 You know,
44:59 uh,
45:00 the KP teachers are like,
45:01 uh,
45:01 like my econometric students,
45:03 right?
45:03 I mean,
45:03 two years later,
45:04 nobody,
45:04 nobody remembers anything,
45:05 uh,
45:06 from what I teach them,
45:07 uh,
45:08 and,
45:09 uh,
45:09 but at the same time,
45:11 the other part is these people say,
45:12 yeah,
45:12 they want to do it,
45:14 but they end up in,
45:15 in these environments which certainly are not encouraging,
45:17 sometimes even hostile to innovation.
45:19 So we want to play around with this idea of,
45:22 you know,
45:22 how,
45:23 how,
45:23 how do we allow them to,
45:25 to,
45:26 for these innovations to persist and continue,
45:28 and the government is quite interested.
45:30 We are in conversation with them of having a boot camp,
45:33 uh,
45:34 and,
45:34 and trying to see,
45:36 you know,
45:36 whether this can be
45:38 initiated.
45:38 By the way,
45:39 PMIU in Punjab,
45:40 I've had a couple of conversations.
45:42 I'm going to give a talk there,
45:43 uh,
45:44 uh,
45:44 next month or later this month,
45:46 uh,
45:47 on this idea.
45:47 They've tried their own version of a boot camp.
45:50 Uh,
45:50 and I think that,
45:51 uh,
45:52 they're very keen on finding results here
45:54 and on this other study that I'm doing
45:57 with one on,
45:58 uh,
45:58 summer learning losses.
45:59 So,
45:59 so there is,
46:00 there is a lot of interest
46:02 in there,
46:02 uh,
46:03 the backward districts are very hard to implement,
46:05 right?
46:05 I mean,
46:05 it's,
46:06 uh,
46:06 it's,
46:06 it's,
46:06 it's,
46:07 there's a lot of
46:08 issues there,
46:08 but they're keen,
46:09 particularly on those,
46:11 uh,
46:11 parts.
46:14 Thanks.
46:15 uh,
46:15 why don't we go to Mike to kind?
46:17 Mike has a few questions for you.
46:34 Oh,
46:35 he's asking if someone can ask,
46:37 ask for him.
46:38 Oh,
46:38 I,
46:38 I can read the question.
46:39 What sort of investments,
46:41 including complementary investments in the enabling infrastructure,
46:43 however you choose to define that term,
46:44 need to be made.
46:46 If you are to do something like this at a larger scale,
46:49 so we talked about,
46:50 you know,
46:50 a couple of things that need to happen,
46:52 right?
46:53 A lot of Zene is here.
46:55 I mean,
46:55 a lot of the material has been constructed.
46:57 There is,
46:58 uh,
46:59 the videos,
47:00 lesson plans,
47:00 the activity books,
47:01 the guidebooks.
47:03 Uh,
47:04 we,
47:04 we in Islamabad,
47:04 we're getting a lot of positive feedback on the workbooks that are used because
47:08 the experiment was really around of unbundling tip and looking at specifics of it.
47:13 So we're getting a lot of feedback.
47:14 So those are already there.
47:16 And I think they're,
47:16 they're pretty usable
47:18 at scale.
47:20 Uh,
47:20 the tech tool,
47:21 you know,
47:21 is,
47:22 is in existence.
47:23 We need to just give,
47:24 you know,
47:24 access to the tech tool.
47:26 In fact,
47:26 in,
47:27 in moving forward,
47:27 Ahmed,
47:28 one of the things
47:29 we are thinking about is,
47:30 is,
47:31 is giving these people who are already trained by
47:33 tech and are no longer part of the program,
47:36 giving them access to all this material,
47:38 uh,
47:38 through the tech tool.
47:40 So that part,
47:41 I think,
47:41 can be done.
47:43 We are playing around with this training part,
47:45 right?
47:45 So one of the things is training at scale is,
47:48 is,
47:49 is,
47:49 is really
47:50 An issue,
47:51 right?
47:51 I mean,
47:51 training seems to be just a
47:54 uh a real bottleneck
47:55 uh in doing things at scale.
47:58 So one of the things is,
47:59 you know,
48:00 we do have it.
48:01 We wanted to try this randomization
48:04 of different training models.
48:05 Uh,
48:06 Zab had developed this,
48:08 this kind of,
48:08 you know,
48:09 this online training stuff.
48:11 We already have that,
48:12 so that material has been developed.
48:14 Other people are doing it,
48:15 but that hasn't properly been piloted.
48:17 It would be very nice to see.
48:19 If kind of more digital training models
48:23 are as effective because those can be scaled.
48:25 So we have talked about it.
48:26 The federal government is very interested in those kinds of things,
48:29 is if we can reduce these,
48:31 these training costs.
48:32 The third is assessment.
48:34 Uh,
48:34 large scale assessment,
48:35 right?
48:36 In Pakistan,
48:37 I mean,
48:37 you know,
48:38 some of my best friends are Pakistanis,
48:39 but large scale assessments in Pakistan are quite suspect,
48:42 uh,
48:43 in terms of protocols,
48:45 in terms of all kinds of things.
48:47 Uh,
48:47 we are,
48:47 we are working with them,
48:48 you know,
48:48 we are doing this TIP ICT.
48:51 Uh,
48:51 scale up,
48:52 uh,
48:52 I think maybe with a bank,
48:54 uh,
48:54 surface,
48:54 uh,
48:55 in Balochistan and all that,
48:57 and I think these assessment protocols
48:59 on how to do it.
49:00 By the way,
49:00 we had a huge success
49:03 in KP Howard,
49:04 which is
49:05 that.
49:07 You know,
49:07 we were kind of,
49:08 everybody's worried,
49:09 right,
49:09 starting from Kartik onwards,
49:11 the difference between performance on our tests,
49:13 this leap style test,
49:15 versus the 5th grade exam,
49:17 which seems to dominate
49:18 thinking around 5th grade,
49:20 right,
49:20 for all teachers.
49:21 They're really concerned about covering the material
49:23 and they were kind of worried about tip taking away from them.
49:26 Uh,
49:26 Zenna had a conversation.
49:27 Zenna and Malia had a conversation with the,
49:30 with the KP examination Authority.
49:33 And they thought that,
49:34 you know,
49:35 if we wanted,
49:35 we could work with them
49:38 to modify their exams,
49:40 which could focus
49:41 more on foundational learning also,
49:44 which I thought was a pretty big thing
49:46 uh for us to do.
49:48 So I think those are the kinds of things
49:50 that I would really,
49:51 you know,
49:52 take into account when,
49:54 uh,
49:55 when thinking about,
49:56 uh,
49:57 thinking about scale.
49:58 Can,
49:59 can I add to that?
50:00 Yeah,
50:00 please go ahead.
50:00 Yeah,
50:01 can I add a quick piece of that?
50:02 I think though,
50:03 if,
50:03 if we go back,
50:04 and there,
50:04 there's a couple of questions on hindsight,
50:06 uh,
50:06 in,
50:07 in the chat also,
50:07 if we go back and look at what it
50:09 is that actually made implementation of TIPS successful,
50:13 I would say there were like two key things that I would point to.
50:17 Number one,
50:17 I mean,
50:17 yes,
50:18 of course,
50:18 you had COVID and you had that window of opportunity
50:20 where policymakers were willing to let you into schools,
50:22 whereas otherwise they weren't.
50:23 So that was definitely the end.
50:25 But beyond that point.
50:27 Uh,
50:27 what encouraged like teachers to do takeout,
50:29 right?
50:30 One,
50:30 as Tahai said earlier,
50:31 we replaced tip
50:33 with the regular curriculum.
50:34 It wasn't an add-on.
50:36 So we actually took out regular textbooks
50:38 and we replaced them with tipP curriculum,
50:40 which was fantastic for teachers because regular textbooks are
50:44 so far beyond the capability of not just students,
50:47 but also teachers.
50:48 Like the curriculum is so ridiculous.
50:49 Ridiculously ambitious
50:51 that here we had a program where we replace
50:53 those textbooks with literally teaching at the right level,
50:57 right?
50:57 Like this is,
50:58 this is where your kids are at,
50:58 this is what you need to do next.
51:00 And so they love that.
51:02 Number 12,
51:03 we made lesson plans such
51:05 that we really reduced the workload on teachers.
51:09 We made super easy,
51:10 fairly scripted lesson plans.
51:12 Trained them on that,
51:13 and it just
51:13 made the lift for teachers so much easier,
51:16 that,
51:16 that,
51:17 like the one feedback we've continuously gotten qualitatively is,
51:20 you know,
51:20 the lesson plans were super easy to use.
51:23 And so,
51:24 in,
51:24 in scaling any such thing,
51:26 those two pieces would have to be looked at very carefully,
51:28 one,
51:29 to create space for such a program,
51:30 and two,
51:31 to create like a low lift for teachers,
51:33 so it's easy for them to implement.
51:37 OK,
51:37 let me respond to a couple of questions.
51:39 First of all,
51:39 uh,
51:39 the variation,
51:41 most classroom,
51:42 most schools in 4th grade have only one section.
51:45 I'd say over 90%.
51:47 So this is really within class variation for most places,
51:50 right?
51:50 So
51:51 in some places you have multiple sections of 4th grade,
51:54 we are combining the two,
51:55 but most of it is within,
51:57 I think in terms of surprises or what we should have done,
52:00 I think the school choice,
52:02 uh,
52:02 tech choice,
52:03 this organizational
52:04 component
52:05 versus the individual component,
52:07 I wasn't expecting that.
52:08 I mean,
52:09 you know,
52:09 our thinking as economists was mainly thinking of
52:11 the teacher as the unit of analysis.
52:13 And thinking about their choice,
52:15 uh,
52:16 material.
52:17 So what is going on?
52:18 I'm,
52:18 I'm becoming quite interested.
52:20 In kind of the school as an organizational unit
52:23 and what happens to decision making within the school,
52:25 what happens to behavioral stuff within the school.
52:28 People are working on it.
52:29 So that's another thing that,
52:31 you know,
52:31 I would,
52:31 I,
52:32 I,
52:32 I am,
52:32 I,
52:32 I am going to do some forensic focus groups,
52:35 uh,
52:36 this May,
52:37 uh,
52:37 in,
52:38 in,
52:38 in a couple of weeks,
52:39 uh,
52:39 before schools close in KP.
52:41 So I think this is a question that
52:43 I want to get a,
52:43 uh,
52:44 get a,
52:44 uh,
52:45 uh,
52:46 better,
52:47 uh,
52:48 better sense of that.
52:49 Um
52:52 Martin,
52:52 do you have a question?
52:55 What is,
52:56 uh,
52:56 yes,
52:57 uh,
52:57 thank you so much.
52:58 Uh,
52:58 first,
52:58 thank you,
52:59 because I think this is uh really
53:01 operationally relevant research and it will be useful for many of our projects.
53:04 So thank you.
53:05 My question,
53:06 and I apologize if I missed it,
53:07 but I was wondering if
53:09 you have any measure of how the intensity of the implementation or,
53:12 or the length.
53:13 of the implementation
53:14 also affects the results.
53:16 And if not,
53:17 well then perhaps you have any plans to measure the effects after a while.
53:21 I'm asking because with,
53:22 especially with technology,
53:23 some
53:24 programs,
53:25 uh,
53:25 tend to vanish their effect after a while,
53:27 but some others have the opposite.
53:28 The teachers,
53:29 uh,
53:30 get used to the methodology and
53:31 the effects tend to increase.
53:32 So I,
53:33 I was just wondering,
53:35 uh,
53:35 that's on the plans.
53:35 Thank you.
53:36 Yeah,
53:36 that,
53:36 that's a great question,
53:37 Martin.
53:37 So,
53:38 as I said,
53:39 we are very interested
53:41 in measuring the persistence.
53:43 Right,
53:43 not just for fade out,
53:45 but also behaviorally,
53:46 precisely the the comments that you raised,
53:49 and we want to see whether it's the,
53:50 the enabling environment that is the binding constraint
53:53 afterwards
53:54 or whether it is kind of individual intrinsic teacher capability.
53:57 So yeah,
53:58 that's something that we are,
53:59 uh,
54:00 that we are,
54:00 uh,
54:01 uh,
54:03 definitely very interested.
54:05 The intensity,
54:06 you know,
54:06 the problem is
54:08 the incentive,
54:08 intensity use is endogenous.
54:11 Right?
54:11 I mean,
54:11 you know,
54:12 you can't really use that as an explanation because certain
54:14 types of teachers will use it more intensively and,
54:17 and other types of teachers will use it less intensively.
54:19 We do have some measures.
54:21 The tech tool,
54:22 uh,
54:23 does record stuff,
54:24 right?
54:24 So we do have,
54:25 for instance,
54:26 the most,
54:27 most data that we have is on how many teachers,
54:31 uh,
54:31 utilize the,
54:32 the quiz grading,
54:34 uh,
54:34 capability.
54:35 So we have that.
54:36 So we can take a look at that.
54:38 I was actually thinking about.
54:40 Doing a comparison between,
54:41 uh,
54:42 between,
54:43 uh,
54:43 the optional group,
54:44 the people who do use technology
54:46 on the intensive margin versus the mandated group,
54:49 right?
54:49 Is it that people are more motivated to use it
54:52 or,
54:53 or not,
54:54 but to use intensity as a right-hand side variable to explain headline scores is,
54:58 is complicated,
54:59 right,
54:59 in terms of the endogeneity.
55:01 Uh,
55:02 just very quickly,
55:03 the rollout was at the same time for all the schools,
55:06 right?
55:07 I think within,
55:07 within,
55:08 within a few weeks.
55:10 So then,
55:10 you know,
55:11 listen,
55:11 it's a,
55:11 it's a,
55:12 it,
55:12 it,
55:13 it takes time.
55:14 Uh,
55:14 Amir,
55:15 on the,
55:16 on the,
55:18 the
55:20 issue of
55:21 the 40 day,
55:22 uh,
55:23 for that time period,
55:24 the curriculum was not done,
55:25 OK?
55:26 So this is like 3 hours a day roughly.
55:28 Uh,
55:29 no,
55:29 the parents were fine.
55:30 I,
55:30 I mean,
55:30 listen,
55:31 I think that,
55:32 you know,
55:33 remember,
55:34 we are still coming over from the hangover of COVID at that time.
55:37 And I think that this idea of COVID was very nice because
55:41 interestingly the teachers were not very defensive,
55:43 right?
55:44 I mean,
55:44 otherwise when you tell teachers,
55:45 hey,
55:45 you need to change your behavior,
55:46 you didn't do well,
55:47 whenever you talk of learning gaps,
55:49 learning losses,
55:50 I mean,
55:51 the teachers think,
55:52 wow,
55:52 it's their fault,
55:53 uh,
55:54 and so they actually become very reactive and sometimes,
55:56 you know,
55:57 certainly not cooperative.
55:58 I mean,
55:58 here it was very clear there was an enemy which was COVID.
56:01 Uh,
56:02 so it was not really teachers who were at fault.
56:04 So teachers were very,
56:05 so,
56:05 so one part,
56:06 teachers were very,
56:08 uh,
56:08 supportive of this intervention.
56:10 Uh,
56:10 they thought these learning gaps existed.
56:12 Most of the teachers said
56:14 that the textbooks are really difficult.
56:16 Uh,
56:17 parents.
56:19 We did talk to parents.
56:20 I did talk to parents and all that.
56:22 Uh,
56:22 I think at that time coming out of COVID,
56:24 they were also concerned
56:26 that these kids had not been in school for so long
56:28 and therefore they were more worried about that these kids were losing
56:31 discipline and all this and his boys running around and all that.
56:34 So I don't think that at that particular time parents were,
56:37 were,
56:38 were also an issue.
56:40 In Islamabad,
56:40 we are having a parental uh component of there's a parental treatment on too.
56:46 And parents like the easier workbooks.
56:49 They think that they can actually monitor students' homeworks better.
56:53 Uh,
56:53 they were relying less on tutors.
56:56 So I think that this idea that the curriculum is difficult
56:59 is,
56:59 I want to talk about one last thing before once you have your hand up
57:01 on what I did not expect.
57:03 Hey,
57:03 listen,
57:04 97% of our students speak Pashto at home,
57:08 which is the main language of KP,
57:10 OK.
57:11 Uh,
57:11 the entire curriculum is in English and Urdu.
57:14 OK,
57:15 that's a fact.
57:17 Now
57:18 when you look at teachers,
57:19 I observed a lot of classrooms.
57:21 The teacher,
57:22 when he wants to impose discipline and yell at the kids,
57:24 which they do quite a bit in boys' schools,
57:26 uh,
57:27 it's all in Pashto,
57:28 uh,
57:29 when they want to,
57:30 when they start a lecture in Urdu,
57:33 uh,
57:33 on the board,
57:34 when people ask questions,
57:36 the de facto language turns to Pushto.
57:38 They can't explain,
57:39 and,
57:39 and the math curriculum,
57:41 believe it or not,
57:42 the textbooks for math are in English.
57:45 OK,
57:46 uh,
57:47 in this setting.
57:48 So I think that that's something that,
57:50 you know,
57:51 if you,
57:51 if you said,
57:52 you know,
57:52 in hindsight,
57:53 I think this is something we need to really work on.
57:56 I mean,
57:56 I'm very interested in pushing these studies.
57:58 I mean,
57:58 there are things,
57:58 right,
57:59 this
57:59 learning level variation,
58:01 uh,
58:02 which is not a part of a freelancer plan,
58:04 but when I started looking at within class variation,
58:06 it was so large
58:08 that I thought,
58:08 man,
58:08 I need to really work on this.
58:10 Uh,
58:11 so there are,
58:11 there are a bunch of things that,
58:13 you know,
58:13 we could have done,
58:14 uh.
58:16 Slightly differently,
58:17 I do want to really commend Zab,
58:19 uh,
58:21 and Malia particularly for,
58:23 for implementing this program.
58:25 I've done a lot of implementations in Pakistan,
58:27 you know,
58:28 mostly leap style with Asim and Jishnu,
58:30 information,
58:31 money,
58:31 things of that sort.
58:32 They are,
58:33 they are,
58:34 they are a piece of cake,
58:35 uh,
58:36 in implementing.
58:37 This
58:38 is testing 200,000 kids,
58:40 sorting them out,
58:41 implementing the program,
58:42 introducing technology.
58:44 It was just a heroic thing,
58:46 and I think we were all kind of,
58:48 uh,
58:49 so these are,
58:49 these,
58:49 these things are not easy to implement
58:51 in terms of,
58:52 of,
58:52 of,
58:52 of really doing it
58:54 both as a research project and as a serious implementation project.
58:57 And the problem is we had to test all the kids
59:00 because everybody had to be sorted,
59:02 even though the study sample was 4th graders,
59:05 but we had to do all of them.
59:07 Listen,
59:07 there's a lot of work.
59:08 This is a very rich data set.
59:10 I think that I can,
59:10 I can see myself working on it,
59:12 uh,
59:12 and in the subject area for a while.
59:15 What?
59:16 Yes,
59:16 just to add that,
59:17 yeah,
59:18 uh,
59:18 with all those complications,
59:20 where like,
59:20 you know,
59:21 like the complications also working with the government system and,
59:24 you know,
59:24 get the,
59:24 the authorizations and,
59:26 and I remember that telling,
59:28 uh,
59:28 uh,
59:28 Maliha that that this is our day to day at the bank.
59:32 So,
59:32 so it's like how do we move this huge,
59:35 uh,
59:35 huge thing.
59:36 Um,
59:37 I,
59:37 I,
59:37 I do want to address a little bit of the cost,
59:39 so I think uh there,
59:40 there are some calls,
59:41 the costs are not very high.
59:42 I don't have it in my head right now.
59:43 I don't know if sign up of Tahir have them,
59:45 but
59:45 a lot of the things that are part of the,
59:47 of the intervention are,
59:49 are calls that are already included within the system,
59:51 right?
59:51 Books,
59:52 a little bit of training and things like that.
59:54 So there,
59:55 uh,
59:55 there was no distribution of tablets,
59:57 there were,
59:58 there was nothing like that,
59:59 right?
59:59 The technology I think was there and we profited for that and then there was like,
1:00:03 uh,
1:00:03 uh,
1:00:04 as I now said some
1:00:05 Some work that,
1:00:06 that,
1:00:07 that um that was done to remap like the um the learning goals uh in the app,
1:00:12 right?
1:00:12 So,
1:00:13 but that compared to a program that could be so big is nothing,
1:00:16 right?
1:00:17 So that is very,
1:00:17 very tiny.
1:00:18 So,
1:00:19 so this is very implementable
1:00:21 and then uh scalable,
1:00:22 sorry,
1:00:23 and then the other part that I wanted to say is
1:00:25 this did require and,
1:00:27 and I think uh Tahir is,
1:00:28 is,
1:00:29 is,
1:00:29 is working uh also on another paper uh on
1:00:32 Uh,
1:00:33 on how,
1:00:34 how difficult was or,
1:00:36 or,
1:00:36 or all the things that need to change,
1:00:37 right?
1:00:38 Because there was some reorganization,
1:00:39 you were doing parallel,
1:00:40 right?
1:00:40 So,
1:00:41 so you need to,
1:00:42 to,
1:00:42 to,
1:00:42 to,
1:00:43 uh,
1:00:43 uh,
1:00:43 um,
1:00:44 reorganize kids by level.
1:00:45 So I think that is something that sometimes in
1:00:48 some of our reports are,
1:00:49 uh,
1:00:50 is,
1:00:51 is
1:00:51 taken for granted.
1:00:53 When,
1:00:53 when all that like
1:00:55 serious reorganization needs to happen within some cycles,
1:00:58 um,
1:00:58 I like come back to you.
1:01:00 Yeah,
1:01:00 thank you.
1:01:01 Um,
1:01:01 no,
1:01:02 so thank you,
1:01:02 thank,
1:01:03 thanks to all of you for implementing this.
1:01:05 I know it was difficult,
1:01:06 um,
1:01:06 and thanks for sharing
1:01:08 all of the results,
1:01:09 even some of that,
1:01:10 those baseline
1:01:11 results were fascinating.
1:01:12 Something as
1:01:13 or happening or not happening in 2nd grade.
1:01:16 It seems like everyone,
1:01:17 no one learns the 2nd grade curriculum.
1:01:19 So I think that
1:01:20 that is an interesting thing to probe.
1:01:22 Um,
1:01:23 I also like that you're showing the variation within schools.
1:01:26 We,
1:01:26 we have other studies in the portfolio,
1:01:29 um,
1:01:29 that
1:01:30 as an outcome,
1:01:31 look at how the,
1:01:32 the intervention.
1:01:34 Compressed or
1:01:36 or spread out the distribution of scores and I think that is something that's
1:01:40 so important and I,
1:01:41 and I
1:01:42 would wish that like all evaluations and education
1:01:45 look at that.
1:01:46 So,
1:01:46 so thank you.
1:01:46 So
1:01:47 we look forward to hearing
1:01:49 back from you in the future.
1:01:50 Uh,
1:01:51 Tahir mentioned all sorts of things he wanted to
1:01:53 explore and all sorts of rabbit holes he wanted to go down.
1:01:56 We welcome
1:01:57 you to present,
1:01:58 um,
1:01:59 what you find in the future and so just thank you,
1:02:01 um,
1:02:02 and thanks to everyone who attended.
1:02:05 Hey,
1:02:05 great questions.
1:02:06 Uh,
1:02:06 happy to talk more and respond.
1:02:08 Thank you so much.
1:02:11 Thank you,
1:02:11 everybody.
1:02:15 Thank you.
- add-style
- lp-body-content
Event Date
May 14, 2025, 12-1 pm Eastern Time
Abstract
We ran an eight-week large-scale foundational learning program (TIP) in public primary schools in Khyber Pakhtunkhwa, Pakistan to mitigate COVID-induced learning losses. The program incorporated both targeted instruction and structured pedagogy methods, and some teachers were also randomly assigned access to a low-cost, smartphone application (tech tool) designed to reduce the administrative burden of classroom management. Overall, the program raised student learning by 0.12 SD across English, Math, and Urdu. Learning gains, especially among low-performing students, were highest when the tech tool was randomly offered to teachers as an optional supplementary aid. Offering the paper version of the program (0.14 SD) or mandating the use of the tech tool (0.10 SD) had lower student learning relative to the optional tech (0.19 SD) version of the program. The flexible nature of the optional tech treatment contributed to better outcomes compared to the paper and mandated tech version of the treatment. This suggests that learning outcomes may not monotonically increase with technology take-up.
Speakers
Juan Baron
Senior Economist, World Bank
Tahir Andrabi
Stedman-Sumner Professor of Economics,
Pomona College
Tomoya Murakawa
Research Fellow,
Harvard Center for International Development
Alaka Holla
Senior Economist
Program Manager, Strategic Impact Evaluation Fund
World Bank