00:04 Uh,
00:04 OK,
00:04 we are going to start.
00:05 Uh,
00:06 thank you very much for being here.
00:08 Um,
00:09 this is the first BBL of the AAI,
00:12 uh,
00:12 digital development,
00:13 um,
00:14 Research Initiative at DRG.
00:17 Um,
00:18 the goal of this initiative is to
00:21 ask
00:21 what is the impact of AI and digital transformation
00:25 on growth,
00:27 development,
00:28 and inclusion in low and middle-income countries.
00:31 And as part of this initiative that is less By Anna Salman and Tuando,
00:37 uh,
00:37 we will have a series of events,
00:39 uh,
00:39 to show evidence of the impact of AI on several dimensions,
00:43 and today we have
00:45 Martin Desimone,
00:47 uh,
00:47 Maria Barron,
00:48 and Federico Tiberti,
00:50 uh,
00:51 to present,
00:52 uh,
00:52 work on AI in education in Nigeria.
00:56 Um,
00:57 thank you very much for being here.
00:59 Um,
00:59 Martin,
01:00 the floor is yours.
01:01 Yes,
01:01 thank you so much,
01:02 uh,
01:03 Elina and,
01:04 uh,
01:04 the DAG team for the,
01:06 for the invitation.
01:08 And also we have some other members of the team here,
01:10 including Aura,
01:10 and I'm sure the other ones are connected online,
01:12 but
01:13 thank you for the opportunity to,
01:15 to present.
01:15 Um,
01:16 we're happy to take questions
01:18 along the road,
01:19 but
01:20 we'll probably don't,
01:21 uh,
01:21 we won't be able to see them,
01:22 so maybe you can alert us,
01:24 yeah,
01:24 because most people are
01:25 online.
01:26 Uh,
01:27 so just,
01:27 uh,
01:27 to describe,
01:28 uh,
01:28 what we're going to be talking about in,
01:30 in a nutshell,
01:31 uh,
01:32 this is an evaluation of an after-school program for,
01:34 um,
01:35 virtual tutoring with a free tool,
01:37 and we'll discuss about that a little bit more.
01:40 Uh,
01:40 so it's,
01:41 uh,
01:41 uh,
01:42 2 sessions a week,
01:43 uh,
01:43 of an interaction between students and a large language model,
01:47 more specifically it was Microsoft CoPilott which at the time in which,
01:50 uh,
01:50 we did the intervention was
01:52 powered by Chachi PT4.
01:54 And the objective was to learn English,
01:56 uh,
01:57 uh,
01:57 in content aligned with the Nigerian curriculum
02:00 for senior secondary students.
02:01 So these are students of approximately
02:03 15 years of age,
02:05 uh,
02:05 and for these after school sessions,
02:07 there's a teacher that,
02:08 uh,
02:08 coordinates,
02:09 uh,
02:09 the session.
02:09 We'll talk a little bit more about
02:11 the implementation of,
02:13 uh,
02:13 the actual,
02:14 uh,
02:14 intervention because we think it's important also to understand the,
02:17 the effects.
02:18 Uh,
02:19 it was done in Edo State in Nigeria.
02:21 It's a state in,
02:22 in,
02:22 in the south of the country,
02:25 um.
02:26 Just to give you an idea,
02:28 basically,
02:28 it's a typical,
02:29 uh,
02:30 lower or lower middle income,
02:32 uh,
02:32 country development context,
02:34 and,
02:34 and we can also talk more about that.
02:36 But,
02:36 uh,
02:36 uh,
02:37 the typical challenges that you will face in
02:39 an implementation of this type of program in,
02:41 in,
02:41 in some of these contexts were present,
02:43 uh,
02:43 there as well.
02:45 Um,
02:45 so what was our motivation to start with this?
02:48 So we've,
02:49 uh,
02:49 known for a very long time
02:51 that,
02:52 um,
02:52 one on one tutoring is one of the most effective ways to improve,
02:56 uh,
02:56 learning outcomes.
02:58 Uh,
02:58 this is the famous paper by Benjamin Bloom in 1984 which,
03:02 uh,
03:02 he called it the two sigma problem instead of the two sigma solution.
03:06 And why did he call it the two sigma problem is because
03:08 he found out that one on one tutoring is very effective.
03:12 But of course
03:13 it is not cost effective.
03:14 It's very,
03:15 uh,
03:15 expensive to implement a program like that,
03:17 especially,
03:18 uh,
03:19 even more in a developing context.
03:22 So he called it the 2 sigma problem because he
03:24 was inviting the education community to try to find solutions
03:28 that will have a similar effect but that could be,
03:31 uh,
03:31 scale.
03:32 Um,
03:33 so one of the ideas that we had when the LLNs or
03:36 the large language models which are these systems such as Chat GPT
03:40 that
03:40 interact with the with you in almost in a human-like manner,
03:43 um,
03:44 uh,
03:44 emerged,
03:45 the,
03:45 the idea was to test whether they could help us,
03:48 you know,
03:48 achieve that idea of one on one tutoring
03:51 in a way that was,
03:52 uh,
03:52 scalable and,
03:53 and more cost effective,
03:55 at least that was the,
03:56 the initial motivation.
03:58 Um,
03:59 and of course,
03:59 not,
04:00 uh,
04:00 replacing the teachers but really complementing them.
04:03 And,
04:03 and that was the original idea that,
04:05 uh,
04:06 kind of prompted us to start this was in early 2024.
04:11 Um,
04:12 so there is a lot of,
04:13 uh,
04:13 related literature on,
04:14 on this,
04:15 but not really specifically on this,
04:17 on this topic of using generative artificial intelligence for learning outcomes.
04:21 So
04:22 we kind of frame the,
04:23 the literature in,
04:24 in that context of interventions that have worked
04:27 to improve learning outcomes,
04:28 especially trying to adapt,
04:30 um,
04:31 the,
04:32 to the level of the learner.
04:33 So one of the,
04:34 the,
04:34 the most uh common approaches that has been quite,
04:37 uh.
04:38 Cost effective is,
04:39 uh,
04:39 teaching at the right level.
04:40 There are many,
04:41 uh,
04:42 interventions,
04:42 many of them in South Asia that have been evaluated
04:45 rigorously
04:46 and more recently we have,
04:48 uh,
04:49 interventions of what is called computer assisted adaptive learning.
04:52 Uh,
04:53 these are software that basically,
04:55 uh,
04:56 adapt to the level of the student and therefore adjust
04:59 the difficulty based on the level of the student.
05:01 And they have uh been,
05:04 they have shown to really improve learning outcomes.
05:06 Perhaps the most
05:07 uh known uh paper is the one by
05:10 Muradidaran and,
05:11 and Ganimian in the context of India and uh
05:15 uh the MindSpark uh software.
05:18 So some of these uh um tools really uh use just um
05:23 uh algorithms in some cases machine learning,
05:26 but none of them really use generative artificial intelligence,
05:29 uh,
05:29 in the sense that they can provide this human-like interaction.
05:32 So
05:33 this is something uh relative.
05:34 in terms of generative AI and education,
05:36 there is a more,
05:38 uh,
05:38 recent literature with some,
05:40 uh,
05:40 examples,
05:41 most of them from,
05:42 uh,
05:42 developed countries
05:44 and also most of them,
05:45 uh,
05:45 assessing the effects of the intervention almost immediately after
05:49 you use the tool.
05:50 Like,
05:50 for example,
05:51 you know,
05:51 I interact with the,
05:52 with a,
05:52 uh,
05:53 a virtual tutor or a chachi PT or a similar tool,
05:56 and immediately after I have an assessment.
05:58 Um,
05:59 but that's,
06:00 uh,
06:00 not really the objective that we have here.
06:02 We wanted to see something relatively medium term and aligned
06:06 with the curriculum of the,
06:07 of the Nigeria,
06:08 uh,
06:09 context and
06:10 on top of that in a developing,
06:12 uh,
06:12 country context.
06:14 So there is some emerging studies,
06:15 but at the time we were doing it,
06:17 uh,
06:17 at least to our knowledge,
06:18 there was no,
06:19 no other study using generative AI,
06:22 uh,
06:22 for this.
06:23 Now we have a,
06:24 a few.
06:25 Um,
06:27 So,
06:27 um,
06:28 I wanted to spend some time talking about
06:30 the actual implementation because I think it really,
06:32 uh,
06:33 matters for how we interpret the results.
06:36 So here,
06:37 because we have part of the team that actually,
06:39 uh,
06:39 you know,
06:40 developed the content
06:41 and,
06:41 uh,
06:42 uh,
06:43 uh,
06:43 trained the teachers,
06:44 I will invite Maria to,
06:45 to tell us a little bit more about that experience.
06:48 Thank you,
06:48 Martin,
06:49 and thank you everyone for joining us even though it's very,
06:51 very cold today.
06:53 Um,
06:53 I'm Maria Barron.
06:54 I'm a member of the Global Ethic team,
06:57 uh,
06:57 based in Washington DC and I was very happy to support
07:01 this innovation in Nigeria,
07:02 especially focused
07:04 on designing the materials that were part and supported intervention.
07:08 So you may think,
07:09 why an implementation toolkit for teachers
07:12 if we are trying to just
07:14 implement an AI tutor for students.
07:17 Because for us,
07:17 it was very important to focus on the sustainability of the intervention.
07:21 Usually,
07:22 teachers,
07:22 and especially at the time that we were designing this intervention,
07:26 Had a lot of fears and still do about being replaced
07:30 about not having a role
07:31 uh in the use of these tools
07:33 so actually empowering through knowledge
07:36 was a key factor for us for this to be successful,
07:40 giving them a role
07:41 we call them orchestra conductors.
07:44 And having
07:45 this in-person training that lasted for 3 days,
07:48 all day,
07:49 3 days
07:50 was really helpful in establishing that.
07:53 The first day was focused on addressing these fears,
07:56 having an open conversation,
07:58 key concepts on AI literacy.
08:00 The second day was focused on how
08:03 the implementation of the intervention would take place,
08:06 the prongs they will use,
08:07 what would be the role,
08:09 and also
08:10 about them role playing the intervention themselves.
08:13 Because
08:13 by doing this
08:15 they could practice better how it would look
08:18 on over the course of the
08:20 of the two months that the implementation took place
08:22 and finally the third day we focus on
08:25 thinking about what could go wrong,
08:27 for example,
08:28 connectivity issues,
08:29 it was going to be rainy season,
08:31 what happened if there was a holiday and we couldn't do,
08:34 uh,
08:35 this
08:36 also about we focus on the monitoring strategy that I will get to in the next slides,
08:40 um.
08:42 We also,
08:43 and,
08:43 and in the training
08:44 it was myself,
08:46 my colleague Bra present here,
08:47 and
08:48 our other co-author Elio Locholomi decuo
08:51 that we took the lead on this training.
08:53 It was extremely hard,
08:55 uh,
08:55 hands-on and it was extremely intensive for the teachers.
08:59 However,
09:00 we also try to gather what they thought about the toolkit and the training.
09:04 Uh,
09:05 most of them valued a lot the design that incorporated them.
09:09 They valued the materials
09:11 and they valued the focus
09:12 on actually building capacity on them as well.
09:15 In addition to the implementation toolkit for them,
09:18 we also prepared
09:19 a specific toolkit for the students.
09:21 Um,
09:22 so I already mentioned this,
09:23 and it was approximately 45 teachers in attendance,
09:27 um.
09:28 And
09:29 maybe I just want to focus on that photo of Ura myself and
09:32 and Elliot
09:34 with the,
09:34 it was just it it was I don't know if,
09:36 uh,
09:37 a lot of people here connected
09:39 enjoy football,
09:39 but it was Manchester United and,
09:43 and,
09:44 and Manchester City playing that day and
09:45 there were teachers wearing those jerseys,
09:47 so it was a historical photo.
09:50 So,
09:50 so that's why we,
09:51 we thought it would be amazing.
09:53 Also another pertinent thing to say is that not all the teachers
09:57 were.
09:59 Teaching English,
10:00 uh,
10:01 actually some of them taught other materials,
10:03 uh,
10:03 such as,
10:04 um,
10:04 chemistry,
10:05 mathematics,
10:06 etc.
10:07 The interesting thing was when we chatted with them they were like,
10:11 I want to do this in my class as well,
10:13 um,
10:14 something that we didn't get to do
10:16 and that for me would be amazing to do is do
10:19 a follow up research with the teachers who took part,
10:22 how they are continuing to use AI if they are in what way,
10:27 uh,
10:27 and how it has feed on their working.
10:30 Um,
10:31 OK,
10:32 let me get to the monitoring and evaluation.
10:34 This was designed by
10:35 our colleague,
10:36 uh,
10:37 Federico Manollio,
10:38 uh,
10:39 and this also was
10:41 in the same vein of building capacity among teachers and students.
10:45 It was key to be able to monitor
10:48 the implementation of the,
10:50 of the program.
10:51 Why?
10:51 Because
10:52 working in ET tech,
10:53 we know that
10:54 probably one of
10:56 the most important barriers we have
10:58 is people actually using tools.
11:00 And people actually engaging actively with the tools
11:04 and to be able to do that we developed a monitoring strategy.
11:08 By the way,
11:09 both the monitoring strategy and the tool kit
11:11 and all of the materials will be published alongside the paper
11:14 that uh will be forthcoming
11:17 and.
11:18 And also our team,
11:19 um,
11:20 Martin,
11:21 Federicos,
11:22 Ura Elliot,
11:23 we would meet
11:24 periodically to evaluate the monitoring system
11:27 and to act,
11:28 uh,
11:29 accordingly.
11:30 I think this allowed to course correct in many things such as connectivity,
11:34 for example,
11:35 uh,
11:35 for example,
11:36 if students were not using the correct prompt,
11:39 etc.
11:39 so.
11:40 And this was only for 2 months,
11:42 but I think embedding this monitoring strategies
11:45 is also part of a sustainability
11:48 strategy that we need to take into
11:50 account when deploying this type of interventions,
11:53 slide,
11:53 and I think it's good.
11:55 Oh,
11:57 OK,
11:58 so here you can see,
11:59 uh,
12:01 How,
12:02 how it actually took place,
12:04 you can see that the teacher is moving and looking at what the students are doing
12:09 this for us was key and this was
12:11 what they practice to
12:13 when we trained them.
12:14 um,
12:15 I wish all tetarian.
12:17 would look like this
12:18 and not only like students focus on the screen but also the teacher
12:22 passing through it,
12:23 checking what they are doing,
12:25 being able to
12:27 uh answer to questions being able to guide them through this um
12:30 and over back to Marin.
12:33 Thank you Maria.
12:33 So just,
12:34 uh,
12:34 to,
12:35 before we move into actual
12:36 analysis,
12:37 to give you an idea of how the session was conducted,
12:40 the teacher among the to kids that,
12:41 um,
12:42 Maria and the team developed,
12:43 the,
12:43 the teacher had a guide for each of the sessions that was aligned
12:46 with the topic that they were covering that week in the regular classes.
12:50 So for example,
12:51 that week they were seeing antonyms and synonyms
12:53 or argumentative essay,
12:55 the guide would be about that,
12:57 and the guide is basically an initial prompt
12:59 that the teacher suggest to the students.
13:01 And after that initial prompt,
13:03 the idea was that really each student or each pair
13:05 of students because they were sitting in groups of two,
13:08 they will continue their own interaction with the LLM.
13:11 And after that,
13:11 the role of the,
13:12 the teacher,
13:13 as Maria was saying,
13:13 is to walk around,
13:14 check that they are actually interacting appropriately.
13:17 If they get stuck,
13:18 they also have a list of other prompts that they can suggest,
13:22 and they also can prevent them about potential risks like hallucination of the,
13:26 of the model
13:27 or over reliance on the model,
13:28 etc.
13:29 Uh,
13:30 and importantly,
13:31 I think we,
13:31 we can discuss this later,
13:32 but the prompts were crafted
13:34 very specifically to adjust to,
13:36 um,
13:37 the content of the,
13:38 the context of the south of,
13:40 uh,
13:41 Nigeria
13:41 and also to encourage the LLMs to act as a tutors,
13:45 not to provide,
13:45 uh,
13:46 direct answers to the students,
13:48 and we'll talk a little bit more about that,
13:49 uh,
13:49 later.
13:51 But now talking about the actual experimental design,
13:53 we had an opportunity to do a,
13:54 an,
13:55 an RCT.
13:56 Uh,
13:56 with individual level randomization,
13:58 which is not ideal,
13:59 uh,
13:59 but we were limited by the
14:02 number of schools that actually had computer labs,
14:05 um,
14:06 so this is one of the reasons why
14:08 the interpretation of the results is actually probably taken
14:11 into consideration that there were some spillover effects and therefore
14:15 the effect was attenuated.
14:17 Uh,
14:18 the treatment we discussed it,
14:19 the control was,
14:20 uh,
14:21 business as usual,
14:22 and we have our main outcome measure which was,
14:24 uh,
14:25 English,
14:25 uh,
14:26 so we had an assessment that was developed ahead of the intervention
14:30 with different questions and different,
14:31 uh,
14:32 uh,
14:32 levels of difficulty for the questions
14:34 that was really the main one,
14:35 but we also took advantage of the intervention
14:37 to measure knowledge of AI and knowledge of,
14:40 of digital skills.
14:41 Uh,
14:42 those were assessments developed by,
14:44 by us,
14:44 but we also measured the effect on the final school exam performance on
14:49 English,
14:49 which was not designed by us.
14:50 It was part of the actual
14:52 curriculum of,
14:53 uh,
14:54 the regular classes.
14:56 Um,
14:57 so we,
14:58 we did the randomization initially we had,
15:00 um,
15:01 60657 students assigned to the treatment group and 671 to the control group,
15:07 um,
15:08 but after that we had some,
15:10 uh,
15:10 attrition,
15:11 and
15:13 I think,
15:14 yeah,
15:14 we had some attrition
15:16 and therefore you will see that we try to address that in the analysis with some,
15:20 uh,
15:20 rebounds.
15:21 Uh,
15:22 so just,
15:22 just a second,
15:23 can you please go back?
15:25 So
15:27 it,
15:27 uh,
15:27 apparently part student participation was voluntary.
15:30 So,
15:30 uh,
15:30 yes,
15:31 so what we did,
15:31 or what actually the team did,
15:33 they went to the schools.
15:34 They told them about the inter the,
15:35 the program,
15:36 and the students voluntarily expressed that they
15:39 were interested in participating or not.
15:41 Among those we did the randomization.
15:44 OK,
15:44 so there's no such study.
15:45 I thought that was my point.
15:46 Yeah,
15:46 thank you.
15:47 The parents sign a consent form.
15:49 OK,
15:49 yeah,
15:49 thank you,
15:50 yeah.
15:51 Um,
15:53 Yeah,
15:54 so
15:55 just a uh an overview of what we will be presenting
15:58 it's just first just some descriptive statistics to show that the,
16:01 the,
16:02 the two groups were balanced,
16:04 uh,
16:04 then we'll go into the ITT results,
16:06 some heterogeneity analysis,
16:08 those response analysis,
16:10 because thanks to the monitoring tool we were able to measure attendance
16:14 and then some robustness checks including some sensitivity analysis
16:18 and also the rebound that I mentioned because the attrition was not,
16:21 uh.
16:22 Uh,
16:23 equal among the two groups and then we'll go into some interpretation of the results
16:28 and the and the discussion,
16:30 uh,
16:30 in terms in terms of the sample balance we had,
16:33 uh,
16:33 a pre-intervention questionnaire,
16:35 um,
16:36 and then we had,
16:37 uh,
16:37 uh,
16:38 the information about the performance in previous
16:40 exams that are part of the regular.
16:42 Uh,
16:42 school exams,
16:44 um,
16:44 and then some other information that was collected through the,
16:47 the questionnaire,
16:48 uh,
16:49 based on that using principal component analysis we,
16:52 we built a
16:53 socioeconomic index,
16:54 uh,
16:55 and basically overall we see that the sample is,
16:58 uh,
16:58 balanced,
16:59 um,
17:00 um,
17:01 between the two groups.
17:03 We had the power analysis and,
17:05 and,
17:05 and,
17:05 and this is just to show that we were well powered to detect the effect
17:10 ahead of the intervention.
17:11 We were not sure because the,
17:12 the effect size was really uncertain for us,
17:15 but,
17:15 uh,
17:15 luckily we were able to detect the effect
17:18 and we just wanted to show this to,
17:19 to show that the attrition,
17:21 um,
17:22 uh,
17:22 that we experienced during the,
17:24 during the intervention.
17:26 Um,
17:26 so this is just the intend to treat,
17:28 uh,
17:29 model,
17:29 uh,
17:30 we,
17:31 it's very simple,
17:31 we just,
17:32 uh,
17:32 included the school fixed effects
17:34 and,
17:35 uh,
17:35 this is our main result
17:37 and after this I will pass it over to Federico to go into the detail of the analysis,
17:42 uh,
17:42 but I just wanted to show,
17:44 uh,
17:44 on the,
17:44 on the table on the left you see,
17:46 um.
17:47 The main results on the total score,
17:49 which is an index of the three,
17:51 that I mentioned before,
17:52 English,
17:53 AI,
17:53 and digital skills,
17:55 we also use item response theory to,
17:58 um,
17:58 measure the results,
17:59 taking into consideration the actual empirical difficulty
18:03 of the questions,
18:04 uh,
18:04 and then the third column is the third exam which was not part
18:08 of our,
18:09 um,
18:10 it was not designed by,
18:11 by the research team,
18:12 it was part of the regular school,
18:14 uh,
18:14 exam.
18:15 Uh,
18:15 so you see that the effect is around 0.3 standard deviations,
18:19 uh,
18:20 depending on the,
18:20 on the specification,
18:22 and then,
18:23 uh,
18:23 on the right hand side you will see the desegregation by the three subjects,
18:27 uh,
18:27 that we measure,
18:28 uh,
18:28 the effect,
18:29 uh,
18:30 for English,
18:30 which was our main,
18:31 uh,
18:32 really outcome of interest was around 0.23,
18:36 uh,
18:36 standard deviations.
18:37 Um,
18:38 so
18:39 I will pass it over to Federico now to continue.
18:42 Just let me actually stop,
18:43 uh,
18:44 at this slide for a bit.
18:45 Um,
18:46 I'm gonna be quick because,
18:47 uh,
18:47 but I'm gonna walk you through some of the results,
18:50 uh.
18:51 So here we're particularly,
18:52 so this is basically an intend to treat,
18:54 uh,
18:54 those,
18:54 these are ITT estimates,
18:56 so this means that it's just the difference between
18:58 uh treatment control group at the assignment stage,
19:01 you know,
19:01 these are students uh assigned to treatment versus students uh assigned to,
19:05 to,
19:06 to control group,
19:07 uh,
19:08 ITT estimates,
19:09 which means that the,
19:09 the main,
19:10 the main,
19:11 uh,
19:11 variable of interest is
19:12 being
19:13 assigned to the treatment regardless of takeup.
19:15 We will get into that next.
19:17 Um,
19:18 as Martin said,
19:19 we're particularly happy about the 3rd column result because
19:22 that was not actually measured right after the intervention.
19:24 This is a,
19:25 this is an improvement we see on the,
19:28 uh,
19:28 test results,
19:29 on tests not crafted by our research team,
19:32 tests
19:33 as part of the regular curriculum of the,
19:35 of the students
19:36 at their schools,
19:37 and,
19:38 um,
19:39 this is just,
19:40 this has,
19:40 um,
19:41 school level fixed effects,
19:42 uh,
19:42 robust standard errors,
19:44 and,
19:44 um.
19:45 And yeah,
19:46 so,
19:47 yeah,
19:48 so that thought column,
19:49 I think another great thing if I understand correctly,
19:51 is that it doesn't have the attrition problem,
19:52 right,
19:53 because this is his school exams.
19:54 Well,
19:54 I mean,
19:55 not all of them take some of,
19:57 some of them are absent to the exam,
19:59 but it doesn't think they have that problem,
20:01 yeah,
20:02 OK,
20:02 as opposed to something that's specific,
20:03 OK.
20:05 Um,
20:07 yeah,
20:07 because the attrition problem is mostly at the our our assessment,
20:10 uh,
20:11 so do you have the balance for the school exam
20:14 in terms of the,
20:16 the,
20:16 um,
20:16 share of people who took it.
20:18 Uh,
20:19 it's balanced across,
20:20 yeah,
20:21 yeah,
20:21 it's,
20:21 yeah,
20:21 I tested for that,
20:22 so I could,
20:23 yeah,
20:24 um,
20:25 so yeah,
20:26 this is just a visual we provide on the plot,
20:28 uh,
20:28 we provide.
20:28 Can I ask you a question if you can go back?
20:30 I,
20:30 I just want to understand,
20:31 um,
20:32 all the materials that you prepared for the chatbot,
20:35 how they were related to this third
20:38 term exam.
20:39 I mean,
20:39 you follow the curriculum of that year to prepare the materials,
20:43 yeah,
20:43 so the,
20:43 the curriculum that we follow was done,
20:45 uh,
20:46 the intervention was 6 weeks.
20:47 So the guides,
20:48 uh,
20:49 specific,
20:49 there were some material that was generic about how to use,
20:52 uh,
20:52 an LLM,
20:52 etc.
20:53 but the,
20:54 the specific,
20:54 uh,
20:55 guides
20:56 who are,
20:56 um,
20:57 associated to the material that was supposed to be taught in those six weeks.
21:01 Uh,
21:01 so now the,
21:02 the,
21:02 the third term exam actually covers
21:04 the content for the entire year,
21:07 uh,
21:07 so
21:08 beyond what was covered in the guides that we provide.
21:10 That's another reason.
21:11 It's a particularly encouraging result,
21:14 um,
21:14 so 11 last question.
21:16 So,
21:16 uh,
21:16 this total score weighted,
21:18 am I supposed to think of that as sort of a weighted average of the three columns?
21:22 That's exactly right,
21:24 it has an additional specification which is it
21:26 has the weight of the difficulty of each question
21:30 that exacted the team determined,
21:32 uh,
21:33 because not all the questions had the same difficulty.
21:35 Um,
21:36 we also have unweighted,
21:37 and I think it's very similar.
21:38 It's the same,
21:39 yeah,
21:39 it's very similar.
21:40 OK,
21:40 so the reason I ask this question is that
21:42 because it's a weighted average of these three numbers.
21:46 0.31.
21:47 Yeah,
21:48 yeah,
21:48 it's higher than most.
21:49 It's
21:50 because of that weight,
21:50 yeah,
21:51 it's because of the weight,
21:52 uh,
21:52 I was assigned by the expert team who
21:55 constructed the assessment.
21:56 Uh,
21:57 the,
21:57 the questions are weighted by their,
21:59 by their difficulty.
22:00 I
22:00 see.
22:00 So it's not a,
22:01 not here,
22:01 it's not a weight by share of the exam of these three components.
22:04 It's actually a weighted by difficulty of
22:06 of the question.
22:09 Um,
22:10 there's a question
22:11 from you,
22:12 yeah,
22:13 in the chat.
22:13 So what was the takeup and completion rate?
22:16 Well,
22:16 we'll get into that,
22:17 uh,
22:18 very soon.
22:18 So,
22:19 uh,
22:19 if you bear with me for a minute,
22:21 um,
22:22 but this is just some,
22:23 um,
22:24 heterogeneity,
22:25 um,
22:26 across.
22:27 Uh,
22:28 pre-treatment covariants that we,
22:30 that we,
22:31 uh,
22:31 that we run,
22:32 and we find the,
22:33 the treatment to have been more effective among women,
22:36 which is an interesting finding because,
22:38 uh,
22:39 female students performed worse at the baseline,
22:42 but the,
22:43 the effect among them was stronger than among than among male students,
22:48 uh.
22:50 Uh,
22:51 that's a good interaction in writing.
22:53 It only interaction in writing,
22:54 yeah,
22:54 that's what I,
22:56 um,
22:57 so
22:57 this,
22:58 um.
23:00 The interaction with socio-economic status index,
23:02 as Martin said,
23:03 is the socio-economic status index is just the first component
23:07 of a principal component analysis of
23:10 access to some consumption goods that we,
23:12 uh,
23:12 asked about in the,
23:13 in the baseline and endline surveys,
23:15 like does your house have access to the internet?
23:18 Do you have access to your own computer?
23:20 Does your house have a private space for you to study in?
23:23 Uh,
23:23 uh,
23:24 like a number of those questions we constructed,
23:26 uh,
23:26 uh,
23:26 the first component of,
23:27 of those,
23:28 and,
23:28 uh,
23:29 the
23:30 treatment seems to have been more effective among
23:32 students with a higher value in that first component
23:36 and the treatment has,
23:37 uh,
23:37 also shows a stronger
23:39 effect among students who performed better in
23:42 previous terms exams of the school curricular,
23:45 regular curricular exams.
23:46 That's why we interacted with
23:48 second term exam.
23:50 You will see in the paper that we use the second term exam in lieu
23:54 of the typical like baseline outcome because
23:57 we didn't have like a pre-intervention assessment,
24:00 but uh these
24:01 previous terms,
24:02 school assessments serves us,
24:04 serve us in a way to,
24:05 to
24:06 assess students from in their baseline,
24:08 uh,
24:08 you know,
24:08 academic performance,
24:10 um.
24:11 Yeah,
24:12 so this is actually getting to the question that someone raised in the,
24:15 in the comment section.
24:17 Diego raised in the comment section.
24:18 We actually,
24:19 thanks to,
24:19 as Martin said,
24:20 thanks to our monitoring of the program,
24:22 we have actual data on how many times each student attended our,
24:27 our sessions.
24:28 So you will see that in,
24:29 uh,
24:30 we have like a median,
24:31 a median,
24:32 um,
24:33 attendance of around 70% of the sessions.
24:36 Uh,
24:37 most students attended a high number of sessions.
24:40 Uh,
24:41 and this allows us to refine our analysis so we,
24:43 we can move on from the,
24:45 from the ITT estimates,
24:46 we can move on,
24:47 uh,
24:47 to more sensitive measures of the treatment effect
24:50 based on,
24:51 uh,
24:51 actual,
24:52 actual take-up of the,
24:53 of the,
24:55 of the treatment.
24:56 Uh,
24:57 so what we do here is basically like a,
24:58 like a two-stage this square approach,
25:02 a typical IV approach where we instrument.
25:04 Uh,
25:05 the attendance with the assignment to the,
25:07 to the,
25:08 to the treatment groups.
25:09 So basically it's a two-stage regression in which this,
25:11 uh,
25:12 equation would be the second stage of the two
25:14 stages where an attendance is actually instrumented for by
25:18 the assignment to the treatment groups.
25:20 The rest of the model is as,
25:22 as,
25:22 uh,
25:23 for the sake of,
25:24 uh,
25:25 interpretability it is the same as in the ITT,
25:27 uh,
25:28 results.
25:29 Uh,
25:30 basically,
25:30 this is the,
25:31 the um.
25:34 So,
25:34 so this is not the estimate,
25:36 this is just a mean outcome by bins of attendance.
25:39 So this is actually we,
25:41 we split the,
25:42 the,
25:42 the total
25:43 number of days of attendance in 10 bins of equal,
25:46 uh,
25:47 actually 123499 bins of equal size,
25:51 and we estimate the mean,
25:53 the mean outcome.
25:55 So what we see here is that the.
25:57 Consistent with
25:58 the qualitative evidence that we gathered during
26:00 the actual implementation of the program,
26:03 the first few days were spent,
26:04 you know,
26:05 in just
26:05 teaching students on how to interact with the computer.
26:08 There wasn't,
26:08 there wasn't actually that much learning going on in the first days.
26:11 It was
26:12 actually again getting acquainted with the devices,
26:14 uh,
26:15 you know,
26:16 yeah,
26:16 setting up accounts,
26:18 email accounts.
26:18 Sometimes they didn't even have an email before,
26:20 so.
26:21 They had to set up the whole
26:22 infrastructure and,
26:23 and that's consistent with what,
26:24 what we see here,
26:25 which is
26:26 that the first few days actually show no changes in the mean outcome.
26:31 Changes in the mean outcome appear to speed up after some 4 or 5 days of attendance,
26:37 and they remain sort of linearly increasing as the number of days,
26:41 uh,
26:41 goes,
26:42 goes up.
26:43 So if I go.
26:45 For one slide,
26:46 this is just the,
26:47 this is just the um estimate for each individual day of increased attendance
26:53 in the two-stage list square.
26:55 0.03 uh is,
26:57 is 0.03 standard deviations is our estimate
27:01 across.
27:01 Most of the of the
27:03 of the specifications 0.03 is the estimate of an
27:08 of an additional day of attendance
27:10 based on our two-stage square approach
27:12 to uh which is our it's this coinD of 0.03
27:17 and uh.
27:18 And the fact that the that the
27:21 that this goes up linearly as the number of
27:24 days goes up without showing any like leveling off,
27:27 and,
27:27 you know,
27:28 it's an encouraging sign,
27:29 sign of scalability,
27:30 scalability of the program because they,
27:32 we don't see the effect,
27:34 sort of,
27:35 um.
27:36 Leveling off after a number of days and it was actually a pretty short intervention
27:40 where most of the students,
27:42 uh,
27:42 attended less than,
27:43 less than 10 sessions.
27:46 Could you repeat how long each session was?
27:49 15,
27:49 1 hour and a half.
27:51 Thank you.
27:52 So regarding the attendance,
27:54 I wonder if it's correlated with attrition.
27:56 Did you see that the kids who didn't present the
27:59 test at the end were the ones who only had,
28:01 I don't know,
28:01 2 or 3 sessions.
28:03 Or
28:04 are they were different from other kids?
28:06 There,
28:06 there was a statistically significant,
28:08 uh,
28:09 statistically significant difference there,
28:11 not super substantially different,
28:13 is,
28:13 is significant in terms of the magnitude of the difference,
28:16 but there was a statistically significant
28:18 attrition difference in attrition there.
28:20 We adjust the results based on attrition to using a number of techniques we,
28:24 we do,
28:24 you know.
28:26 Inverse probability waiting by the chance of,
28:28 you know,
28:28 a treating or not.
28:30 We also use sharp rebounds to sort of
28:33 trim the sample
28:35 and basically assume worst case scenarios
28:37 which rebounds are pretty brutal technique,
28:39 at least in my previous research to show that they
28:41 used to kill all the results and they don't hear,
28:43 so I was,
28:44 we're very relieved by that.
28:46 Uh,
28:46 but,
28:47 and,
28:47 and you,
28:48 because you have amazing data already,
28:50 but I,
28:50 I wonder if you got to learn a little bit about what hold behind some
28:55 kids sleep,
28:56 why did they are a treat or why they didn't complete the sessions,
28:58 because there were
29:00 quite this is more,
29:00 we don't have data on that,
29:01 but from a more qualitative point of view,
29:03 there are several things.
29:04 The first one is that for many of them it was really the first time using a computer.
29:08 Uh,
29:08 in fact,
29:08 one of the things that we were worried about is that.
29:11 They will spend a lot of time typing
29:13 because if you're not familiar with the computer,
29:14 you might type for a long time,
29:16 but the initial prompt,
29:18 we actually um
29:19 downloaded the prompts in their desktop so they could copy paste
29:24 or but,
29:24 but I,
29:25 I actually think that didn't work that well so but that was the,
29:27 the original idea.
29:29 And the second one is that um
29:31 uh we actually have,
29:32 although some of these might be um
29:34 controlled by the,
29:35 the school fixed effects,
29:36 but
29:37 uh there were a lot of problems with implementation in terms of um.
29:41 Uh,
29:42 floodings and the students couldn't go to school and,
29:45 and things like that,
29:46 um,
29:47 so that could also have,
29:47 uh,
29:48 affected them
29:49 and,
29:49 um,
29:50 yeah,
29:50 11 thing that we will discuss at the end is also whether,
29:54 you know,
29:54 if the effect was,
29:55 um,
29:56 lower,
29:56 not because they were not benefiting from the actual intervention,
29:59 but because
30:00 we're just attending less those who were,
30:01 uh,
30:02 poor performance before the intervention.
30:05 I don't know if you have any other thoughts on that,
30:07 on why it could have been.
30:11 I think they're
30:12 uh
30:14 I think it would be great to carry out some additional qualitative
30:17 research to understand a little bit more what would have happened.
30:21 Um,
30:22 one thing that we were discussing is
30:24 trying to find
30:25 some ways to explain
30:27 why it benefited more girls.
30:30 For example,
30:30 it may be that teachers
30:33 Um,
30:34 and this,
30:35 and there is a very hierarchical way of looking at education there.
30:39 So maybe interacting
30:40 with an AI tutor,
30:42 you could ask anything
30:43 and you wouldn't be ashamed of asking that,
30:45 right?
30:46 So I would love to,
30:48 to interview some students,
30:49 to interview some teachers,
30:51 and exactly
30:52 delve deeper into what was the mechanism.
30:56 Sorry,
30:56 and maybe one more potential reason for
30:59 student attrition or some students are attending.
31:02 This was done after school.
31:04 So for many of them,
31:05 maybe they had to travel back home to far distances.
31:07 What we tried to do in the program was provide incentives for them to stay back,
31:10 so snacks and meals,
31:12 um,
31:12 so at least they wouldn't have that concern,
31:13 but that's a potential reason why some of them may have not attended some sessions
31:18 or.
31:19 So basically a question we ask ourselves immediately after running the,
31:23 uh,
31:23 the,
31:24 you know,
31:24 the dose response analysis,
31:25 what would be
31:26 the effect of such an innovation if it,
31:28 if it were to be carried out over a longer span of time
31:31 and if you were to assume and if you were to buy
31:34 that uh there is a constant treatment effect here and uh,
31:37 and that linear sort of dose response relationship,
31:40 um,
31:42 which we believe
31:43 appear reasonable in our case,
31:45 uh,
31:45 and,
31:46 and this.
31:47 And keeping our attendance at the at the sample at our sample level of uh 72%
31:53 and a program of of a duration of
31:55 the entire school year could potentially have effects over
31:58 over 1.5 standard deviations that will,
32:00 that's what we,
32:02 and I,
32:02 I wanted to add here that basically we follow the methodology of the paper by
32:07 Muralidagan and
32:07 and Ganimian from from India.
32:09 But we follow a more conservative approach because we assume
32:12 zero effects for those who didn't attend at all,
32:15 um,
32:16 so that estimate is assuming that those who did not attend
32:19 didn't benefit at all basically.
32:24 Well,
32:24 and this is just a bunch of uh robustness checks that we run.
32:27 We probably don't need to spend as much time here.
32:29 The results are sensitive to,
32:31 uh,
32:31 so the results are robust to excluding schools at the sample,
32:35 uh,
32:35 from the sample at once,
32:37 uh,
32:38 iteratively.
32:39 Results are robust to,
32:41 uh,
32:41 alter,
32:42 well,
32:42 this,
32:43 uh,
32:43 the it's not super HD.
32:45 The results are great,
32:46 we promise,
32:47 uh,
32:47 anyway,
32:47 but,
32:48 um.
32:48 Now,
32:49 so this is the rebounds that we've been asked about attrition quite a bit today,
32:52 and uh we actually run the analysis
32:55 with,
32:55 uh,
32:56 waiting inverse probability waiting by,
32:58 by the chance of actually taking,
32:59 taking the assessment,
33:01 and we also conduct rebounds tightened at the school level,
33:04 yes,
33:05 but still rebounds where you trim the sample
33:08 and you assume that uh those who
33:12 Those who missed from the sample due to the attrition are actually
33:15 either
33:16 the students with the worst effect of the of
33:18 the program or the best effect of the program,
33:20 and then you you sort of uh
33:23 uh conduct the same analysis after the
33:26 after the bounces are performed.
33:27 Of course,
33:27 uh,
33:28 the estimates go down a little bit,
33:29 but they don't lose significance
33:32 after after doing that,
33:34 um.
33:36 Perhaps you,
33:36 Martin want,
33:37 want to,
33:38 because it,
33:39 yeah,
33:39 want to touch on this because,
33:41 yeah,
33:41 so maybe I will,
33:41 I will quickly talk about it,
33:43 uh,
33:44 because it's also relates to Diego's question about the,
33:46 the cost of the intervention.
33:47 I was trying to find the exact cost in the paper,
33:49 Diego,
33:50 but,
33:50 uh,
33:50 I will let you know.
33:51 We,
33:52 we can check,
33:52 but
33:53 basically,
33:54 um.
33:55 The effect that that is about 0.3 standard deviations
33:58 we compare it with the paper by uh Evans,
34:00 David Evans
34:01 and Yan um from 2022 in which they
34:04 basically compiled all the RCTs and the education literature
34:08 and it gives us an effect that is approximately 80% larger than most of the.
34:13 Um,
34:13 interventions in education,
34:15 including some of the most impactful ones
34:16 such as structure pedagogy or teaching at the right level,
34:20 uh,
34:20 something that that comparison doesn't take into consideration is the
34:23 duration of the intervention in our case was only,
34:25 um,
34:26 uh,
34:26 6 weeks,
34:27 uh,
34:28 in terms of equivalent years of schooling.
34:30 Um,
34:31 so under the business as usual scenario,
34:33 this is approximately equal to 1.5 years of
34:36 business as usual for the English estimates,
34:39 uh,
34:39 2 for,
34:41 um,
34:41 the total score.
34:43 And when we do equivalent years of schooling per $100
34:47 invested,
34:48 it,
34:48 it doubles because if I remember well,
34:50 and that's what I was trying to find,
34:51 I think it was around $48 per student taking into consideration both variable and
34:56 its costs.
34:57 Um,
35:00 So then,
35:01 uh,
35:01 we also calculated the learning adjusted years of schooling,
35:04 uh,
35:04 for the,
35:05 the six weeks of the intervention and it is around 0.3 or 0.6 per $100.
35:11 So in,
35:12 in,
35:12 in a nutshell,
35:13 it's really
35:14 we believe that it's quite impactful,
35:16 especially if we take into consideration
35:18 some of the caveats like the fact that the randomization was done at the,
35:21 at the individual level.
35:23 Uh,
35:24 we also in the paper we provide estimates of what the lays would look like under
35:28 the assumptions,
35:29 the assumption of a constant un effect that Federico was,
35:32 was describing,
35:34 and it's,
35:34 uh,
35:35 around 1.25,
35:36 um,
35:37 in total,
35:38 uh,
35:39 because we have detailed,
35:41 um,
35:41 estimates on the,
35:42 on the cost of the intervention including.
35:45 Uh,
35:45 some of the,
35:46 the hardware,
35:47 the software,
35:48 and
35:49 the technical,
35:50 uh,
35:50 design of the actual toolkits,
35:52 uh,
35:53 we calculate,
35:54 uh,
35:54 the benefit to cost ratio which is really high and,
35:56 and higher than,
35:58 uh,
35:58 you know,
35:59 we compare with the Agri paper in terms of,
36:01 uh,
36:01 the,
36:02 the benefit-cost ratio of different interventions and it's really
36:05 on the upper bound of the different,
36:07 uh,
36:07 interventions.
36:09 This is,
36:09 uh,
36:10 I forgot I had this one,
36:11 but this is what I was saying about the paper of
36:13 Evans and,
36:14 and Yuan,
36:15 um,
36:16 yeah.
36:18 So
36:20 for the question you develop the tool
36:22 in-house completely.
36:24 You didn't source anything.
36:25 It was fully developed by Maria Aura,
36:28 and then we provided the comments.
36:31 The idea,
36:31 and this is important,
36:32 it's one of the policy implications,
36:34 is that
36:35 really the,
36:36 the,
36:37 the prompt was trying to encourage the LLM to act as a tutor instead
36:40 of provide direct responses and trying to use some of the science of learning.
36:44 Um,
36:45 so
36:46 we believe that,
36:47 you know,
36:47 the,
36:47 the effect of,
36:49 of the,
36:50 of the intervention is not just by interacting with
36:52 the LLM but it's the package as a whole.
36:54 And in fact,
36:55 one of the things we wanted to mention is that
36:57 there is some recent evidence that LLMs can actually harm learning in some cases
37:01 or can lead to what some people call cognitive offloading.
37:04 So basically,
37:05 I don't think critically and I just get the answers
37:08 and uh
37:09 our interpretation is that as
37:11 almost any educational technology really depends on how you use it.
37:14 And in our case it was used in a way that was aligned with the curriculum.
37:18 It was
37:19 with an important role for the teacher and it was also,
37:21 uh,
37:22 based on the science of learning for the prompts.
37:24 What are you gonna say something?
37:25 No,
37:25 I was just going to say this is part of
37:27 why we had teachers serve as the facilitators of.
37:30 The sessions.
37:31 We didn't want the students to just
37:32 um have
37:34 maybe
37:34 free freedom to do as they wanted.
37:37 And we wanted the teachers to go on and make sure that they were using it correctly.
37:39 And this is also why we had the teacher's role play to sort of
37:42 identify what those risks could be and be able to address them effectively.
37:47 Yeah,
37:47 just a quick addition to that,
37:49 um,
37:50 I pasted in the chat for those interested one of the sample prompts we have used.
37:55 You will notice that it is very long,
37:57 and the reason why it's very long is because,
37:58 as Martin was saying,
38:00 we were,
38:01 or I was at least very worried
38:03 about this not being,
38:05 um,
38:06 detrimental for students.
38:08 I want,
38:08 I want the LLM which in this case was.
38:11 A pilot chat that we use for free
38:13 to ask them questions and actually encourage their their thinking rather than
38:18 just giving quick answers and,
38:19 and,
38:20 and,
38:20 and just not having good interaction
38:23 that led to further development or what they had already learned.
38:27 In addition to this initial prompt,
38:29 which was long,
38:30 we also put some questions that were follow up and could be suggested.
38:35 They were prepared for for each session.
38:37 Um,
38:38 and the students had also a chance to innovate
38:41 and ask anything they wanted on this specific topic.
38:45 Uh,
38:45 I think
38:47 because I'm,
38:48 I'm on the ethic team,
38:49 I must say that when we did this just last year,
38:52 uh,
38:53 the idea of custom GPTs and and agents
38:56 was not yet a possibility.
38:58 However,
38:59 um,
39:00 if we were to do this again or if somebody here would like to do this again.
39:05 I think the possibility of using uh
39:08 a custom GPT or an agent
39:10 would be,
39:11 would be really need to be explored because it would allowed uh
39:16 to actually introduce the prompts already and program the agent
39:20 so that the student didn't have to paste
39:23 uh or type
39:24 this very long prompt
39:25 to get the same results.
39:27 Also it would allow
39:28 to feed this agent.
39:29 Or custom GPT
39:31 with the specific information,
39:33 for example,
39:33 the curriculum in Nigeria,
39:35 any other materials that could be fed into that,
39:37 etc.
39:38 So technology is moving fast.
39:40 So I think
39:40 this partnership between um
39:42 very high skilled researchers,
39:45 local experts,
39:46 and net tech experts is key.
39:48 So,
39:49 so can you give uh a simple example
39:51 of how these kids use LGBT,
39:53 some,
39:55 some particular examples,
39:56 some prompts,
39:56 some questions,
39:57 how they did it.
39:59 Yeah,
40:00 yeah,
40:00 go ahead.
40:02 Oh.
40:03 Perhaps I,
40:04 I should,
40:04 I should have included a screenshot,
40:05 but on the,
40:06 on the,
40:06 I'm gonna read it from the last page of the paper because we have an,
40:09 an annex there,
40:10 and as Maria said,
40:11 we will make all the materials available as well,
40:13 and
40:14 if someone wants them before we publish the paper,
40:16 we're happy to,
40:16 to share them as well.
40:18 Um,
40:19 but,
40:20 uh,
40:20 I wanted to,
40:21 uh,
40:22 um,
40:22 yeah,
40:22 I have one here.
40:24 OK.
40:24 So
40:25 for example,
40:26 good morning,
40:26 afternoon,
40:27 I am,
40:28 or afternoon,
40:29 I am a student of grade one of senior secondary education in Benin City,
40:32 Nigeria.
40:33 I would like to ask you to support me acting
40:35 as a well-seasoned English grammar tutor to enhance my learning.
40:39 My teacher will be supervising this exercise.
40:41 We are testing using Co-Pilott to complement English lessons.
40:45 We are currently focusing on creative writing.
40:47 In a specific,
40:48 I would like to write my own short story about X,
40:51 and they grab their specific topic they were
40:54 interested in.
40:55 I will need you to reply to my questions in a motivational and engaging tone.
40:59 Your delivery should be clear and descriptive,
41:01 yet to the point using paragraphs and
41:03 Bullets.
41:04 After providing an answer with links to
41:06 sources of information and checking for understanding,
41:09 I would like you to follow up asking me
41:11 questions or proposing an exercise on this topic.
41:15 Wait for my answer,
41:16 and if the answer is correct,
41:17 celebrate my progress.
41:18 If it's incorrect,
41:19 provide encouraging words and provide a hint
41:22 to arrive at the correct answer.
41:24 If my reply this is still incorrect,
41:25 you may provide the answer offering an explanation.
41:28 After checking for my understanding,
41:30 continue with another question or exercise.
41:33 Follow this model for the whole interaction.
41:35 Please confirm that you have understood the task.
41:37 So very long,
41:38 but very focused on the interaction.
41:42 I think uh,
41:43 Anya has a question.
41:46 Anya,
41:47 uh,
41:47 yes,
41:47 um,
41:48 I was mostly curious.
41:51 About some of the technical decisions you made and how you designed the prompts.
41:55 So,
41:56 um,
41:56 as you may know,
41:57 I'm doing something a little bit similar in Nigerian clinics where we are,
42:02 um,
42:03 we're basically reviewing,
42:05 uh,
42:06 health worker notes and we're letting the LLM comment on
42:09 these notes and we did this with an API integration.
42:12 Of the LLM into the,
42:14 you know,
42:14 the electronic patient management system that they have
42:17 in the clinics we work with and then um
42:19 you know we did,
42:20 we,
42:21 we proceed every
42:23 Patient note with a,
42:24 with a prompt that the health workers don't even see,
42:27 right?
42:27 We basically just send off the prompt and then the,
42:30 the patient note with it and then the feedback comes back
42:33 and we spend a lot of time on doing prompt engineering for that.
42:37 So I guess I'd be curious what in terms of prompt engineering
42:40 what you did and I actually don't know what co-pilot uses underlying,
42:44 but did that change?
42:45 Did the model actually change that's used because I
42:48 assume OpenAI keeps feeding their new model versions in.
42:52 Yeah,
42:52 so when,
42:53 when we implemented,
42:54 it was uh
42:55 Uh,
42:55 Chachi PT4,
42:56 the reason why we chose Co-Pilott it was because at
42:59 the time it was the only way to use Chachi PT4
43:01 for free,
43:02 basically,
43:03 uh,
43:03 and for us that it was free and almost without fine tuning
43:07 it was very important because we wanted to check something that could be
43:10 implemented
43:12 anywhere in,
43:13 in at least in the state,
43:14 uh,
43:14 rapidly.
43:15 So Maria can talk more about the,
43:17 the prompt engineer,
43:18 but,
43:18 uh,
43:19 it took a,
43:20 a while.
43:21 And uh
43:22 what we try to do is to really align it with
43:24 the science of learning and some of the key principles yeah
43:27 so and yeah I can walk you through
43:30 how we started this
43:31 so.
43:33 There was a lot of buzz around the time
43:34 that we were designing this evaluation on how to prompt
43:39 um
43:40 Cha GPT mostly
43:41 to act as a tutor.
43:43 There were several prompts running around the internet,
43:46 some of them
43:47 by experts like Ethan Molik,
43:49 etc.
43:51 What I did was just starting from seeing what had worked,
43:55 then just testing it with copilot myself.
43:58 So for example,
43:59 I had a,
44:00 a first version
44:01 that didn't put that much uh emphasis on not providing a direct answer but a hint.
44:08 And I didn't like the results it gave because then
44:11 I felt the interaction was a little bit empty and a kid
44:13 who would be struggling wouldn't get the best out of this exercise
44:17 because if you just give the answer
44:20 you lose a lot of that critical thinking part
44:22 so it was
44:23 constant uh testing
44:25 of the prompts
44:26 and
44:28 my.
44:29 My intention was to also get it as short as possible,
44:31 but as you can see,
44:32 it's not very short.
44:34 Uh,
44:35 however,
44:35 this whole prompting
44:37 could be done now to a custom GPT using
44:41 a GPT if you want to,
44:42 or it can be done in an agent in copilot,
44:45 and you can prompt all of this,
44:47 put all of,
44:47 and it's also for free.
44:49 So
44:50 that is something
44:51 if it's suitable for your project and research is something I would explore.
44:55 There is still.
44:57 So if you were to ask me,
45:00 um,
45:01 you know,
45:01 the Chat GPT 40 and more recent models
45:05 have gone through,
45:06 I don't know,
45:06 they have passed the bar exam,
45:07 etc.
45:08 with agents we still don't have that information.
45:11 However,
45:12 it will be as good as the information you feed it
45:15 and the interaction you have with users,
45:17 uh,
45:17 but I think it's worthwhile to explore it
45:19 because then you can limit
45:21 the information it draws,
45:23 draws on,
45:24 especially if you are working with sensitive information.
45:27 This could be
45:28 an area that is worthwhile to explore,
45:30 but
45:30 happy to chat more and help you through
45:33 this process because I think it's um
45:37 Um,
45:37 I,
45:37 I also must mention my colleagues in the tech team because I
45:41 also share all of these materials with them and they gave me,
45:43 um,
45:44 their thoughts,
45:45 and
45:45 I think
45:47 prompt engineering,
45:48 and,
45:48 and most experts will tell you that
45:50 is going to become an extinct art
45:53 because agents
45:54 will,
45:55 will really get really,
45:56 really good,
45:57 um,
45:58 and also there are new models.
46:01 That will become
46:02 a specific and they will have a different interface
46:05 using Azure models,
46:06 using other
46:07 APIs.
46:08 You can
46:09 do models where you don't need to prompt that much
46:12 to get the best results.
46:13 So
46:14 that's the frontier.
46:15 I think that now prompting is still needed
46:17 and it is useful to learn how to best prompt,
46:20 but at the same time I feel
46:22 in the coming maybe,
46:24 maybe one year,
46:25 maybe less.
46:26 We will see
46:28 less need to prompt that much because all of this will be embedded
46:31 in the model.
46:33 Yeah,
46:33 that's my impression too,
46:35 so.
46:37 Sorry,
46:37 I'm,
46:37 I'm monopolizing a little bit,
46:38 but I would love to compare notes a little bit on what you did exactly.
46:41 So what we did,
46:42 our,
46:43 just for reference,
46:44 our trial is in the field right now,
46:45 so the data is coming in
46:47 and we think that the LLM has a lot of impact on what these,
46:51 what these health workers do.
46:53 We had a lot of more,
46:54 you know,
46:55 security features in place and so on because we
46:56 obviously don't let the LLM actually treat patients,
46:59 so a lot of the trial design was around that.
47:01 But we also spent a ton of time on the prompt engineering.
47:04 We did purposefully not choose a rack model or anything like that because
47:08 the models update so fast that any rack model,
47:10 I think in these contexts is gonna be more costly to design
47:14 instead of just paying the credits for the next better model.
47:18 And so I think we're using 01 mini now or something.
47:21 I don't even know,
47:22 just for the trial,
47:22 right?
47:23 It's,
47:23 you know,
47:24 for scale up you would need to think about the free solution.
47:27 But um
47:28 one of the things that we did in the
47:30 prompt engineering department is we have a couple of
47:31 computer scientists on for them it's more like a
47:34 hobby than an actual project because it's sort of
47:37 the fringes of computer science
47:39 but I mean so they did a lot of work basically
47:42 having the
47:44 prompt engineering automated almost in the sense
47:47 that they we feed patient notes into the
47:50 different.
47:51 You know,
47:51 into the LLM with different prompts and then
47:53 we let the LLM actually evaluate the match of
47:57 this,
47:58 the output from the LLM with the,
48:00 with the medical doctor's assessment of the same patient and so we
48:02 did a couple of these like sort of meta users of GPT 4
48:07 when we did the prompt engineering it was GPT 4 at this point we are
48:10 evolved from that and have,
48:11 we have a,
48:12 we are now on the 3rd or 4th model or something,
48:15 but anyway,
48:16 I think there is a lot of strategies one can use and in all cases my sense is that.
48:21 Building custom,
48:22 I don't know,
48:22 I would love to hear your thoughts,
48:23 but custom models are not the way to go,
48:25 right?
48:25 It's really,
48:27 firstly,
48:27 I have very strong opinions on that because
48:30 most of the times we present this is,
48:32 uh,
48:33 the,
48:33 the initial reaction is like,
48:34 oh,
48:34 but did you customize it or did you fine tune it.
48:37 And uh
48:38 the truth is that as you said is the issue of costs
48:41 but also the issue of of performance in in many cases the accuracy is lower
48:45 than a general purpose LLM uh especially now that the LLMs are getting really good so
48:51 uh I understand that the first thing.
48:52 I think it's usually to to fine tune,
48:54 uh,
48:54 but
48:55 it's,
48:56 I think it's what you're doing is the right thing to do to compare and,
48:59 and,
48:59 and then decide rather than just
49:01 immerse yourself in the,
49:02 in the fine tuning,
49:03 but
49:04 I don't know,
49:04 Maria,
49:04 you agree with that.
49:05 Well,
49:06 sometimes we disagree on this one.
49:08 I am,
49:08 I am not on one side or the other yet because I think
49:12 because agents and custom GPTs are newer,
49:15 we still don't have enough data
49:16 to know how good they're going to become,
49:19 um.
49:20 However,
49:21 I do think that the same way it happened with LLMs,
49:25 they will get better
49:26 and they will get better if you interact
49:28 with them more,
49:30 so with more user experience
49:31 and if you feed them more information.
49:33 So obviously it's not going to perform as well if
49:36 you just feed a couple of documents into that,
49:39 right?
49:39 Um.
49:41 Then again,
49:42 I,
49:42 there is also one other caveat.
49:44 So
49:45 these custom GPTs and,
49:47 and agents,
49:48 they should
49:49 before
49:50 they need to be paid for,
49:51 but now they can be created for free.
49:55 So
49:55 for example,
49:56 I
49:57 with the CKL we do this AI toolsday,
49:59 right?
49:59 So we created a a custom GPT to teach,
50:02 uh,
50:02 people in the World Bank how to best use AI.
50:05 And it has become
50:07 extremely better because
50:08 more people are using it.
50:10 It's very anecdotal evidence,
50:12 so I don't have a lot of information on this,
50:14 but at first it was
50:16 a bit dumb.
50:16 It was just like,
50:17 OK,
50:17 this part,
50:18 it will,
50:19 you will ask something and it will draw whatever answer from this,
50:23 but now it's getting better and better because people are uploading use cases,
50:27 they are loading their own materials,
50:29 uploading ICRs,
50:30 etc.
50:30 so.
50:31 I think
50:34 We need to,
50:35 I don't have a fully formed opinion on that yet.
50:38 I think we need to
50:39 test and see what happens.
50:42 But
50:43 this I just one more,
50:44 can I ask one more question about
50:45 the technical side,
50:46 so the big pushback,
50:48 so we have gates funded
50:50 and gates were totally um.
50:53 They are really on the train of a they want voice input
50:57 because they,
50:57 they said that they don't think these health workers
51:00 will in public clinics or whatever will actually type in
51:04 um in English
51:05 or even you know in,
51:07 in,
51:07 in whatever local like Hausa in our case
51:11 um
51:12 but they were also super concerned about the on like the need to be online
51:17 um have you thought about using
51:20 Smaller offline models.
51:23 We,
51:23 we might also disagree here with Maria,
51:25 but,
51:26 uh,
51:27 uh,
51:27 I think what I've seen from small language models is that at least for tutoring the,
51:31 the accuracy is quite low yet.
51:33 Hopefully,
51:34 you know,
51:34 over the next,
51:34 uh,
51:35 year it will change a lot,
51:36 but
51:37 for now it,
51:38 it was very low to,
51:39 to,
51:39 to try
51:41 this kind of context.
51:41 I don't know,
51:42 Maria.
51:43 So I don't know if my colleague Leandro is in this,
51:46 uh,
51:47 call,
51:47 but if not,
51:48 we are actually testing,
51:49 uh,
51:50 WhatsApp
51:51 based and fit with Chat GPT,
51:54 um,
51:55 application,
51:56 uh,
51:56 to build lesson plans for teachers,
51:59 and it works pretty well,
52:01 um.
52:03 Again,
52:04 it took a lot of time and this is an organization in Brazil who
52:06 developed this and it took them a long time to fine tune it.
52:10 But I do think we need to think about how,
52:13 I mean,
52:13 using the API you can basically
52:16 use all of that power,
52:17 um.
52:19 Using WhatsApp-based solutions,
52:20 right,
52:21 and I think especially when we are thinking about low to middle income countries,
52:26 this is something that we should explore more
52:29 because connectivity is,
52:31 is a struggle,
52:31 but that's not a local model.
52:32 That's not a small model,
52:33 right?
52:34 Um,
52:35 it is a small model because it's adapted only to the Brazilian Caribbean.
52:39 OK.
52:41 Yeah,
52:42 but it's still on,
52:43 I think it's,
52:43 it's fine tuned,
52:44 but not necessarily uh 11,
52:47 right?
52:47 But then,
52:48 yeah,
52:49 anyways,
52:49 we'll,
52:49 we'll,
52:49 we'll see.
52:50 But
52:51 I think there's a lot of potential for,
52:52 for that approach as well.
52:54 I think as of now
52:55 I haven't seen accuracy that is very high,
52:57 uh,
52:58 for small language models.
53:00 But
53:02 Everything might change in a,
53:02 in a month.
53:04 This is very fast,
53:05 um.
53:07 Uh,
53:08 yeah,
53:08 and of course,
53:09 uh,
53:09 I think the big advantage that,
53:10 uh,
53:12 the big,
53:12 the big advantage that Maria was highlighting is that
53:15 it's of those have the capability of being offline,
53:17 which for us it was a huge problem,
53:19 you know,
53:19 the internet connection was very unstable,
53:21 especially in some of the schools,
53:23 so that would be a,
53:24 uh,
53:25 a very big advantage.
53:26 And at the end of the day it also depends on your specific use case
53:29 if the model can retrieve information from a very limited.
53:33 Source of documents or,
53:35 well,
53:35 in that case,
53:35 uh a small language model might work,
53:37 but
53:38 for a tutor that really requires to
53:40 really have a lot of flexibility and versatility,
53:43 I think we're,
53:44 we're not there yet in my view.
53:46 Yeah.
53:48 Yeah,
53:48 I don't know,
53:49 maybe instead of going to the implications and,
53:51 and other ideas we could just ask if there are any other questions.
54:01 You know.
54:03 I'll just say that this is really great work,
54:05 very impressive,
54:06 especially up in Nigeria.
54:11 So 11 quick question is that the,
54:12 uh,
54:13 the,
54:13 these students that expressed an interest.
54:17 These are not random students
54:19 and so in some sense what you're obtaining is upper bounds
54:23 of the effects or not.
54:28 It's a good question.
54:29 I don't know if it's upper bounds because
54:30 we also have other things that attenuated the,
54:33 the effect,
54:34 uh,
54:35 but definitely what we don't have unfortunately is data on,
54:38 on the other students
54:39 because otherwise we could compare if they,
54:41 if they are,
54:41 uh,
54:42 uh,
54:43 we could see they're comparable.
54:45 Uh,
54:45 the other,
54:45 the other big thing that we need to take into consideration is that
54:49 even though these are public schools.
54:51 And
54:52 they're really,
54:53 you know,
54:53 resource constrained environments.
54:55 They are also the only schools where we could find
54:57 at least a space to have a computer lab,
54:59 so they're probably not representative of all the schools.
55:02 So no,
55:02 the,
55:02 the only reason I bring that up is that because when,
55:05 when we make these,
55:06 uh,
55:07 the calculations for the effectiveness would be.
55:10 I think we have to be clear as
55:12 to
55:12 the fact that these are in some sense sort of privilege it's a privileged setting in
55:17 I think there are there are biases working in opposite directions here.
55:19 I think there are some,
55:20 some,
55:21 uh,
55:22 biases towards.
55:24 Higher results in those senses that you mean for example we are testing this
55:27 in schools that have like at least
55:29 infrastructural capacity to host such a such program
55:33 and
55:34 even the
55:35 sorting into uh the eligibility for the lottery uh
55:39 it was based on expressed interest so as you said
55:42 we're still talking about the subset of students who
55:46 Were more likely to the motivation to be engaged in in in something like
55:50 that at the same time they're attenuating bias concerns at at the same time because
55:54 there was a lot of interaction between
55:57 control and treatment students
55:59 because this was randomized at the individual level and we even saw
56:02 some uh control students.
56:05 Some control group students show up at the sessions,
56:08 especially the first,
56:09 uh,
56:09 few days,
56:10 yeah,
56:10 yeah,
56:10 it was hard for the,
56:11 you know,
56:12 at the school level it was hard to control that,
56:15 so that would operate in the opposite
56:16 direction because those students are likely getting
56:19 at least some spillover
56:20 effects from the,
56:21 from the intervention without.
56:24 Being actually,
56:25 you know,
56:26 selected for it,
56:27 so,
56:27 uh,
56:27 there are,
56:28 there are some concerns in both directions.
56:32 Yeah.
56:34 Uh,
56:34 but,
56:35 uh,
56:35 uh,
56:35 before the intervention,
56:36 just to give some context,
56:37 this,
56:38 uh,
56:38 computer labs didn't have access to the Internet.
56:40 In fact,
56:41 many of them didn't have computers,
56:42 to be honest,
56:43 we,
56:43 we even,
56:44 uh,
56:45 leveraged the an asset disposal program that the World Bank has,
56:48 and then we found out about it.
56:50 This basically these are computers that the World Bank doesn't use anymore,
56:52 but they're still.
56:53 Uh,
56:54 you know,
56:54 and we donated,
56:55 I think it was around more than 200 computers if I remember well,
56:58 so,
56:59 uh,
57:00 um,
57:01 it is true that these were schools that,
57:04 you know,
57:04 had at least a room to act as a computer lab,
57:06 which is not common,
57:07 but also it's not that,
57:08 uh,
57:09 I don't,
57:09 I don't wanna give you the idea that these were advanced,
57:11 uh,
57:11 schools or private schools in the south of the country,
57:14 yeah,
57:15 yeah,
57:16 and those costs,
57:17 uh,
57:18 you said you have hardware costing your cost effective.
57:21 Yeah,
57:21 we,
57:21 we,
57:22 we calculated the,
57:23 the cost with the proper amortization for the timing that uh
57:27 it was used,
57:27 so all those details and are in the,
57:30 the working paper.
57:35 It's a really nice thing for.
57:37 Yeah.
57:38 Thank you so much and extremely interesting what you're doing in,
57:41 in the health sector.
57:43 That's a really fascinating.
57:45 I'm,
57:45 I'm,
57:45 I'm so sorry I can't be there in person.
57:47 I would love to talk some more about
57:49 all those things if you're thinking about scale up,
57:51 um,
57:52 you know,
57:53 lots of stuff to discuss.
57:55 Absolutely,
57:56 yes,
57:58 thanks.
- add-style
- lp-body-content
On February 18, World Bank experts Martin De Simone, María Barron, and Federico Tiberti gave a brown bag talk for the AI and Digital Development research initiative, presenting the main findings of their study 'From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria.'
The speakers presented key findings from their study, highlighting the impact of large language models on virtual tutoring in Nigerian secondary education. They discussed the significant learning gains observed among students using Microsoft Copilot, as well as the challenges encountered during implementation