College Field Specialization and Beliefs about Relative Performance: An Experimental Intervention to Understand Gender Gaps in STEM

videoType

dynamic-media

videoDmUrl

keyFrameImage

timestamp

00:01 Thanks so much.

00:02 Um,

00:02 I'm thrilled to be talking to all of you today and,

00:05 and meeting with some of you as well,

00:07 um,

00:07 to talk about my,

00:08 my work and my job market paper.

00:10 Um,

00:10 so,

00:10 this project is about college field

00:12 specialization and beliefs about relative performance,

00:16 where I ran an experimental intervention to understand gender gaps in STEM.

00:21 So field specialization and college major

00:24 choice are hugely consequential decisions,

00:27 and understanding how people make those choices,

00:31 and in particular how it varies across groups is really important for policy.

00:36 We hear a lot about shortages of STEM workers and STEM skills,

00:40 with some policymakers calling for an increase in the number of STEM graduates.

00:45 Um,

00:45 in addition to overall shortages,

00:48 women are underrepresented in many quantitative fields such as economics,

00:53 computer science,

00:54 and engineering.

00:55 Even though more than half of all bachelor's degrees go to women,

00:59 there are only 1/3 of the degrees in economics

01:02 and about 10% in computer science and engineering.

01:05 Um,

01:06 and this gender gap in STEM education

01:08 has implications for both equity and efficiency.

01:12 So the fields with the fewest women also tend to be the highest paying ones,

01:17 so differences in field specialization contribute to the gender pay gap.

01:21 And furthermore,

01:22 if we're in a world where people should specialize according

01:25 to comparative advantage or the thing they're relatively best at,

01:29 then removing barriers that are preventing them

01:32 from doing that would increase overall efficiency.

01:36 So we know that differences in mathematical aptitude or performance

01:40 explain very little of the gender gap in specialization,

01:44 but gender differences in beliefs about performance,

01:47 conditional on actual performance,

01:49 may be partially responsible for gender differences in educational choices.

01:54 And this is supported by observational evidence from

01:57 sociologists,

01:58 psychologists,

01:59 economists,

02:00 education researchers,

02:01 um,

02:01 that men and women have different beliefs

02:04 about their own performance or competence,

02:06 um,

02:07 as well as theory from economics and others that these beliefs should matter.

02:12 But there's been really limited causal evidence on this mechanism.

02:17 So in this work,

02:18 this is the first experimental evidence isolating

02:21 the effect of beliefs about relative performance

02:25 on field specialization in college.

02:27 So the setting I use to do this is 5700 undergraduate students

02:32 in seven large introductory STEM courses here at the University of Michigan.

02:38 Um,

02:38 I'm gonna start by documenting

02:41 gender differences in beliefs about relative performance.

02:44 So just at a descriptive level,

02:46 what do these patterns look like.

02:47 Um,

02:48 in my experimental intervention,

02:51 I provide students with information about

02:53 their relative performance in STEM courses,

02:56 and then I measure how both their

02:58 beliefs and their actual academic behavior change.

03:03 To preview what I find,

03:04 um,

03:05 absent any intervention,

03:07 men are more overconfident than women.

03:10 Um,

03:10 and then I find that my,

03:12 uh,

03:12 my intervention of information provision

03:14 induces substantial belief updating,

03:17 and the gender gaps in biased beliefs

03:20 close by between 3.5,

03:22 depending on the measure.

03:24 And then,

03:24 finally,

03:25 the information provision shrinks gender gaps in STEM persistence by about 10%.

03:32 And the changes appear to be driven by lower performing,

03:35 overconfident men,

03:37 updating their beliefs downwards and taking less STEM.

03:41 So,

03:41 I interpret all of this as evidence that

03:44 beliefs about relative performance do matter for field specialization

03:49 and STEM persistence,

03:50 but it mainly operates through male

03:52 overconfidence rather than female underconfidence.

03:58 So the setting I use to do this,

04:00 I study undergraduate students across 7 large introductory STEM courses

04:05 at the University of Michigan,

04:07 and I did this in the fall of 2019.

04:09 My sample is about 55,700 students,

04:12 and they're spread across introductory.

04:15 Biology,

04:16 chemistry,

04:17 computer science,

04:18 economics,

04:19 engineering,

04:19 physics,

04:20 and statistics courses.

04:21 Um,

04:22 just as a note,

04:22 um,

04:23 when I refer to STEM throughout,

04:24 um,

04:25 I do include economics,

04:26 uh,

04:27 as,

04:27 as a sort of

04:28 similar quantitative male dominated field,

04:30 although definitions of STEM vary a bit.

04:33 Um,

04:34 this,

04:34 this sample is just slightly under 50% women,

04:37 um,

04:38 because these are introductory courses,

04:40 the majority are in their first or second year of college.

04:44 Um,

04:44 and this is a,

04:45 this is a population that's already very interested in STEM.

04:49 So I surveyed them at the beginning of the semester,

04:52 and

04:53 72% of the women,

04:54 85% of the men are considering or intending a STEM major

04:59 at the beginning of the course.

05:00 Um,

05:01 so you can mostly think of this,

05:02 um,

05:03 project as as targeting STEM persistence with students who

05:06 are already invested in STEM in some way.

05:10 Um,

05:10 so I use two main sources of data in this project.

05:14 The first is university administrative data from the University of Michigan.

05:19 So this is

05:21 background and transcript data that allows me

05:23 to measure a number of pre-treatment,

05:26 demographic,

05:27 and academic characteristics about these students,

05:31 including high school performance,

05:33 socioeconomic status,

05:35 and,

05:36 and prior performance in college.

05:39 Um,

05:40 I,

05:40 I also have their course taking outcomes one semester after the intervention.

05:45 So,

05:46 uh,

05:46 that's going to be my,

05:47 my primary behavioral measure is the,

05:50 the number of STEM courses or credits that

05:52 students take following the semester of the intervention.

05:55 Um,

05:55 and because this is administrative data,

05:58 uh,

05:58 I have universal coverage.

05:59 The only reason a student wouldn't appear in this data is if they,

06:03 uh,

06:04 disenroll from the University of Michigan.

06:07 I combine that with survey data,

06:09 uh,

06:10 so I,

06:10 I survey students to get information about their

06:13 beliefs,

06:14 about their own performance,

06:15 and the performance of others.

06:18 Um,

06:18 and I do these surveys at two points in time.

06:21 So,

06:21 the first is the

06:22 Beginning of the semester,

06:24 or before the intervention occurs,

06:26 um,

06:26 and then I do a second set of surveys at the end of the semester,

06:30 or after the intervention occurs,

06:32 um,

06:32 and the,

06:33 the end of semester is going to be the source for the,

06:35 um,

06:35 the beliefs outcomes for this project.

06:38 Um,

06:39 the response rate for these,

06:40 the,

06:40 the end of term survey was about 50%.

06:43 Uh,

06:44 the incentives offered for these,

06:46 for these surveys varied by course and was

06:49 sort of at the discretion of the instructor,

06:51 whether they,

06:51 they incentivized with with course credit,

06:54 um,

06:55 or extra credit for the,

06:56 for the surveys.

07:00 So,

07:01 sort of before I,

07:02 I get into my intervention and what it did,

07:05 I want to start with some motivating evidence for this hypothesis

07:08 that men and women have different beliefs about their relative performance,

07:13 and that those differences might be responsible

07:16 for some of the gender differences in

07:18 enrollment and persistence in STEM.

07:22 So,

07:22 I measure beliefs about relative performance,

07:25 uh,

07:26 using these student surveys that I mentioned,

07:28 and again,

07:29 I measure them at two points in time.

07:31 So,

07:31 I measure,

07:32 uh,

07:33 students' beliefs at the beginning of the semester,

07:35 uh,

07:35 which is around September,

07:37 October,

07:38 um,

07:38 and then again at the end of the semester.

07:40 Which is in December,

07:41 right before their final exams.

07:44 Um,

07:45 and just to,

07:46 to specify in this section,

07:47 when I'm,

07:47 I'm looking at descriptive patterns and beliefs,

07:50 I'm limiting to control students only to see what students'

07:54 beliefs look like and how they change over time,

07:57 in the absence of any intervention.

07:59 Um,

08:00 I also am limiting to students who responded to both of the surveys,

08:04 uh,

08:04 so that I'm not confounding changes over time with changes in the sample.

08:11 So,

08:11 I have two different measures of beliefs about relative performance,

08:16 and the first is the students' prediction of their final percentile rank

08:21 in the course.

08:22 So,

08:23 specifically on the on the survey,

08:25 I asked them,

08:26 or they tell me,

08:28 In terms of my final grade,

08:29 I expect I will do better than 1% of my classmates in

08:34 Econ 101,

08:35 for example,

08:37 um,

08:37 and I can compare their answer to their actual realized

08:41 performance or percentile rank in the course.

08:45 So I'm going to show

08:47 a few graphs that look like this,

08:49 and so what you're seeing here on the on the x

08:52 axis we have the students realized percentile or their actual performance

08:57 in the course,

08:58 where 0 would sort of be the lowest performing and 100 is the top performers,

09:04 and I plot that on the y axis against what they predict their percentile to be

09:09 at the beginning of the semester.

09:11 And so if a student was perfectly accurate in their prediction,

09:15 they would fall along this gray 45 degree line.

09:18 Um,

09:18 a student who was overpredicting,

09:20 uh,

09:20 who sort of thinks they'll do better than they actually do,

09:23 would fall above the line,

09:25 uh,

09:25 and those who are underpredicting

09:27 would fall below the line.

09:29 Um,

09:30 and I'm,

09:30 I'm gonna graph this as a,

09:31 as a bin scatter plot,

09:33 where I,

09:33 I group students and,

09:34 and plot the within bin means.

09:39 So,

09:40 uh,

09:41 this is what it looks like when I,

09:42 when I plot students the beginning of semester beliefs,

09:45 and there's sort of two interesting things in this picture.

09:48 So,

09:49 The first is that the,

09:50 the slope through these points is fairly flat.

09:53 So that means,

09:54 you know,

09:55 without much information,

09:56 this is consistent with students mostly just guessing about how they'll do.

10:01 Um,

10:01 and that makes sense,

10:02 because again,

10:03 uh,

10:03 they answer this early in the semester,

10:05 they haven't received many signals of their performance,

10:08 uh,

10:08 90% of them answer this question before they've even taken an exam in the course.

10:13 Um.

10:14 And so sort of the,

10:16 the second interesting point here is that most of

10:18 these points fall above the 45 degree line,

10:22 which means that the average student is overpredicting how they'll do.

10:27 Um,

10:27 so,

10:28 you know,

10:28 sort of in sum,

10:29 everybody is roughly guessing,

10:31 and everyone is guessing that they'll do a little above average.

10:36 Um,

10:37 but of course,

10:37 I'm interested in gender differences.

10:40 So,

10:40 when I break this down by gender,

10:43 when I'm showing the,

10:44 the men here are the open blue circles,

10:46 and the women are the,

10:46 the solid orange.

10:49 The,

10:49 the really striking thing here is

10:51 that throughout the true performance distribution,

10:54 the men think they'll do better than the women on average.

10:59 Uh,

10:59 and so I can,

11:00 I can formally test this by,

11:02 um,

11:02 estimating,

11:03 uh,

11:03 regression through these points with an interaction of,

11:07 um,

11:07 of,

11:08 of gender and the,

11:09 uh,

11:10 predicted percentile.

11:12 And I,

11:13 I find that,

11:14 uh,

11:14 the difference in intercept for men and women is 8 percentiles,

11:18 with no difference in the slope.

11:20 And so that means for

11:22 a man and a woman with the same actual performance in a STEM course,

11:26 a man thinks he'll do 8th percentile ranks better than the woman does.

11:33 And so this is what students predict at the beginning of the semester,

11:37 right,

11:37 um,

11:38 without much information,

11:39 but we would expect that as students learn about their performance

11:43 over the semester,

11:44 they would update their beliefs.

11:47 And in fact that's exactly what happens,

11:49 right?

11:50 So I'm now plotting on the y axis,

11:52 the end of semester beliefs.

11:54 So students are now predicting their performance in December,

11:57 right before their final exams when they have most of their grades.

12:02 And so the sort of key thing compared to the previous graph is

12:06 that all of these points have shifted closer to the 45 degree line,

12:11 meaning that on average students have become more accurate in their predictions.

12:16 But there is still this gender gap that

12:19 uh remains among the lower performing students,

12:22 with lower performing men particularly overconfident.

12:26 And so this difference in intercept is still 8 percentile ranks.

12:31 Um,

12:31 I,

12:31 I can't detect a difference in the slopes,

12:33 although this is a,

12:34 a somewhat um limited part of my sample.

12:38 Um,

12:39 maybe I'll,

12:39 since we've,

12:39 we're,

12:40 um,

12:41 after our 1015 minutes,

12:42 um,

12:42 maybe I'll pause and,

12:43 and see if anyone has any questions or hand raises,

12:45 um,

12:46 that we should

12:48 address.

12:49 OK,

12:49 sure.

12:50 Uh,

12:50 there is uh David.

12:51 David has a question.

12:54 Hi,

12:54 hi,

12:54 I,

12:54 I wanted to ask about the way you were measuring expectations here.

12:58 I mean,

12:59 we know,

13:00 you,

13:00 you know,

13:00 people use fifty-fifty a lot when they're uncertain.

13:03 We know that when you ask what do you expect,

13:06 some people are saying mean,

13:07 some people are saying medium,

13:08 some people are

13:09 saying mode,

13:11 you know,

13:12 you,

13:12 you're not,

13:13 did you

13:13 sort of,

13:14 why did you ask it this way,

13:15 or did you think about sort of

13:18 incorporating uncertainty on estimates when you're

13:20 asking that question and you know,

13:22 should we think about uncertainty in interpreting this figure?

13:26 Yeah,

13:26 so that's,

13:26 that's a fair question,

13:27 you know,

13:28 sometimes when

13:29 behavioral economists,

13:30 um,

13:31 elicit beliefs in,

13:32 in a lab setting,

13:33 they're able to,

13:34 you know,

13:34 ask more complicated questions to get at the full distribution of beliefs.

13:38 Right,

13:38 like what's the percent chance

13:40 you get this grade versus this grade versus this grade,

13:42 um,

13:43 you know,

13:43 in,

13:43 in a field setting,

13:44 I was,

13:44 I was limited in the number of questions I could ask,

13:48 um,

13:48 and so I,

13:49 I focused on,

13:50 uh,

13:50 you know,

13:51 this item and another item I'll go over in a bit,

13:53 um,

13:54 about,

13:55 uh,

13:55 just sort of a simple prediction,

13:57 um,

13:58 and.

13:59 And it is sort of,

14:00 you know,

14:00 how students interpret it,

14:01 um,

14:02 you know,

14:03 it was true in this,

14:04 this previous slide that there were a lot of

14:07 beliefs sort of clustered around

14:09 50,

14:09 right?

14:09 Everyone is,

14:10 is sort of guessing that they'll do about average,

14:13 um,

14:13 which without a lot of information is,

14:16 is a,

14:16 you know,

14:17 a fairly

14:18 sort of expected thing.

14:20 Um,

14:21 but I,

14:21 and I have,

14:22 I have one more measure of beliefs that I'll,

14:23 I'll go over,

14:24 um,

14:25 and let me know if,

14:26 did that answer your question.

14:28 OK.

14:30 OK,

14:30 good.

14:30 Harris,

14:31 please go ahead.

14:33 Hi.

14:33 So,

14:34 uh,

14:34 you mentioned that uh the response rate

14:37 to the end line was about 50%.

14:40 And I was wondering if there was a risk of you know selection

14:43 in your analysis and how you may have dealt with that.

14:47 Yeah,

14:47 so,

14:48 so,

14:48 yes,

14:48 with,

14:48 you know,

14:49 with,

14:49 with survey response,

14:50 um,

14:51 it's not,

14:51 it's not full response,

14:52 and there are certain groups that are sort of

14:54 more or less likely to respond to the survey.

14:57 I,

14:57 I do a number of things to address that,

15:00 um,

15:00 and in particular when I,

15:01 when I start talking about the uh the outcomes at the end.

15:06 So

15:07 one thing is that my behavioral outcomes that use administrative data.

15:11 There's no missingness in those.

15:13 So sort of any behavioral responses you see,

15:16 you can,

15:16 you can think of as,

15:17 you know,

15:18 that that captures the whole population.

15:20 Um,

15:20 there is sort of a question about

15:22 translating between the,

15:23 the survey beliefs and the,

15:25 the behavioral responses,

15:26 because those,

15:27 those might not actually quite be the same groups.

15:29 Um,

15:29 I,

15:29 I do a couple of things to address that in the paper.

15:32 Um,

15:33 I,

15:33 I do a,

15:34 um,

15:34 an in.

15:36 Uh,

15:36 propensity weighting where I,

15:37 I wait the survey respondents to look like the full sample.

15:41 Um,

15:41 I also do something sort of going the other way where I only look at

15:45 administrative data for people for whom I also have,

15:48 um,

15:48 survey responses,

15:49 and,

15:50 you know,

15:50 the conclusions generally look the same with with these different methods,

15:53 um,

15:54 but,

15:55 uh,

15:56 and,

15:56 but,

15:56 but you can sort of see here,

15:57 um,

15:58 in these pictures,

15:58 at least.

16:00 It,

16:00 it does span the distribution of,

16:02 of performance in the class,

16:04 um,

16:05 for example,

16:05 so I am holding that constant in all of these.

16:11 OK,

16:11 good.

16:12 So there are questions from Clement and then Pierre,

16:14 and uh then you can continue,

16:16 Stephanie.

16:19 Hey,

16:20 um,

16:21 my question is,

16:21 um,

16:22 so it could be that

16:24 in,

16:25 um,

16:27 that you failed,

16:29 for example,

16:29 and that,

16:30 that could be different from your true beliefs.

16:33 So it could be that men actually

16:35 have a uh a certain belief but then when they respond,

16:38 they,

16:39 they're more likely to,

16:40 to try to project uh uh a better image uh outwards.

16:44 So is there any way to distinguish between those two things?

16:48 Um,

16:49 so you,

16:49 you cut out a little bit at the beginning,

16:51 but I,

16:52 I think you're,

16:53 you're asking like,

16:54 is this capturing their true belief?

16:56 Is that sort of your question?

16:58 Yeah,

16:58 so,

16:59 so,

17:00 you know,

17:00 I,

17:00 I ask this in a sort of simple way on a survey.

17:03 Um,

17:03 it's,

17:03 it's not necessarily an incentive compatible measure,

17:06 which is,

17:07 you know,

17:07 I think fair to point out,

17:09 um.

17:10 One,

17:10 you know,

17:10 if,

17:10 if I had actually made

17:13 sort of incentivize these students to give an accurate answer,

17:16 that itself would have been a treatment,

17:17 um,

17:18 which would have interfered with,

17:19 with my treatment.

17:20 Um,

17:21 but as I,

17:21 you know,

17:22 I think as I'll,

17:22 as I'll sort of show,

17:23 even if there's some

17:25 error in this and some,

17:26 you know,

17:27 possibly interesting things about,

17:28 um,

17:29 you know,

17:30 projecting or Sort of projecting overconfidence,

17:32 even if you're not overconfident,

17:34 um,

17:35 these measures are correlated with behavior,

17:38 um,

17:38 both in the control group and then the

17:40 experiment by changing these beliefs changes some behavior.

17:43 Um,

17:44 so,

17:44 so even if they're not a perfect

17:46 accurate representation of beliefs,

17:48 I think they're capturing something real and something interesting,

17:51 um,

17:51 in terms of gender differences.

17:56 Pierre,

17:56 go ahead,

17:57 please.

17:58 Thanks,

17:58 Stephanie.

17:59 Uh,

17:59 a quick question,

18:00 is there,

18:00 it's hard to tell because you've been the data,

18:02 is there,

18:03 uh,

18:04 more variance among men and women?

18:06 And if so,

18:07 you know,

18:07 does that have some implication that is,

18:09 you know,

18:09 there's a few men who are super overconfident,

18:11 but on average they're the same.

18:13 Yeah,

18:14 you know,

18:14 I,

18:14 I did look at this a while back to see,

18:17 to see it,

18:17 it,

18:18 it that this question,

18:18 like,

18:19 are there sort of,

18:19 um,

18:20 differences in,

18:21 in variants,

18:22 and,

18:22 you know,

18:22 some other work suggests that men are in survey response in

18:25 general tend to be more at extremes compared to women.

18:28 Um,

18:29 I didn't quite find that,

18:30 but I don't think,

18:31 I'm not sure I quite have the,

18:32 the power to,

18:33 to detect that.

18:34 Um,

18:35 and I looked early on if it sort of changed,

18:37 if the treatment changed the variant,

18:39 um,

18:40 but I don't really find much there.

18:47 OK,

18:47 um,

18:48 all right,

18:48 I'll keep going.

18:49 So,

18:50 uh,

18:50 so I have 22 different measures of relative performance beliefs,

18:54 the one I just showed you about percentile rank,

18:57 um,

18:57 the second measure is the estimate of the median

19:01 course grade for students who major in STEM.

19:04 So

19:05 specifically on the survey,

19:06 I asked these students,

19:08 when thinking just about students who declare a major in math,

19:13 science,

19:14 engineering,

19:14 or economics,

19:15 what do you think was their median grade in this course?

19:19 Um,

19:19 and so this is,

19:20 you know,

19:20 it's getting at a couple of things.

19:21 It's getting at sort of the The difficulty of the class

19:24 and it's really getting at the selection into STEM,

19:27 right,

19:27 sort of where students think the bar is in order to continue in the field

19:33 and like the previous measure,

19:34 I can compare this

19:36 to the actual median that I can calculate using historical transcript data.

19:41 So just to give an example,

19:43 uh,

19:44 suppose the,

19:44 the true median in Econ 101 for students who go on to major in STEM

19:49 is a B+,

19:51 but I think it's an.

19:53 That means I'm overestimating the median,

19:55 or I,

19:56 I think the bar,

19:57 I think this median for STEM majors is higher than it actually is.

20:02 So,

20:03 again,

20:03 I see very striking gender differences in this measure,

20:07 uh,

20:07 in a way that suggests men are are setting the

20:10 bar for STEM majors lower than it actually is,

20:13 and women are setting it too high.

20:15 So,

20:15 specifically,

20:16 uh,

20:16 women are 17% points more likely to

20:20 overestimate this median grade,

20:22 um,

20:23 and men are 11 points more likely to underestimate it.

20:27 Um,

20:27 and to,

20:28 to sort of,

20:28 you know,

20:29 make,

20:29 make clear what this means,

20:30 this is working in the direction of

20:32 men being relatively overconfident and women being

20:36 relatively underconfident,

20:38 right?

20:38 So,

20:39 if the men think that others are doing worse than they actually are,

20:43 that means they seem better in comparison.

20:46 Um,

20:46 and the opposite is true for women.

20:49 Um,

20:49 so,

20:50 again,

20:50 these are what students think,

20:51 uh,

20:51 at the beginning of the semester,

20:53 and these proportions look very,

20:55 very similar when I ask them again at the end of the semester,

20:58 um,

20:59 which we would expect because this is not a piece

21:01 of information that is sort of readily available to them.

21:06 Um,

21:07 so I just sort of showed you these raw gender differences in beliefs,

21:11 um,

21:12 and,

21:12 you know,

21:12 we're sort of interested in how much do they matter

21:14 or are they actually affecting behavior in some way.

21:18 Um,

21:18 so,

21:19 I can also see among the same set of students in the control group,

21:22 the,

21:23 the gender gap in STEM credits in the subsequent semester is about 2 credits,

21:28 um,

21:28 and that,

21:29 that's,

21:29 that's saying that men in my sample take about half of a STEM course

21:33 more than women in the next semester.

21:37 Um,

21:37 so I,

21:37 I go into more detail in the paper,

21:40 and I'm happy to talk more about how I do this,

21:42 but I do a correlational exercise,

21:44 essentially a decomposition,

21:46 where I want to see how much of that credit gap,

21:49 um,

21:50 among the control students is explained by all of

21:53 the characteristics I can measure in the data.

21:56 Um,

21:57 so,

21:57 you know,

21:57 their,

21:58 their prior high school performance,

21:59 their,

22:00 um,

22:01 their high school GPA,

22:02 their socioeconomic background,

22:04 all of that.

22:05 And I find that even controlling for

22:07 a bunch of academic and demographic characteristics,

22:11 these two types of performance beliefs I just showed

22:15 account for about 7% of the gap in STEM course taking.

22:19 Um,

22:20 and just to sort of contextualize,

22:22 you know,

22:22 sort of that magnitude,

22:23 that's the same amount that is explained by,

22:26 uh,

22:26 the students' math skills when they start college

22:28 or their performance on a math placement exam.

22:32 Um,

22:32 and,

22:32 you know,

22:33 I,

22:33 I don't want to sort of

22:34 Uh,

22:36 over overstate or or overinterpret this,

22:38 this precise number because this is a correlation,

22:41 um,

22:42 and this is sort of

22:43 exactly why I,

22:44 I need an experiment,

22:45 but it's sort of suggestive.

22:47 Um,

22:48 right,

22:48 so I've,

22:48 I've showed that these,

22:50 these different measures of beliefs about relative performance,

22:53 first of all,

22:54 they differ significantly by gender,

22:56 in a way that suggests overconfidence among men.

23:00 Uh,

23:01 these beliefs change over time,

23:02 just naturally as students learn about themselves,

23:06 and they are at least correlationally related to behavior.

23:11 Uh,

23:12 but,

23:12 again,

23:13 so all of this so far is,

23:14 is evidence that beliefs matter for behavior,

23:17 and they might be responsible for part of the gender gap in STEM persistence,

23:22 but it's,

23:22 it's just been descriptive.

23:24 Um,

23:24 so,

23:25 to get at the causal relationship between beliefs and academic behavior,

23:29 I implement an intervention where I attempt

23:31 to exogenously vary beliefs about relative performance.

23:36 Um,

23:37 are there any questions before I,

23:39 uh,

23:41 jump into the,

23:41 the intervention?

23:43 Yeah,

23:43 so there,

23:43 there were two from the previous Q&A.

23:45 Theresa,

23:46 can you start,

23:46 please,

23:47 and then.

23:48 Yeah,

23:48 sure.

23:48 Um,

23:49 I was just wondering if you could,

23:50 uh,

23:51 help provide a bit more context for this,

23:53 this research in,

23:54 in terms of the,

23:55 uh,

23:55 social psychology and,

23:56 and sociology literature.

23:58 This is,

23:58 I feel like this has been investigated elsewhere,

24:00 and I'd,

24:01 I'd like to know,

24:02 uh,

24:02 you know,

24:02 what the,

24:03 what the,

24:03 uh,

24:04 contribution of this is on top of that.

24:06 Yes,

24:07 absolutely.

24:07 Um,

24:08 so I,

24:08 I alluded to this very quickly.

24:10 I try not to,

24:11 to dwell on the literature,

24:12 but

24:13 you're completely right that

24:14 researchers across fields,

24:15 so not just economists,

24:17 but,

24:17 um,

24:17 sociologists,

24:18 education researchers,

24:19 um,

24:20 social,

24:20 social psychologists,

24:21 they've,

24:22 they've documented a lot of what I just showed you.

24:24 So essentially descriptive differences

24:27 in,

24:27 in beliefs between men and women about their,

24:29 their confidence or their sort of beliefs about their.

24:32 Competence in in performance,

24:34 um,

24:35 as well as as correlational evidence that,

24:38 right,

24:38 these things are sort of associated with,

24:40 with major choice and,

24:41 and hypothesize that that these could explain gender gaps,

24:45 um,

24:46 but there,

24:46 there hasn't really been any causal evidence on this.

24:49 They're just sort of um relying on

24:51 the correlational relationship between beliefs and behavior.

24:55 Um,

24:55 so that's,

24:55 that's where my work comes in.

24:59 But I will,

24:59 I will say that my,

25:00 you know,

25:00 it's the,

25:01 the descriptive differences I've shown are,

25:02 are,

25:02 are consistent with,

25:04 with prior work about gender differences,

25:06 um,

25:07 and,

25:08 uh,

25:08 particularly in,

25:09 in male dominated or sort of male type domains.

25:16 OK,

25:17 thank you.

25:17 Uh,

25:18 can you please go ahead.

25:19 Yes,

25:20 uh,

25:20 this is key.

25:21 My question is about selection bias here.

25:23 You kind of address it just now in the,

25:26 uh,

25:26 in the slides later,

25:27 but

25:27 I just want to clarify this.

25:30 You,

25:30 uh,

25:30 the center that you

25:32 selected are those

25:33 students that already enrolled in those STEM

25:37 courses.

25:38 You center them

25:39 at the beginning

25:41 of the semester after they have enrolled in those courses,

25:44 and then you censor them again.

25:46 Yeah,

25:46 you survey them again at the end of the course.

25:48 Am I right?

25:50 So,

25:50 um,

25:51 that the,

25:51 the quality,

25:52 Lilith,

25:53 could you hear that better?

25:54 I,

25:54 it was a bit fuzzy,

25:55 um.

25:57 Oh.

25:59 Or,

25:59 or do you want to ask your question again,

26:01 Kei?

26:02 OK.

26:03 Uh,

26:03 my question is

26:04 about selection bias.

26:07 The center you have are the students have already confirmed enrolled in those

26:12 1 courses.

26:14 Not the general

26:16 population of men and women

26:18 in University of Michigan.

26:20 So there is already a selection bias that these are the students that show interest

26:25 in continuing STEM courses.

26:28 I see.

26:28 Yeah,

26:28 so yes,

26:29 thank you for,

26:29 for repeating and,

26:30 and sorry,

26:31 I was having some,

26:31 some trouble hearing you,

26:32 um.

26:33 Yes,

26:33 so you're,

26:34 you're correct that,

26:35 as I mentioned,

26:36 you know,

26:36 my,

26:36 my sample are students who are in these intro STEM courses,

26:39 so they've already

26:40 revealed some interest in,

26:42 in the field,

26:43 um,

26:44 you know,

26:44 in terms of,

26:46 of internal validity,

26:47 um,

26:47 I am,

26:48 I am randomizing within these courses where they've already enrolled.

26:51 So when I'm getting it at the causal effect of my intervention,

26:54 that's,

26:55 that's sort of not an issue,

26:56 um,

26:56 but,

26:57 but there is a sort of generalizable question.

26:59 Uh,

27:00 about external validity and,

27:01 and sort of who do these students represent and

27:04 what would happen if we targeted students earlier on,

27:07 um,

27:07 you know,

27:08 this is the point of the,

27:09 of the pipeline that that I'm able to explore,

27:12 um,

27:12 but I think there are some really interesting questions,

27:15 and I'll I'll maybe get to this more at the end about

27:17 where these beliefs.

27:18 From and and sort of how they might play out even earlier.

27:22 Um,

27:22 I,

27:22 I think it's important to think about,

27:24 uh,

27:25 you know,

27:25 maybe targeting beliefs before we get to this point,

27:28 um,

27:28 but that's,

27:29 um,

27:29 the,

27:29 the sample I have is you're,

27:31 you're totally right to be thinking of it as

27:33 as people who have already selected into at least

27:35 sort of an intro level of STEM.

27:45 Just a reminder that once you have done your or complete your question,

27:49 you can load your hands.

27:50 I'm not sure if Tristan or Clement.

27:52 No.

27:54 OK,

27:55 yeah,

27:55 you can continue to dance no more hands.

27:57 Great,

27:58 um,

27:59 OK,

27:59 so that was all sort of descriptive evidence.

28:02 So,

28:02 so what do I actually do?

28:03 What is my experimental intervention?

28:06 So my intervention involves two different treatment arms.

28:10 So in the first arm,

28:12 I provide students with

28:14 information about their relative performance

28:16 in these STEM courses,

28:18 and I'll,

28:19 uh,

28:20 I tell them how they're doing.

28:21 Relative to their classmates as well as the performance of a typical STEM major,

28:27 and I'll show you just in a second what what exactly this information looks like,

28:31 but it

28:33 corresponds to the two different measures that I previously went over.

28:37 Um,

28:38 I have a second treatment arm,

28:40 which only a subset of higher performing students

28:43 are eligible for.

28:45 And so in the second treatment,

28:46 they get all of the same information,

28:49 but it's,

28:49 it's paired with an explicit encouragement message to

28:53 consider or stay in a STEM major.

28:55 So the same information,

28:57 but,

28:57 but framed.

28:58 In a more positive way,

29:00 and,

29:00 and this arm was designed based on sort

29:02 of some general behavioral economic research suggesting that

29:06 additional sort of nudges or salients might be necessary to move behavior,

29:10 um,

29:11 as well as some some work suggesting that encouragement might be

29:14 especially important for women

29:16 in these settings.

29:19 So,

29:20 what do these messages look like?

29:21 Here,

29:21 here's an example of the the just pure informational message.

29:25 Uh,

29:25 this is a sample for,

29:26 for a student in the biology course.

29:29 Um,

29:30 and to sort of go through what this looks like,

29:32 uh,

29:32 the sort of key information in the intervention is,

29:36 is contained here in this histogram,

29:38 in this picture.

29:39 So they see a number of things.

29:40 So,

29:41 first,

29:41 they see the sort of full distribution of course scores,

29:45 um,

29:45 as of the middle of the semester,

29:48 um,

29:48 and at the point when they get this,

29:50 uh,

29:51 everyone has taken at least one exam,

29:53 many have taken 2 in their courses,

29:55 as well as any other assignments that they've completed.

29:58 Um,

29:59 and so I,

30:00 within this distribution,

30:01 I highlight

30:02 their percentile,

30:04 so sort of where they fall,

30:05 uh,

30:06 you're at the whatever it is percentile,

30:08 um,

30:09 as well as the second piece of information,

30:11 which is the,

30:11 the median grade for STEM majors.

30:14 Um,

30:15 and so that that information is sort of repeated again,

30:18 both their,

30:18 their raw score and their percentile.

30:21 Um,

30:23 Uh,

30:23 as well as,

30:24 again,

30:24 sort of repeating the information about,

30:26 uh,

30:26 median grades,

30:27 both for STEM majors,

30:29 as well as

30:30 for all students in the class,

30:31 um,

30:31 and for students in that subject to give more context about what grades mean.

30:36 Uh,

30:37 and then the message concludes with some basic information about,

30:40 uh,

30:40 scheduling,

30:41 advising appointments,

30:42 and,

30:43 uh,

30:43 course registration.

30:46 So,

30:46 you,

30:47 you are,

30:47 are likely wondering,

30:48 right,

30:48 I'm,

30:48 I'm giving students information about,

30:50 about their grades,

30:51 about these courses,

30:52 but,

30:53 you know,

30:53 what,

30:53 what do they already know?

30:54 Do they already know all of this?

30:56 Um,

30:56 and so it,

30:57 it varies a bit by the course,

30:58 but in general,

30:59 uh,

31:00 if,

31:00 you know,

31:00 you've ever been,

31:01 been a college student,

31:02 you,

31:03 you know,

31:03 you can,

31:04 you already know.

31:04 Or can sort of very easily see your raw score in the course,

31:08 um,

31:09 but you generally don't know your exact percentile,

31:11 at least in,

31:12 in this setting.

31:13 So

31:14 some of the instructors do provide a distribution of exam grades,

31:18 but,

31:18 but the students aren't able to calculate their exact percentile in the,

31:22 in the full distribution,

31:23 um.

31:25 And in terms of sort of,

31:26 uh,

31:26 the distribution of,

31:27 of grades,

31:28 some of these courses are curved,

31:30 so,

31:30 you know,

31:30 in,

31:31 in theory,

31:31 students sort of know the full distribution of grades.

31:34 Um,

31:34 there's also a way at Michigan to look up historical course medians online,

31:40 um,

31:41 but those are,

31:41 are just sort of overall,

31:43 they're not specific for a certain population like STEM majors,

31:47 um,

31:48 which,

31:49 uh,

31:49 you know,

31:49 could be particularly important as you're thinking

31:51 about whether to continue in SSTEM.

31:53 Um,

31:54 and then I'll just,

31:55 just note,

31:55 even though they have a lot of this information available to them,

31:58 as I showed earlier,

31:59 uh,

32:00 they're,

32:00 they're pretty inaccurate,

32:01 right?

32:01 Even though they have it,

32:02 they,

32:02 they have pretty substantial biases in,

32:04 in both of

32:05 the measures that I'm targeting.

32:07 Um,

32:08 and,

32:08 and even for the information that is,

32:10 is very readily available,

32:11 so just the overall course median,

32:13 um,

32:14 less than a third of students can identify that.

32:17 So there's,

32:17 there's lots of room to,

32:19 to sort of intervene on these beliefs.

32:22 Um,

32:23 and then I have,

32:24 uh,

32:24 the,

32:24 the second treatment arm,

32:25 right,

32:26 which is,

32:27 I'm calling the encouragement treatment message.

32:29 It looks very similar to the informational message.

32:32 It has all of the same content.

32:34 This,

32:34 um,

32:35 this histogram,

32:36 the information

32:37 is exactly the same.

32:39 But the,

32:40 the differences are just that it's,

32:41 it's framed in a much more positive rather than a neutral way.

32:45 Um,

32:45 so rather than telling them,

32:47 here's how you're doing,

32:48 uh,

32:49 I tell them you're performing like a STEM major,

32:52 um,

32:52 and again,

32:52 only high performing students are,

32:54 are eligible for,

32:55 for this treatment arm.

32:59 Um,

32:59 and then finally,

33:01 I do have a,

33:01 a control message where students,

33:04 uh,

33:04 get sort of the same basic information about

33:06 advising,

33:07 um,

33:08 and course registration.

33:09 Um,

33:10 backpacking at Michigan is just what we call course registration,

33:13 um,

33:13 and I just,

33:14 I remind them of their,

33:15 their raw score in the course,

33:17 um,

33:17 and sort of My my reason for sending a

33:20 control message was to hopefully partially mitigate any spillovers.

33:25 So the idea is that,

33:26 you know,

33:27 your friend got a message about their grades,

33:30 and

33:31 you don't wonder,

33:31 oh,

33:32 why didn't I also get a message about my grade in this course.

33:37 Um,

33:38 so the,

33:38 the intervention is delivered a single time.

33:41 Um,

33:41 it's delivered in the middle of the semester,

33:43 and I,

33:43 I targeted it to be around the time that they would be

33:47 registering or making their decisions for courses for the next term.

33:51 Um,

33:52 they received these messages as emails.

33:54 Uh,

33:54 they could also view them online,

33:57 uh,

33:57 through a site that they,

33:58 they used in these courses.

34:00 Um,

34:00 the,

34:01 the majority of students,

34:02 um,

34:02 83% of those who were sent the message did

34:05 view it in some way.

34:07 Um,

34:07 that's likely an upper bound on how many

34:09 actually sort of read it and internalized it,

34:12 but,

34:12 uh,

34:13 that's sort of the best measure I have of,

34:14 of take up.

34:17 Um,

34:17 in terms of the,

34:18 the design,

34:19 uh,

34:19 this is an individual level treatment assignment,

34:22 so I,

34:22 I assign at the student level,

34:24 um,

34:24 but I,

34:24 I stratify within,

34:26 um,

34:27 course,

34:27 gender,

34:28 and performance,

34:29 uh,

34:30 so whether they are

34:31 above or below the median in their course at the,

34:33 at the time of randomization.

34:36 Um,

34:37 and just one sort of

34:38 Technical note,

34:39 um,

34:40 students,

34:40 uh,

34:41 you know,

34:41 this is across 7 courses.

34:42 Students could be taking 2 of these courses,

34:44 possibly 3 at once.

34:46 Um,

34:47 and,

34:47 and so if that was the case,

34:48 I,

34:48 I randomly pick a course for them to be in for the study,

34:52 um,

34:52 and they're,

34:53 they're just not in the study for,

34:54 for the other course,

34:55 um,

34:55 not even a control message,

34:57 uh,

34:57 so that I,

34:58 I can just do this sort of at the unique student level.

35:02 Um,

35:02 and just to sort of summarize the,

35:04 the design given these,

35:05 these two different treatment arms,

35:07 so I have my,

35:07 my full sample of students.

35:10 And based on whether they're

35:11 below or above the course median,

35:13 that determines what what treatment they can get.

35:15 So for the,

35:16 the lower performing students,

35:18 the ones

35:18 below the median in their course,

35:20 um,

35:21 they can be assigned to the control or

35:23 the information only condition with equal probability.

35:26 Um,

35:26 and if,

35:27 if a student is in the top half of their course,

35:30 they can be assigned to the control,

35:32 the information only,

35:33 or the encouragement treatment with also equal probability.

35:39 Um,

35:39 in terms of how I analyze this,

35:41 um,

35:42 I'll just,

35:42 I'll just mention upfront that I did pre-register

35:45 this this project with the AEA's trial registry,

35:49 um,

35:49 and,

35:49 and all of my analyses,

35:51 including the covariates I use,

35:52 were pre-specified,

35:54 um,

35:54 although I do have some exploratory outcomes that I can discuss at the end.

35:58 Um,

35:59 for the most part,

35:59 I'm interested in gender differences,

36:02 and so I'm gonna estimate male and female effects,

36:05 um,

36:06 separately,

36:06 and I,

36:07 I do that with a,

36:07 with a single regression where I interact,

36:10 uh,

36:10 treatment with gender.

36:12 Um,

36:13 I control for a number of pre-treatment,

36:16 demographic,

36:17 and academic characteristics about these students.

36:20 Um,

36:21 I,

36:21 I include strata fixed effects because I,

36:23 I had a stratified design.

36:25 Um,

36:26 and I'm gonna show,

36:27 uh,

36:28 today I'm gonna show intent to treat effects,

36:30 um,

36:31 so the effect of being sent an intervention message.

36:34 So,

36:35 beta one is the intent to treat effect on men.

36:38 Beta 1 plus beta 3 is the intent to treat effect on women.

36:42 Um,

36:43 but as,

36:43 as I mentioned earlier,

36:44 uh,

36:45 83% of students

36:47 viewed the message,

36:48 so,

36:48 um,

36:48 you could,

36:49 you could scale up by,

36:50 uh,

36:50 by 1.2% if you wanted to get a treatment untreated effect.

36:55 Um,

36:56 and just one note,

36:57 so,

36:57 uh,

36:58 this was,

36:58 this is a multi-armed treatment with,

37:00 uh,

37:00 I had the pure information,

37:02 and then the information plus encouragement arm.

37:05 Um,

37:05 in general,

37:06 I don't actually detect any difference between the two treatments,

37:10 either on students' beliefs or on their ultimate behavior.

37:14 Um,

37:15 and so for that reason,

37:16 um,

37:17 and sort of for simplicity and also for for power reasons,

37:20 for most of what I'm presenting today,

37:22 I combine the two treatment arms.

37:24 So you should sort of think of this as a general effect of

37:27 receiving information because all of the treated

37:29 students receive the same set of information.

37:32 Um,

37:33 but for the,

37:34 for the higher performing students,

37:35 the effects are going to be an average of the,

37:38 the information only,

37:39 and the information plus encouragement,

37:41 um,

37:42 but I don't actually distinguish between those two.

37:46 Um,

37:46 OK,

37:47 and this is probably a good,

37:48 another good place to,

37:49 to pause for questions before I get into what I found.

37:53 OK,

37:53 good.

37:53 Um,

37:54 yeah,

37:54 Alvaro,

37:54 go ahead,

37:55 please.

37:57 Hi Stephanie,

37:57 uh,

37:58 quick clarifying question for those of us who

38:00 didn't go to undergrad school in the US.

38:02 So

38:03 here

38:04 they,

38:05 they register for courses in the next semester

38:07 within,

38:08 within like you were saying in mid-semester.

38:10 So because otherwise you,

38:12 if,

38:13 if updating is

38:15 occurring and occurs properly in terms of relative performance,

38:17 then by the end of the semester you sort of know how you do.

38:20 And if you're resistant to courses,

38:21 then,

38:22 then there is no,

38:23 there shouldn't be,

38:24 I mean,

38:24 your confidence effect should,

38:26 should.

38:27 To mitigate,

38:27 right?

38:28 So,

38:29 so here,

38:30 I mean,

38:30 I guess that if there is an issue here,

38:32 it's going,

38:33 it's coming

38:34 from the fact that you're

38:35 registering early when you don't have that much information

38:38 for the,

38:38 you know,

38:39 upcoming,

38:39 upcoming courses.

38:41 Yeah,

38:41 so,

38:42 so thanks for that clarifying question.

38:43 Um,

38:44 yeah,

38:44 so in general at,

38:45 at Michigan at least,

38:47 um,

38:47 you register for the next term's courses

38:50 in the middle of the,

38:52 the current semester,

38:53 so you have

38:54 some information,

38:55 you know,

38:56 a bit limited on,

38:57 on your performance up to that point,

38:59 and that's what they're basing.

39:00 These decisions on,

39:01 um,

39:01 and I,

39:01 I targeted the intervention.

39:03 It was sort of this,

39:04 this trade-off,

39:05 right?

39:05 I did it in the middle of the semester so that there

39:06 was enough information so that this wasn't a completely noisy signal,

39:11 um,

39:12 but I wanted to,

39:12 to target it at the time of decision making so that it really had a chance to,

39:17 to actually affect their behavior.

39:19 I'm very quickly to follow up.

39:20 So,

39:20 and then when you were saying two additional credits at the beginning that you

39:25 STEM credits that you take in,

39:26 in the next semester,

39:27 are you still able to find like actual

39:30 The declaration of major,

39:31 major declaration like later on,

39:33 do you have enough time of observation to,

39:35 to hear us,

39:36 especially for the,

39:37 the,

39:37 the,

39:37 the upcoming results that you're gonna show I guess.

39:39 Yeah,

39:40 so the answer is right now I do not because not enough time has passed.

39:43 I only have sort of one full semester of data on these students,

39:47 um,

39:48 since I,

39:48 I did this about a year ago,

39:50 um,

39:50 I will be getting that soon,

39:52 but,

39:52 uh,

39:52 yeah,

39:52 so eventually I will be able to look at their,

39:55 their major declaration

39:56 and ultimately what they,

39:58 what degree they,

39:59 they graduate with,

40:00 um,

40:00 and I'm,

40:01 I'm of course interested in that,

40:02 but in,

40:02 in the short term,

40:03 um,

40:03 I can look just at their behavior in the next semester.

40:07 Thank you.

40:09 Thank you.

40:10 OK,

40:10 there is one more hands up.

40:12 Uh,

40:12 Anu,

40:13 please.

40:15 Hi,

40:15 Stephanie.

40:16 So you mentioned uh spillover effects.

40:18 Do you actually estimate if there are spillover effects on control classmates

40:23 within the same course or on students' friends maybe in other classes?

40:28 Yeah,

40:28 so I,

40:30 I,

40:30 I cannot directly estimate spillover effects.

40:32 I,

40:32 I think it's a sort of important thing to,

40:34 to think about in this context.

40:37 Um,

40:38 I,

40:39 I,

40:39 I,

40:39 I will maybe mention this at the end again as well,

40:41 but I,

40:42 I had initially planned a second round of this intervention that

40:47 had to get canceled because of COVID and the university shutting down and,

40:50 and all of that.

40:51 Um,

40:51 but in that round,

40:52 I,

40:53 I really want to get at this,

40:54 and I had designed some survey questions to

40:57 ask students about essentially their their networks and their

41:00 peers and their friends and sort of who they

41:02 studied with or discussed grades with in these courses

41:06 so that I would be able to actually see,

41:08 OK,

41:09 do control students who have treated friends change their behavior

41:12 differently than control students who don't have treated friends,

41:15 for example,

41:16 um,

41:16 to actually get it at the spillover.

41:19 But unfortunately,

41:20 I wasn't able to collect that data and I can't directly estimate it.

41:24 Um,

41:24 I think,

41:25 you know,

41:25 if anything spillover in this case would would sort of bias me towards,

41:29 towards zero.

41:33 Thanks.

41:35 OK,

41:36 I don't see the,

41:36 uh,

41:37 yeah,

41:38 Tom,

41:38 please go ahead.

41:39 So follow up on this,

41:41 uh,

41:42 Suppose that

41:43 the extent to which you discuss your grades,

41:45 your performance differs by gender.

41:48 So then your,

41:49 your,

41:49 your issue is that it might,

41:51 it's just not

41:53 some common effect that biases one way or the other.

41:55 It's,

41:55 it can have a,

41:56 a gender differentiated bias.

41:59 So,

41:59 how do you think,

42:00 how do you want to think about this?

42:02 Yeah,

42:03 so,

42:03 again,

42:04 it's,

42:04 it's possible,

42:05 and I think it,

42:06 given my results,

42:07 um,

42:08 you would worry that maybe the women particularly discuss their grades,

42:11 and so that biases me towards,

42:13 towards finding nothing for the women.

42:15 Um,

42:16 uh,

42:16 again,

42:17 I,

42:17 I don't quite have the survey data to,

42:19 to rule that out,

42:20 um,

42:20 although I did do some sort of early focus grouping,

42:24 talking to students,

42:25 um,

42:25 early on to,

42:26 to get a sense.

42:27 Of of exactly this,

42:28 right?

42:28 Like,

42:28 do you talk to each other about grades?

42:30 If you got a message telling you your percentile in the class,

42:34 would you discuss that with your peers?

42:35 And I think the fact that these messages are very personalized,

42:38 um,

42:39 and students view grades generally as a pretty private thing,

42:43 um,

42:44 they're not necessarily likely to just forward an email or,

42:47 or bring it up,

42:48 um,

42:49 but it,

42:50 it is always possible.

42:56 OK,

42:57 good.

42:57 So I don't see more hands up and please don't forget

43:00 to load your hands in when you are with your question.

43:03 Go ahead,

43:03 Stephanie,

43:04 please.

43:05 Great.

43:05 um

43:07 OK,

43:07 so I just sort of went over the the intervention.

43:10 So what did it do?

43:11 Um,

43:12 I'm gonna start sort of,

43:13 uh,

43:14 first set of results is about the effect of the intervention on student beliefs.

43:18 Did it,

43:18 did it actually cause them to update their beliefs in the ways we might expect?

43:22 Um,

43:23 so again,

43:24 I,

43:24 I measure belief outcomes in,

43:27 in two different measures very similar to what I went over before.

43:31 Um,

43:31 so the first is the,

43:33 the error in their own percentile belief where I,

43:36 I subtract their,

43:37 their actual realized percentile from what they predict it will be,

43:42 um,

43:42 and so the outcome is the size of the error,

43:45 um,

43:45 and,

43:46 and just one.

43:47 Slight change from from what I did before.

43:48 I'm,

43:49 I'm now using the um the realized percentile percentile.

43:53 I'm measuring that as their,

43:55 their mid-semester performance because the final performance could

43:59 itself be an outcome that's affected by treatment.

44:02 So I use the the sort of pre-treatment measure of their actual performance.

44:06 Um,

44:07 the,

44:07 the mid-semester performance is also the thing that

44:09 I sort of directly tell them about.

44:11 Um,

44:12 the,

44:13 uh,

44:14 so if,

44:14 if they have a positive error,

44:16 that means they're overpredicting or overconfident in their performance,

44:20 um,

44:21 and a negative error means that they're

44:23 underpredicting or underconfident about their percentile rank.

44:27 Um,

44:28 and then the,

44:28 the second measure is,

44:30 um,

44:30 again,

44:30 this measure of their,

44:31 um,

44:31 how accurate they are about the,

44:33 the median grade for STEM majors,

44:36 um,

44:36 and whether they're overestimating that value.

44:41 So,

44:42 uh,

44:42 I'm going to start with this measure of the error in a student's percentile belief,

44:47 um,

44:47 and,

44:47 and first I'm just showing control means.

44:50 So

44:51 recall from those,

44:52 those scatter plots earlier

44:53 that the average student was somewhat overpredicting,

44:57 uh,

44:57 which is why both of these control means are positive,

45:00 um,

45:00 and men overpredict by more than women,

45:04 right?

45:04 The control mean is larger for men.

45:07 Um,

45:08 but then I estimate the treatment effect on these beliefs,

45:11 um,

45:12 these darker bars are showing the estimated

45:14 treatment effect with a 95% confidence interval.

45:17 Um,

45:18 and so,

45:18 you'll see here,

45:19 um,

45:20 I,

45:20 I find no average effect on this error measure

45:23 for men or women.

45:25 But remember,

45:26 uh,

45:27 people have both positive and negative errors,

45:30 and they can update in either direction.

45:33 Um,

45:34 so these,

45:34 these null effects across sort of all men and all women

45:37 are actually masking some very real changes in their beliefs.

45:41 So,

45:42 I further break this measure down

45:45 by gender,

45:46 but now also by performance.

45:48 So recall that the um

45:51 the lower performing students tended to be overconfident,

45:54 right?

45:55 So the,

45:55 the below median students have positive errors,

45:59 um,

45:59 and especially the lower performing men are especially overconfident.

46:04 Um,

46:05 whereas the higher performing students were somewhat underconfident,

46:08 so they had negative errors,

46:11 um,

46:11 the women were,

46:12 were,

46:12 were slightly more underconfident than the men.

46:16 And so the effect of the treatment is that,

46:18 uh,

46:19 it causes men to update their beliefs in both directions,

46:24 um,

46:25 depending on where they are.

46:26 So the

46:27 lower performing,

46:28 overconfident men

46:30 update downwards,

46:31 um,

46:31 and so this,

46:32 because they were overconfident,

46:33 This,

46:33 this is an accurate,

46:35 uh,

46:35 uh,

46:35 revision of beliefs,

46:37 whereas the higher performing,

46:39 uh,

46:39 somewhat under confident men update upwards,

46:42 um,

46:43 also by about 4 percentiles.

46:45 Um,

46:46 and the,

46:47 the women's beliefs about their percentile

46:49 don't seem to change.

46:51 And so sort of the,

46:53 the results,

46:53 um,

46:54 especially if we,

46:54 if we focus on these,

46:55 these lower performing students here,

46:58 the,

46:58 the change in the gap,

46:59 uh,

47:00 in,

47:00 in the air in this belief

47:02 closes by 4.5 percentiles or more than half.

47:08 Uh,

47:09 so the,

47:10 the,

47:11 the second measure I have is about what they believe the,

47:14 the median grade is for STEM majors.

47:16 And so now I'm looking at the

47:18 proportion of students who are

47:20 underestimating the STEM median.

47:23 Um,

47:23 so,

47:24 you'll recall that men were more likely to underestimate that quantity

47:28 to sort of set the bar lower than it truly was,

47:32 um,

47:32 by about 10% points compared to women.

47:36 The effect of the intervention is that men correctly update that belief,

47:40 um,

47:41 so,

47:41 so they correct that by 5% points,

47:45 uh,

47:45 and the result is that the,

47:46 the gender gap in,

47:47 in this measure,

47:48 um,

47:49 under predicting the median,

47:50 closes by 4% points or

47:53 about a third.

47:55 Um,

47:55 and so,

47:56 so both of these,

47:57 these changes so far,

47:59 uh,

47:59 mean that men are becoming less overconfident

48:03 about their performance relative to others,

48:05 right?

48:05 They,

48:05 they sort of updated about their percentile,

48:07 and they're also updating about how,

48:09 how others are doing.

48:12 Um,

48:12 and then the,

48:13 the sort of flip side of this is the proportion who are

48:16 overestimating the STEM median,

48:18 um,

48:18 and women were much more likely to do that in the control group,

48:22 to,

48:22 to set the bar too high.

48:24 Uh,

48:24 and the result of the intervention is that women correct that belief,

48:28 um,

48:28 by,

48:29 again,

48:29 about 5% points,

48:31 and men don't change,

48:33 and so the,

48:34 the gap closes by about a third.

48:38 So,

48:39 to,

48:39 to just sort of summarize the belief effects so far,

48:43 um,

48:43 I've shown that information provision causes

48:46 men to correct beliefs about themselves,

48:48 um,

48:49 and particularly low performing men correct their overconfidence.

48:53 Um,

48:53 it causes both men and women to correct their beliefs about others,

48:57 um,

48:57 in a way where men are correcting their relative over.

49:00 Confidence,

49:01 women are correcting their relative underconfidence,

49:04 um,

49:04 and so the gender gaps in these beliefs close significantly.

49:07 Um,

49:08 and if it's true that beliefs affect behavior,

49:12 then these changes would lead us to

49:14 expect men to decrease their STEM persistence and

49:18 possibly for women to increase.

49:22 Um,

49:23 So maybe before I,

49:25 I show the,

49:26 the final results,

49:26 um,

49:27 was there another question or two?

49:30 Yeah,

49:30 there are 2.

49:31 Takki,

49:32 go ahead,

49:33 please.

49:35 Um,

49:36 hi,

49:37 Stefani,

49:37 this is Tabaki,

49:38 and,

49:38 uh,

49:39 I was wondering actually,

49:41 so,

49:41 uh,

49:42 I know that,

49:42 uh,

49:43 this is,

49:43 uh,

49:44 these results are,

49:46 uh,

49:47 it's not controlling for selection in this team,

49:49 but I was just

49:50 wondering if,

49:51 uh,

49:52 there is any selection,

49:54 uh,

49:54 if more high ability women relative to more high ability men.

49:58 Select into STEM major and if

50:01 high ability women have

50:03 this,

50:04 uh,

50:04 tendency to underestimate themselves more compared to low ability women,

50:08 if that could change your results,

50:10 I mean,

50:10 uh,

50:11 it would be probably nice to see if this actually exists,

50:13 that more high ability

50:15 women relative to,

50:17 uh,

50:17 more high ability men actually select into STEM

50:20 majors or it's something which doesn't exist,

50:22 then,

50:22 then I guess that wouldn't be too much of a concern.

50:26 Um,

50:28 So,

50:29 you're,

50:29 you're asking,

50:30 um,

50:31 if,

50:32 if it's the case that that the,

50:33 the women who end up in my sample are sort

50:35 of like more prepared than the men because they,

50:38 they have more positive selection?

50:40 Right,

50:41 yes,

50:42 relatively in STEM,

50:43 if more,

50:44 uh more high ability women relative to men are kind of selecting their.

50:48 Um,

50:48 I don't find that.

50:49 I think I took out the.

50:52 Sort of all of the some some more of the descriptive statistics,

50:54 but um in general they have similar performance,

50:58 and if anything,

50:59 the men have slightly higher

51:01 um like math standardized test scores and that sort of thing,

51:04 although the women

51:05 have slightly higher

51:07 overall GPA,

51:09 um,

51:10 but

51:11 But that doesn't quite get it like compared to the,

51:13 the,

51:14 the students who don't select into STEM.

51:16 Um,

51:16 so I haven't,

51:17 I haven't looked a ton on,

51:18 on that selection margin since it's not,

51:20 that's not what I'm,

51:21 I'm targeting,

51:22 um,

51:23 but I think there is some,

51:24 some other evidence that that,

51:25 that could matter and,

51:27 uh,

51:27 and you,

51:27 and you do want to target earlier on.

51:30 Thank you.

51:34 Uh,

51:34 no,

51:34 go ahead,

51:35 please.

51:37 Thanks.

51:39 Is it my turn?

51:41 OK,

51:42 this is key.

51:42 Sorry,

51:43 I have one more clarifying question.

51:45 The intervention,

51:47 uh,

51:47 stated the percentile distribution

51:50 of the students related to the

51:52 STEM major in the past.

51:54 So,

51:55 am I right that the grading policy of the University of Michigan is always on a curve?

52:00 The students

52:01 are all,

52:02 uh,

52:02 there's no like everybody gets A's,

52:05 so the percentile distribution doesn't make sense.

52:08 Or

52:09 they,

52:09 they always go through a curve that

52:11 top 5% get A,

52:14 the next 15% get B,

52:16 the rest get C.

52:17 Uh,

52:18 there,

52:18 there is some kind of underlying distribution that you have in mind

52:22 for the intervention to work.

52:25 Uh,

52:26 yeah,

52:26 so the sort of,

52:27 the short answer is that it,

52:28 it varies by these courses.

52:29 So some of them have a strict curve,

52:31 some of them don't necessarily stick to a curve,

52:34 some of them have a curve that only works in one

52:37 direction,

52:38 right?

52:38 They say like,

52:38 we guarantee at least this many A's and B's.

52:41 Um,

52:42 I,

52:42 I think,

52:43 uh,

52:45 You know,

52:45 that this,

52:46 this information

52:47 is,

52:47 is relative to the course that they are

52:50 in,

52:50 um,

52:51 and,

52:51 and sort of how they,

52:53 uh,

52:53 what the distribution of grades in that course means,

52:55 right?

52:55 I,

52:55 I tell them

52:56 like the the the median grade in bio or in econ 101 for STEM majors,

53:03 so it's,

53:03 it's,

53:03 it's letting them compare

53:05 where they are in that class,

53:07 uh,

53:08 to,

53:08 to STEM majors who also took that class.

53:12 Um,

53:13 I,

53:13 I can't necessarily speak to sort of,

53:15 uh,

53:16 maybe what it would look like in other

53:18 settings.

53:19 That's sort of an interesting question,

53:20 but did that,

53:21 did that get at your question?

53:25 I,

53:25 I,

53:26 I'm just worrying that the distribution that people have

53:29 in mind is not really like a bell shape.

53:32 If professors give grades not in a bell shape,

53:35 then you give,

53:36 you give students information in terms of the

53:38 percentiles may not make too much sense.

53:41 Or do you give information on lesser grades?

53:44 Perhaps that makes more sense,

53:45 but I don't know.

53:46 Yeah,

53:47 so they,

53:47 they sort of get both,

53:48 right?

53:48 Like in,

53:48 in those,

53:49 in those

53:50 pictures,

53:50 if,

53:50 if you are

53:51 here for that,

53:52 right,

53:52 they,

53:53 they get the full distribution of the grades,

53:55 and,

53:55 and generally those look pretty bell shaped in in all of these courses,

53:59 um,

53:59 so they do see the full distribution,

54:01 and then it points out their position in it,

54:04 as well as the,

54:05 the median,

54:06 um,

54:06 to sort of show them,

54:08 you know,

54:08 where,

54:09 where the median falls relative to them.

54:13 OK,

54:14 thank you.

54:17 OK,

54:17 now,

54:18 Ana,

54:18 please.

54:20 Uh,

54:20 so Stephanie,

54:21 there are a number of things that are floating in my head with these results.

54:24 So one is overconfidence a bad thing?

54:28 Uh,

54:29 so here you see that the average student,

54:31 both male and female,

54:32 is overconfident.

54:34 And the only reason there's a gender gap is because

54:37 the female students are less overconfident than the male students.

54:40 The situation would be very different if the gender gap was coming from the fact.

54:44 That the female students were actually underconfident on average and

54:47 the male students were on average overconfident.

54:50 So

54:51 I'm starting to see how,

54:53 you know,

54:53 the,

54:53 the fact that the gender gap now reduces because,

54:56 uh,

54:57 the lower

54:58 uh ability overconfident male students are primarily the ones who are adjusting.

55:02 So I think it's good that underconfident,

55:04 high ability or high performance males.

55:07 Students also adjust upwards,

55:09 but

55:10 nothing is really going on with the female students.

55:12 So I'm just trying to,

55:13 so I'm curious to see what you find on student behavior,

55:16 but

55:17 would you be worried that if

55:19 the overconfident,

55:21 low performing male students actually have some negative impacts on them,

55:26 uh,

55:26 and maybe they would have actually done better if

55:29 they continued to be a little bit overconfident.

55:32 Yeah,

55:33 so I'm,

55:33 I'm gonna maybe defer this question because I have some discussion at the end,

55:37 sort of getting at exactly this,

55:39 sort of what it means for,

55:40 for the women,

55:41 what it means for the men who are discouraged and how we think about welfare,

55:44 um,

55:45 so,

55:46 but,

55:46 but please like raise your hand again.

55:48 If I don't answer your question,

55:50 um,

55:50 but I,

55:50 but I definitely want to discuss this because it's really important in thinking,

55:54 like,

55:54 sort of,

55:55 was this a good intervention?

55:56 Do we want to do this?

55:58 Do we want to discourage certain people from,

56:00 from STEM?

56:00 So,

56:01 I will come back to that.

56:03 Sure,

56:03 thanks.

56:05 OK,

56:05 go ahead,

56:05 Stephanie,

56:06 please.

56:06 Great.

56:07 Um,

56:07 OK,

56:08 so we saw these effects on beliefs that

56:10 based on the way men updated and women updated,

56:14 maybe we would expect men to take less STEM based on,

56:17 on sort of correcting their overconfidence.

56:21 So,

56:21 uh,

56:22 as I,

56:23 as I mentioned earlier,

56:23 I have a,

56:24 I have a short term measure of field specialization or major choice,

56:28 um,

56:29 using the administrative data.

56:31 Um,

56:31 so my primary outcome here is the number of STEM credits

56:34 that a student takes in the semester following the intervention,

56:38 and as more time passes,

56:39 I'll be able to measure their actual major

56:42 declaration and the degree that they receive.

56:45 So here's,

56:46 here's the,

56:47 the control students.

56:48 So I,

56:49 I mentioned briefly earlier that the,

56:51 the gender difference between men and women

56:53 is,

56:53 is about 2 STEM credits,

56:55 or men take about half of a STEM course more than women among the control group.

57:01 The result of the intervention here is that

57:04 men decrease their STEM credits by 0.28 credits,

57:09 um,

57:10 with,

57:10 with no effect for women,

57:12 um,

57:12 and

57:13 again,

57:13 if

57:14 to sort of help,

57:14 help with that,

57:15 that magnitude,

57:16 um,

57:17 you know,

57:17 what does 0.28 credits mean?

57:19 Um,

57:19 so,

57:19 you know,

57:20 a single student can't take 0.28 fewer credits.

57:22 This is an average,

57:24 so this would be equivalent to about 120 men,

57:28 or 7% of the treated men in my sample,

57:33 taking one fewer 4 credit course.

57:36 So you can sort of

57:38 decide if you think that's large or small.

57:39 I consider it fairly big for

57:41 the sort of magnitude of the intervention.

57:44 Um,

57:45 and,

57:45 and women,

57:46 again,

57:46 don't change their behavior,

57:48 and so the result is that the gap in,

57:50 in course taking,

57:51 this two credit gap,

57:53 closes by 10%.

57:58 So,

57:59 right to to sort of summarize the,

58:01 the main results so far.

58:03 I started by showing systematic gender differences in women's

58:07 versus men's beliefs about relative performance in STEM,

58:10 and so I showed that men overpredict their percentile rank by more.

58:15 Um,

58:15 men underestimate how STEM majors do,

58:18 while women overestimate.

58:20 And then I showed that

58:21 a low-cost,

58:22 light touch intervention

58:24 narrowed gender gaps in beliefs by between 1/3 and 15%.

58:28 Um,

58:29 and specifically men updated about their own rank,

58:31 as well as about others in a way that that corrected their overconfidence,

58:36 um,

58:36 and women updated about others,

58:38 um,

58:38 in a way that that updated their underconfidence.

58:41 Um,

58:42 and ultimately the intervention closed gaps in STEM persistence

58:46 by 10%,

58:47 um,

58:47 but this happened because men took 3% fewer STEM credits,

58:52 um,

58:52 and no change for women.

58:55 Um,

58:56 so,

58:56 since we have about 15 minutes left,

58:58 um,

58:58 I,

58:58 I want to get into a little bit of,

59:00 of this other stuff that I,

59:01 that I have built in to,

59:03 to dig into what's going on and get at some of the other questions.

59:06 Um,

59:08 So,

59:08 uh,

59:09 a couple of additional outcomes I look at to,

59:12 to sort of dig into what's going on.

59:15 So,

59:16 um,

59:17 I,

59:17 I showed

59:18 short term effects on

59:20 course taking,

59:21 but I'm ultimately interested in,

59:23 in major choice,

59:24 right?

59:24 I,

59:24 I don't observe it yet,

59:25 but I can speculate,

59:26 um,

59:27 or use the data in a couple of ways to,

59:29 to,

59:30 to guess what might happen.

59:32 So,

59:33 the first I ask students directly in the survey,

59:36 if they plan to major in STEM,

59:38 so their subjective,

59:39 um,

59:40 major intention.

59:41 Um,

59:41 I also do an exercise,

59:43 uh,

59:44 where I,

59:45 uh,

59:45 I predict how much the changes in short-term

59:49 course taking might translate into degree receipt,

59:53 using a,

59:53 a historical sample of students,

59:55 and the,

59:55 the,

59:56 the relationship between course taking and major choice,

59:59 and then the observed changes in course taking I see.

1:00:02 And so both of those exercises,

1:00:05 um,

1:00:05 I find pretty small negative,

1:00:08 statistically insignificant effects on on this possible longer term

1:00:12 major choice

1:00:14 for the for the men who are discouraged.

1:00:16 Um,

1:00:17 so,

1:00:17 you know,

1:00:17 I'll be able to measure this directly with with follow-up data,

1:00:20 um,

1:00:21 but,

1:00:21 but it sort of

1:00:22 implies that longer term effects are likely to be small.

1:00:26 Um,

1:00:27 you,

1:00:27 you also might be wondering,

1:00:28 you know,

1:00:28 whether the intervention had any effect on students' effort or performance.

1:00:33 Um,

1:00:33 there's,

1:00:33 there's a fair amount of,

1:00:34 of literature about

1:00:35 the effect of performance feedback on,

1:00:37 on performance.

1:00:39 Um,

1:00:39 it,

1:00:40 it could actually go either way,

1:00:41 right?

1:00:41 There might be a,

1:00:42 a discouragement effect of receiving bad news,

1:00:45 uh,

1:00:46 but on the other hand,

1:00:47 learning that you're sort of doing worse than you thought could,

1:00:49 could sort of lead you to increase your effort to,

1:00:52 to compensate.

1:00:53 Um.

1:00:54 I,

1:00:55 I don't find any,

1:00:57 any change in performance or effort on,

1:00:59 um,

1:00:59 on either the sort of final

1:01:01 exam grade in the course,

1:01:02 or the,

1:01:03 the final course score.

1:01:06 Um,

1:01:06 and then finally,

1:01:07 right,

1:01:07 I,

1:01:07 I showed that that men take less STEM,

1:01:10 but,

1:01:10 right,

1:01:10 what,

1:01:11 what do they take instead?

1:01:12 Um,

1:01:13 and so,

1:01:14 you know,

1:01:14 we might expect that that certain,

1:01:16 uh,

1:01:17 uh,

1:01:19 They might,

1:01:19 they might shift into other subjects,

1:01:21 um,

1:01:21 and that's sort of important for thinking about welfare.

1:01:24 So,

1:01:25 it,

1:01:25 it seems like

1:01:26 the,

1:01:26 the men may have,

1:01:27 have shifted into psychology and humanities courses instead,

1:01:31 at least in the short term,

1:01:33 um,

1:01:33 but again,

1:01:33 we'll,

1:01:34 we'll sort of,

1:01:34 um,

1:01:35 I'll look at that in the long term as well.

1:01:37 Um,

1:01:38 and then,

1:01:39 uh,

1:01:40 you know,

1:01:40 we,

1:01:40 we also might expect that certain types of students' beliefs and behavior

1:01:45 would be particularly susceptible to an information intervention.

1:01:50 Um,

1:01:50 so I also look at heterogeneity along a number

1:01:52 of dimensions that we might expect to matter.

1:01:56 So just sort of to quickly go over some of those,

1:01:59 um.

1:01:59 I,

1:02:00 I look at the,

1:02:01 uh,

1:02:02 the initial size of the error that students had,

1:02:05 as well as the direction based on that

1:02:08 baseline survey to see if it's,

1:02:10 um,

1:02:10 if,

1:02:11 if it's sort of the students who were the most biased to update the most,

1:02:14 um,

1:02:15 and,

1:02:15 and that is what I find.

1:02:16 So students with,

1:02:17 with the largest initial errors update the most and,

1:02:20 and in the correct direction.

1:02:22 Um,

1:02:22 so I,

1:02:23 I.

1:02:23 interpret that as,

1:02:24 you know,

1:02:24 students are updating in the direction of the truth.

1:02:28 Um,

1:02:29 I also find that,

1:02:30 um,

1:02:31 sort of the younger students,

1:02:32 the first year students,

1:02:33 um,

1:02:34 those who haven't yet declared a major,

1:02:36 and those who are already interested in STEM are

1:02:40 the ones who change their behavior the most.

1:02:42 So

1:02:43 students that we might expect to be on

1:02:44 the margin are the ones whose behavior changes.

1:02:48 Um,

1:02:48 and then finally,

1:02:50 right,

1:02:50 I,

1:02:50 I have this,

1:02:51 this set of STEM courses that,

1:02:52 that actually look fairly different,

1:02:54 right?

1:02:54 We have,

1:02:55 um,

1:02:56 physics and engineering that are extremely male dominated,

1:02:58 and then we have biology,

1:03:00 which is actually female dominated.

1:03:02 Um,

1:03:03 I don't quite have the power to compare,

1:03:05 uh,

1:03:06 across class and across gender,

1:03:08 um,

1:03:09 but to try and get at this,

1:03:10 I,

1:03:11 I look at,

1:03:12 um,

1:03:12 how the effects vary by how male dominated the field is.

1:03:16 Sort of the percent of women in the course,

1:03:18 um,

1:03:19 and,

1:03:19 and it sort of

1:03:20 seems like,

1:03:22 uh,

1:03:22 again,

1:03:22 a bit,

1:03:23 a bit noisy,

1:03:23 but the men may have sort of adjusted their beliefs the most in the more

1:03:28 male dominated fields.

1:03:30 And so

1:03:31 I interpret that as being consistent with,

1:03:33 you know,

1:03:33 men are more biased in the male dominated fields,

1:03:36 possibly because of gender stereotypes,

1:03:39 and they're therefore

1:03:40 more susceptible

1:03:42 to information.

1:03:44 Um,

1:03:45 all of these are,

1:03:45 are a bit underpowered,

1:03:47 as I,

1:03:47 as I think I mentioned earlier,

1:03:48 um,

1:03:49 I had planned for,

1:03:50 for twice the sample size,

1:03:51 but unfortunately,

1:03:53 uh,

1:03:53 the second round was,

1:03:54 was canceled.

1:03:56 Um.

1:03:58 OK,

1:03:58 um,

1:03:59 and then just a little bit of discussion,

1:04:02 given,

1:04:02 given time,

1:04:03 I'm gonna skip

1:04:04 the sort of context and the wider literature,

1:04:06 but if you want to talk about other papers,

1:04:08 I'm happy to.

1:04:10 I'll get some of the sort of the other questions that have come up.

1:04:14 Or,

1:04:15 um,

1:04:16 you know,

1:04:16 sort of

1:04:17 really fundamental questions that underlie this work and,

1:04:19 and what I find,

1:04:20 um,

1:04:22 so

1:04:22 the first is,

1:04:23 is Stephanie,

1:04:25 yeah,

1:04:25 would you mind to take a question from Tuan

1:04:28 because maybe it's related with the previous section.

1:04:30 Sure,

1:04:31 go ahead down,

1:04:32 please.

1:04:32 Yes,

1:04:32 I,

1:04:33 um,

1:04:33 so,

1:04:34 so I feel that,

1:04:34 you know,

1:04:35 looking at

1:04:36 major choices is one aspect of,

1:04:38 of decision making.

1:04:40 And then there,

1:04:41 there,

1:04:41 there's another one that you,

1:04:42 you talk about a little bit which is effort

1:04:44 because if I realize my score is low,

1:04:46 it,

1:04:46 it might be due to

1:04:48 some inner ability that

1:04:50 might lead me to choose a different field.

1:04:52 And also how much effort I'm going to put

1:04:54 in the,

1:04:55 in,

1:04:55 in studying.

1:04:57 And if we do this,

1:04:59 you know,

1:04:59 how much effort in my choice of measure will depend

1:05:01 only on how much I'm going to adjust effort,

1:05:03 but how much I also believe others are going to adjust.

1:05:06 In terms of effort.

1:05:08 So it's essential to me how,

1:05:09 how,

1:05:09 how,

1:05:10 how you want to think about,

1:05:12 about that margin,

1:05:13 that kind of intensive margin of,

1:05:15 of adjustment

1:05:17 and,

1:05:17 and the color is,

1:05:18 is

1:05:19 not finding an effect in your,

1:05:22 in your test might be related to the fact that some people who decide to drop.

1:05:26 Will decrease the threat while people who decide to stay will,

1:05:28 will increase the effort level.

1:05:30 So on that,

1:05:31 it's unclear whether we should expect a positive or a negative.

1:05:37 Yeah,

1:05:37 so again,

1:05:37 sorry,

1:05:38 there's a bit of like interference.

1:05:39 It was a bit hard to hear,

1:05:40 but I,

1:05:40 I think

1:05:41 you were saying it's,

1:05:42 it's,

1:05:42 it's a bit hard to think about sort of what direction

1:05:44 the effort response might be.

1:05:46 Is,

1:05:46 is that what you were saying?

1:05:47 Um.

1:05:49 No,

1:05:49 I'm just saying one margin which is I think as

1:05:52 important

1:05:53 as the choice of the major or dropping in or not

1:05:57 is how

1:05:58 students who get information are going to adjust their effort

1:06:01 into studying.

1:06:03 And that aspect

1:06:05 not only depends on their own action but what also they believe others are going to,

1:06:09 others are going to put effort in.

1:06:11 And so I,

1:06:11 I,

1:06:12 I'm not sure

1:06:13 what religious form.

1:06:15 Tests is,

1:06:16 is,

1:06:16 is going to,

1:06:17 to,

1:06:17 to,

1:06:17 to correctly

1:06:19 capture the impact of your intervention.

1:06:23 Yeah,

1:06:23 so,

1:06:23 you know,

1:06:23 I'm,

1:06:24 I'm a bit limited.

1:06:24 I,

1:06:25 I have their,

1:06:26 their final,

1:06:26 um,

1:06:27 their performance on the final exam in these courses,

1:06:29 and then I have their final like raw score

1:06:32 in the course.

1:06:33 That's sort of my measure of,

1:06:35 it's,

1:06:35 it's not directly effort,

1:06:36 it's sort of the,

1:06:37 the output of effort and and some other factors,

1:06:40 um,

1:06:40 and I don't find,

1:06:41 I don't find changes there,

1:06:43 um,

1:06:43 but

1:06:44 if I was able to directly measure,

1:06:45 say,

1:06:46 you know,

1:06:46 study hours or something like that,

1:06:47 that would be interesting as well,

1:06:49 but,

1:06:49 um,

1:06:49 I'm not able to.

1:06:55 So Stephanie,

1:06:56 there are 2 more questions.

1:06:57 Do you want to take them now?

1:07:02 Um,

1:07:02 I,

1:07:02 if it's OK,

1:07:03 I might,

1:07:03 I might push just so I can

1:07:06 address some of these things which I think address other things that,

1:07:08 that have come up,

1:07:09 and then I'll leave some time at the end.

1:07:11 Um,

1:07:11 yeah,

1:07:12 OK,

1:07:13 sounds good,

1:07:14 um.

1:07:15 Yeah,

1:07:15 so just,

1:07:16 you know,

1:07:16 sort of the the the fundamental questions that have come out of what I find,

1:07:22 um,

1:07:22 and,

1:07:22 and one is sort of the formation of

1:07:24 these beliefs or where the biases come from and why

1:07:27 there are differences by gender in the first place.

1:07:30 Um,

1:07:30 that's a bit sort of

1:07:32 outside the scope of what I do,

1:07:33 but other work suggests that that students learn about their abilities

1:07:38 through their prior academic performance in in both high school.

1:07:41 School and college,

1:07:42 um,

1:07:43 and I,

1:07:43 I find that myself,

1:07:44 right?

1:07:44 Like over time,

1:07:46 in the absence of my intervention,

1:07:47 students seem to be,

1:07:48 be learning about themselves,

1:07:50 um,

1:07:51 but even controlling for all of these prior measures of performance

1:07:55 doesn't explain the,

1:07:56 the gender gap in beliefs.

1:07:58 And,

1:07:58 and so I think there's really a role for

1:08:00 stereotypes in explaining where these beliefs come from.

1:08:03 So there's work by by psychologists and,

1:08:06 and behavioral.

1:08:06 Economist about biased beliefs,

1:08:09 how they form and how they persist,

1:08:10 um,

1:08:11 and in sort of like formal models of,

1:08:13 of stereotypes,

1:08:14 where without,

1:08:15 you know,

1:08:16 without full information,

1:08:17 people use shortcuts,

1:08:18 um,

1:08:19 to,

1:08:19 to,

1:08:20 to form beliefs about groups or,

1:08:22 or yourself as part of a group,

1:08:23 um,

1:08:23 and so exaggerated stereotypes like men being better at

1:08:27 STEM can persist despite very small true differences.

1:08:31 Um.

1:08:32 And then I think,

1:08:33 you know,

1:08:33 another,

1:08:34 another big question is,

1:08:35 OK,

1:08:36 if the beliefs about relative performance and underconfidence

1:08:40 don't explain the women's decisions,

1:08:41 like

1:08:42 even updating their beliefs wasn't enough to change their STEM behavior,

1:08:45 so,

1:08:45 so what is,

1:08:46 what sort of can

1:08:48 move that decision for women?

1:08:50 There's,

1:08:51 um,

1:08:51 you know,

1:08:51 some

1:08:52 sort of an explanation that often comes up is about

1:08:55 comparative advantage,

1:08:56 and if,

1:08:57 uh,

1:08:57 you know,

1:08:58 even if they update their beliefs about STEM,

1:09:00 the women are so much better in their non-STEM courses

1:09:03 that it's not enough to sort of pull them into STEM.

1:09:07 And um other work has,

1:09:08 has,

1:09:09 has suggested this,

1:09:10 and I find some support for that in the data just given

1:09:13 how much stronger women's performance is in non-STEM courses.

1:09:17 Um,

1:09:18 there's also some,

1:09:19 some other work suggesting that non-academic factors like

1:09:22 peer effects,

1:09:23 role models,

1:09:24 sort of information about the field,

1:09:26 um,

1:09:26 could affect women's decisions,

1:09:28 that might matter for them.

1:09:30 Um,

1:09:30 it could also be that,

1:09:31 you know,

1:09:32 types of beliefs other than the,

1:09:34 the precise ones that I targeted in this intervention

1:09:37 might be the ones that really matter for women.

1:09:40 Um,

1:09:42 And then just,

1:09:44 just to sort of,

1:09:45 uh,

1:09:45 get to,

1:09:46 there was an earlier question that I that I punted,

1:09:48 um,

1:09:48 so I want to make sure I,

1:09:49 I talk about this.

1:09:51 So,

1:09:52 uh,

1:09:52 you know,

1:09:54 the

1:09:56 was,

1:09:56 was this intervention good,

1:09:57 essentially?

1:09:57 How do we think about welfare,

1:09:59 given that

1:10:00 it,

1:10:00 it works by discouraging some men from STEM?

1:10:04 Um,

1:10:05 and so there,

1:10:05 there's a number of things that,

1:10:06 that

1:10:07 we want to think through and,

1:10:08 and whether this was a,

1:10:09 a welfare improving policy.

1:10:12 Um,

1:10:12 so,

1:10:13 first is,

1:10:13 you know,

1:10:14 do these effects that I found on short term

1:10:16 course taking persist to major declaration and degree,

1:10:20 um,

1:10:21 and,

1:10:21 and if they do,

1:10:22 right,

1:10:22 what do men pursue instead,

1:10:24 and what are the associated outcomes,

1:10:26 um,

1:10:26 with those other majors,

1:10:28 um.

1:10:29 It's also important to think about sort of the interaction between,

1:10:32 uh,

1:10:33 major choice and performance in a field,

1:10:35 right?

1:10:36 So,

1:10:37 uh,

1:10:37 is it actually better to be sort of a mediocre

1:10:39 STEM major and and stay in STEM given the,

1:10:41 the labor market returns,

1:10:43 or is it better to be sort of a higher performing non-STEM major?

1:10:47 Um,

1:10:47 that's not actually something we really have much causal evidence on.

1:10:50 Um.

1:10:51 Another thing to consider,

1:10:53 um,

1:10:53 the,

1:10:54 the information might have just shifted the timing of certain men leaving STEM,

1:10:57 so they just exited STEM somewhat faster,

1:11:00 which could be welfare improving for them.

1:11:02 Um.

1:11:04 And then there are a couple of important

1:11:05 spillover effects that could be playing out.

1:11:07 So,

1:11:08 for example,

1:11:09 um,

1:11:09 if some,

1:11:10 some of these majors in this setting are capacity constrained,

1:11:13 and so men leaving could free up spots for others,

1:11:16 could,

1:11:16 could possibly,

1:11:17 um,

1:11:18 improve,

1:11:18 you know,

1:11:19 efficient sorting into these majors,

1:11:21 um.

1:11:22 And then finally,

1:11:23 you know,

1:11:23 we,

1:11:23 we have a peer effects literature about the,

1:11:26 the effects of,

1:11:27 of having different types of students around you,

1:11:29 right?

1:11:30 So,

1:11:30 it might be better just to have fewer men around for the women who are left behind.

1:11:34 Um,

1:11:35 there could also be peer effects of having sort

1:11:36 of a different composition of peers who are somewhat,

1:11:39 uh,

1:11:40 less low achieving or less overconfident.

1:11:43 Again,

1:11:43 that could sort of the,

1:11:44 the peer effects literature is a bit mixed on what direction we would expect.

1:11:48 Um.

1:11:50 OK.

1:11:50 Uh,

1:11:51 so I'll just

1:11:52 wrap up,

1:11:53 uh,

1:11:53 in the last two minutes.

1:11:55 And just sort of say that this this project represents the first

1:12:00 experimental evidence from the field about

1:12:03 biased performance beliefs and gender differences

1:12:06 in STEM persistence and enrollment,

1:12:09 so the results,

1:12:10 sort of the way they played out.

1:12:12 are a bit surprising and the welfare implications as I just went over,

1:12:16 are somewhat ambiguous,

1:12:18 but I'm advancing understanding of an important topic

1:12:21 that we didn't have great causal evidence on before

1:12:24 how students make decisions and how it varies by gender.

1:12:28 Um,

1:12:29 it's just as important to know what mechanisms are not important or what

1:12:33 policies don't work for a given goal like retaining women in STEM.

1:12:38 Um,

1:12:38 if we're in,

1:12:38 you know,

1:12:39 a world with limited resources,

1:12:40 we don't want to pour them all into something that,

1:12:42 that won't work.

1:12:44 Um,

1:12:44 and this is important to know,

1:12:45 given,

1:12:45 you know,

1:12:46 sort of the conversation we often hear about women,

1:12:48 you know,

1:12:48 women just need to lean in or advocate for themselves.

1:12:51 Um,

1:12:52 my work suggests that's not actually the,

1:12:54 the solution.

1:12:55 Um,

1:12:56 so I'll just end by saying we need

1:12:57 to continue to do careful research into additional mechanisms

1:13:01 and policies that can improve diversity in STEM.

1:13:05 Um,

1:13:05 and I'll just say thank you and,

1:13:07 and acknowledge that,

1:13:08 um,

1:13:08 I,

1:13:09 I had some really fantastic research partners at the

1:13:12 University of Michigan who helped me with this project.

1:13:14 So

1:13:15 thank you all.

1:13:15 Um,

1:13:16 if you have a question or two,

1:13:16 I'm happy to stick around,

1:13:17 um,

1:13:18 before,

1:13:19 before you jump off.

1:13:20 Thank you so much.

showAllTimestamps

no

transcript

Thanks so much. Um, I'm thrilled to be talking to all of you today and, and meeting with some of you as well, um, to talk about my, my work and my job market paper. Um, so, this project is about college field specialization and beliefs about relative performance, where I ran an experimental intervention to understand gender gaps in STEM. So field specialization and college major choice are hugely consequential decisions, and understanding how people make those choices, and in particular how it varies across groups is really important for policy. We hear a lot about shortages of STEM workers and STEM skills, with some policymakers calling for an increase in the number of STEM graduates. Um, in addition to overall shortages, women are underrepresented in many quantitative fields such as economics, computer science, and engineering. Even though more than half of all bachelor's degrees go to women, there are only 1/3 of the degrees in economics and about 10% in computer science and engineering. Um, and this gender gap in STEM education has implications for both equity and efficiency. So the fields with the fewest women also tend to be the highest paying ones, so differences in field specialization contribute to the gender pay gap. And furthermore, if we're in a world where people should specialize according to comparative advantage or the thing they're relatively best at, then removing barriers that are preventing them from doing that would increase overall efficiency. So we know that differences in mathematical aptitude or performance explain very little of the gender gap in specialization, but gender differences in beliefs about performance, conditional on actual performance, may be partially responsible for gender differences in educational choices. And this is supported by observational evidence from sociologists, psychologists, economists, education researchers, um, that men and women have different beliefs about their own performance or competence, um, as well as theory from economics and others that these beliefs should matter. But there's been really limited causal evidence on this mechanism. So in this work, this is the first experimental evidence isolating the effect of beliefs about relative performance on field specialization in college. So the setting I use to do this is 5700 undergraduate students in seven large introductory STEM courses here at the University of Michigan. Um, I'm gonna start by documenting gender differences in beliefs about relative performance. So just at a descriptive level, what do these patterns look like. Um, in my experimental intervention, I provide students with information about their relative performance in STEM courses, and then I measure how both their beliefs and their actual academic behavior change. To preview what I find, um, absent any intervention, men are more overconfident than women. Um, and then I find that my, uh, my intervention of information provision induces substantial belief updating, and the gender gaps in biased beliefs close by between 3.5, depending on the measure. And then, finally, the information provision shrinks gender gaps in STEM persistence by about 10%. And the changes appear to be driven by lower performing, overconfident men, updating their beliefs downwards and taking less STEM. So, I interpret all of this as evidence that beliefs about relative performance do matter for field specialization and STEM persistence, but it mainly operates through male overconfidence rather than female underconfidence. So the setting I use to do this, I study undergraduate students across 7 large introductory STEM courses at the University of Michigan, and I did this in the fall of 2019. My sample is about 55,700 students, and they're spread across introductory. Biology, chemistry, computer science, economics, engineering, physics, and statistics courses. Um, just as a note, um, when I refer to STEM throughout, um, I do include economics, uh, as, as a sort of similar quantitative male dominated field, although definitions of STEM vary a bit. Um, this, this sample is just slightly under 50% women, um, because these are introductory courses, the majority are in their first or second year of college. Um, and this is a, this is a population that's already very interested in STEM. So I surveyed them at the beginning of the semester, and 72% of the women, 85% of the men are considering or intending a STEM major at the beginning of the course. Um, so you can mostly think of this, um, project as as targeting STEM persistence with students who are already invested in STEM in some way. Um, so I use two main sources of data in this project. The first is university administrative data from the University of Michigan. So this is background and transcript data that allows me to measure a number of pre-treatment, demographic, and academic characteristics about these students, including high school performance, socioeconomic status, and, and prior performance in college. Um, I, I also have their course taking outcomes one semester after the intervention. So, uh, that's going to be my, my primary behavioral measure is the, the number of STEM courses or credits that students take following the semester of the intervention. Um, and because this is administrative data, uh, I have universal coverage. The only reason a student wouldn't appear in this data is if they, uh, disenroll from the University of Michigan. I combine that with survey data, uh, so I, I survey students to get information about their beliefs, about their own performance, and the performance of others. Um, and I do these surveys at two points in time. So, the first is the Beginning of the semester, or before the intervention occurs, um, and then I do a second set of surveys at the end of the semester, or after the intervention occurs, um, and the, the end of semester is going to be the source for the, um, the beliefs outcomes for this project. Um, the response rate for these, the, the end of term survey was about 50%. Uh, the incentives offered for these, for these surveys varied by course and was sort of at the discretion of the instructor, whether they, they incentivized with with course credit, um, or extra credit for the, for the surveys. So, sort of before I, I get into my intervention and what it did, I want to start with some motivating evidence for this hypothesis that men and women have different beliefs about their relative performance, and that those differences might be responsible for some of the gender differences in enrollment and persistence in STEM. So, I measure beliefs about relative performance, uh, using these student surveys that I mentioned, and again, I measure them at two points in time. So, I measure, uh, students' beliefs at the beginning of the semester, uh, which is around September, October, um, and then again at the end of the semester. Which is in December, right before their final exams. Um, and just to, to specify in this section, when I'm, I'm looking at descriptive patterns and beliefs, I'm limiting to control students only to see what students' beliefs look like and how they change over time, in the absence of any intervention. Um, I also am limiting to students who responded to both of the surveys, uh, so that I'm not confounding changes over time with changes in the sample. So, I have two different measures of beliefs about relative performance, and the first is the students' prediction of their final percentile rank in the course. So, specifically on the on the survey, I asked them, or they tell me, In terms of my final grade, I expect I will do better than 1% of my classmates in Econ 101, for example, um, and I can compare their answer to their actual realized performance or percentile rank in the course. So I'm going to show a few graphs that look like this, and so what you're seeing here on the on the x axis we have the students realized percentile or their actual performance in the course, where 0 would sort of be the lowest performing and 100 is the top performers, and I plot that on the y axis against what they predict their percentile to be at the beginning of the semester. And so if a student was perfectly accurate in their prediction, they would fall along this gray 45 degree line. Um, a student who was overpredicting, uh, who sort of thinks they'll do better than they actually do, would fall above the line, uh, and those who are underpredicting would fall below the line. Um, and I'm, I'm gonna graph this as a, as a bin scatter plot, where I, I group students and, and plot the within bin means. So, uh, this is what it looks like when I, when I plot students the beginning of semester beliefs, and there's sort of two interesting things in this picture. So, The first is that the, the slope through these points is fairly flat. So that means, you know, without much information, this is consistent with students mostly just guessing about how they'll do. Um, and that makes sense, because again, uh, they answer this early in the semester, they haven't received many signals of their performance, uh, 90% of them answer this question before they've even taken an exam in the course. Um. And so sort of the, the second interesting point here is that most of these points fall above the 45 degree line, which means that the average student is overpredicting how they'll do. Um, so, you know, sort of in sum, everybody is roughly guessing, and everyone is guessing that they'll do a little above average. Um, but of course, I'm interested in gender differences. So, when I break this down by gender, when I'm showing the, the men here are the open blue circles, and the women are the, the solid orange. The, the really striking thing here is that throughout the true performance distribution, the men think they'll do better than the women on average. Uh, and so I can, I can formally test this by, um, estimating, uh, regression through these points with an interaction of, um, of, of gender and the, uh, predicted percentile. And I, I find that, uh, the difference in intercept for men and women is 8 percentiles, with no difference in the slope. And so that means for a man and a woman with the same actual performance in a STEM course, a man thinks he'll do 8th percentile ranks better than the woman does. And so this is what students predict at the beginning of the semester, right, um, without much information, but we would expect that as students learn about their performance over the semester, they would update their beliefs. And in fact that's exactly what happens, right? So I'm now plotting on the y axis, the end of semester beliefs. So students are now predicting their performance in December, right before their final exams when they have most of their grades. And so the sort of key thing compared to the previous graph is that all of these points have shifted closer to the 45 degree line, meaning that on average students have become more accurate in their predictions. But there is still this gender gap that uh remains among the lower performing students, with lower performing men particularly overconfident. And so this difference in intercept is still 8 percentile ranks. Um, I, I can't detect a difference in the slopes, although this is a, a somewhat um limited part of my sample. Um, maybe I'll, since we've, we're, um, after our 1015 minutes, um, maybe I'll pause and, and see if anyone has any questions or hand raises, um, that we should address. OK, sure. Uh, there is uh David. David has a question. Hi, hi, I, I wanted to ask about the way you were measuring expectations here. I mean, we know, you, you know, people use fifty-fifty a lot when they're uncertain. We know that when you ask what do you expect, some people are saying mean, some people are saying medium, some people are saying mode, you know, you, you're not, did you sort of, why did you ask it this way, or did you think about sort of incorporating uncertainty on estimates when you're asking that question and you know, should we think about uncertainty in interpreting this figure? Yeah, so that's, that's a fair question, you know, sometimes when behavioral economists, um, elicit beliefs in, in a lab setting, they're able to, you know, ask more complicated questions to get at the full distribution of beliefs. Right, like what's the percent chance you get this grade versus this grade versus this grade, um, you know, in, in a field setting, I was, I was limited in the number of questions I could ask, um, and so I, I focused on, uh, you know, this item and another item I'll go over in a bit, um, about, uh, just sort of a simple prediction, um, and. And it is sort of, you know, how students interpret it, um, you know, it was true in this, this previous slide that there were a lot of beliefs sort of clustered around 50, right? Everyone is, is sort of guessing that they'll do about average, um, which without a lot of information is, is a, you know, a fairly sort of expected thing. Um, but I, and I have, I have one more measure of beliefs that I'll, I'll go over, um, and let me know if, did that answer your question. OK. OK, good. Harris, please go ahead. Hi. So, uh, you mentioned that uh the response rate to the end line was about 50%. And I was wondering if there was a risk of you know selection in your analysis and how you may have dealt with that. Yeah, so, so, yes, with, you know, with, with survey response, um, it's not, it's not full response, and there are certain groups that are sort of more or less likely to respond to the survey. I, I do a number of things to address that, um, and in particular when I, when I start talking about the uh the outcomes at the end. So one thing is that my behavioral outcomes that use administrative data. There's no missingness in those. So sort of any behavioral responses you see, you can, you can think of as, you know, that that captures the whole population. Um, there is sort of a question about translating between the, the survey beliefs and the, the behavioral responses, because those, those might not actually quite be the same groups. Um, I, I do a couple of things to address that in the paper. Um, I, I do a, um, an in. Uh, propensity weighting where I, I wait the survey respondents to look like the full sample. Um, I also do something sort of going the other way where I only look at administrative data for people for whom I also have, um, survey responses, and, you know, the conclusions generally look the same with with these different methods, um, but, uh, and, but, but you can sort of see here, um, in these pictures, at least. It, it does span the distribution of, of performance in the class, um, for example, so I am holding that constant in all of these. OK, good. So there are questions from Clement and then Pierre, and uh then you can continue, Stephanie. Hey, um, my question is, um, so it could be that in, um, that you failed, for example, and that, that could be different from your true beliefs. So it could be that men actually have a uh a certain belief but then when they respond, they, they're more likely to, to try to project uh uh a better image uh outwards. So is there any way to distinguish between those two things? Um, so you, you cut out a little bit at the beginning, but I, I think you're, you're asking like, is this capturing their true belief? Is that sort of your question? Yeah, so, so, you know, I, I ask this in a sort of simple way on a survey. Um, it's, it's not necessarily an incentive compatible measure, which is, you know, I think fair to point out, um. One, you know, if, if I had actually made sort of incentivize these students to give an accurate answer, that itself would have been a treatment, um, which would have interfered with, with my treatment. Um, but as I, you know, I think as I'll, as I'll sort of show, even if there's some error in this and some, you know, possibly interesting things about, um, you know, projecting or Sort of projecting overconfidence, even if you're not overconfident, um, these measures are correlated with behavior, um, both in the control group and then the experiment by changing these beliefs changes some behavior. Um, so, so even if they're not a perfect accurate representation of beliefs, I think they're capturing something real and something interesting, um, in terms of gender differences. Pierre, go ahead, please. Thanks, Stephanie. Uh, a quick question, is there, it's hard to tell because you've been the data, is there, uh, more variance among men and women? And if so, you know, does that have some implication that is, you know, there's a few men who are super overconfident, but on average they're the same. Yeah, you know, I, I did look at this a while back to see, to see it, it, it that this question, like, are there sort of, um, differences in, in variants, and, you know, some other work suggests that men are in survey response in general tend to be more at extremes compared to women. Um, I didn't quite find that, but I don't think, I'm not sure I quite have the, the power to, to detect that. Um, and I looked early on if it sort of changed, if the treatment changed the variant, um, but I don't really find much there. OK, um, all right, I'll keep going. So, uh, so I have 22 different measures of relative performance beliefs, the one I just showed you about percentile rank, um, the second measure is the estimate of the median course grade for students who major in STEM. So specifically on the survey, I asked these students, when thinking just about students who declare a major in math, science, engineering, or economics, what do you think was their median grade in this course? Um, and so this is, you know, it's getting at a couple of things. It's getting at sort of the The difficulty of the class and it's really getting at the selection into STEM, right, sort of where students think the bar is in order to continue in the field and like the previous measure, I can compare this to the actual median that I can calculate using historical transcript data. So just to give an example, uh, suppose the, the true median in Econ 101 for students who go on to major in STEM is a B+, but I think it's an. That means I'm overestimating the median, or I, I think the bar, I think this median for STEM majors is higher than it actually is. So, again, I see very striking gender differences in this measure, uh, in a way that suggests men are are setting the bar for STEM majors lower than it actually is, and women are setting it too high. So, specifically, uh, women are 17% points more likely to overestimate this median grade, um, and men are 11 points more likely to underestimate it. Um, and to, to sort of, you know, make, make clear what this means, this is working in the direction of men being relatively overconfident and women being relatively underconfident, right? So, if the men think that others are doing worse than they actually are, that means they seem better in comparison. Um, and the opposite is true for women. Um, so, again, these are what students think, uh, at the beginning of the semester, and these proportions look very, very similar when I ask them again at the end of the semester, um, which we would expect because this is not a piece of information that is sort of readily available to them. Um, so I just sort of showed you these raw gender differences in beliefs, um, and, you know, we're sort of interested in how much do they matter or are they actually affecting behavior in some way. Um, so, I can also see among the same set of students in the control group, the, the gender gap in STEM credits in the subsequent semester is about 2 credits, um, and that, that's, that's saying that men in my sample take about half of a STEM course more than women in the next semester. Um, so I, I go into more detail in the paper, and I'm happy to talk more about how I do this, but I do a correlational exercise, essentially a decomposition, where I want to see how much of that credit gap, um, among the control students is explained by all of the characteristics I can measure in the data. Um, so, you know, their, their prior high school performance, their, um, their high school GPA, their socioeconomic background, all of that. And I find that even controlling for a bunch of academic and demographic characteristics, these two types of performance beliefs I just showed account for about 7% of the gap in STEM course taking. Um, and just to sort of contextualize, you know, sort of that magnitude, that's the same amount that is explained by, uh, the students' math skills when they start college or their performance on a math placement exam. Um, and, you know, I, I don't want to sort of Uh, over overstate or or overinterpret this, this precise number because this is a correlation, um, and this is sort of exactly why I, I need an experiment, but it's sort of suggestive. Um, right, so I've, I've showed that these, these different measures of beliefs about relative performance, first of all, they differ significantly by gender, in a way that suggests overconfidence among men. Uh, these beliefs change over time, just naturally as students learn about themselves, and they are at least correlationally related to behavior. Uh, but, again, so all of this so far is, is evidence that beliefs matter for behavior, and they might be responsible for part of the gender gap in STEM persistence, but it's, it's just been descriptive. Um, so, to get at the causal relationship between beliefs and academic behavior, I implement an intervention where I attempt to exogenously vary beliefs about relative performance. Um, are there any questions before I, uh, jump into the, the intervention? Yeah, so there, there were two from the previous Q&A. Theresa, can you start, please, and then. Yeah, sure. Um, I was just wondering if you could, uh, help provide a bit more context for this, this research in, in terms of the, uh, social psychology and, and sociology literature. This is, I feel like this has been investigated elsewhere, and I'd, I'd like to know, uh, you know, what the, what the, uh, contribution of this is on top of that. Yes, absolutely. Um, so I, I alluded to this very quickly. I try not to, to dwell on the literature, but you're completely right that researchers across fields, so not just economists, but, um, sociologists, education researchers, um, social, social psychologists, they've, they've documented a lot of what I just showed you. So essentially descriptive differences in, in beliefs between men and women about their, their confidence or their sort of beliefs about their. Competence in in performance, um, as well as as correlational evidence that, right, these things are sort of associated with, with major choice and, and hypothesize that that these could explain gender gaps, um, but there, there hasn't really been any causal evidence on this. They're just sort of um relying on the correlational relationship between beliefs and behavior. Um, so that's, that's where my work comes in. But I will, I will say that my, you know, it's the, the descriptive differences I've shown are, are, are consistent with, with prior work about gender differences, um, and, uh, particularly in, in male dominated or sort of male type domains. OK, thank you. Uh, can you please go ahead. Yes, uh, this is key. My question is about selection bias here. You kind of address it just now in the, uh, in the slides later, but I just want to clarify this. You, uh, the center that you selected are those students that already enrolled in those STEM courses. You center them at the beginning of the semester after they have enrolled in those courses, and then you censor them again. Yeah, you survey them again at the end of the course. Am I right? So, um, that the, the quality, Lilith, could you hear that better? I, it was a bit fuzzy, um. Oh. Or, or do you want to ask your question again, Kei? OK. Uh, my question is about selection bias. The center you have are the students have already confirmed enrolled in those 1 courses. Not the general population of men and women in University of Michigan. So there is already a selection bias that these are the students that show interest in continuing STEM courses. I see. Yeah, so yes, thank you for, for repeating and, and sorry, I was having some, some trouble hearing you, um. Yes, so you're, you're correct that, as I mentioned, you know, my, my sample are students who are in these intro STEM courses, so they've already revealed some interest in, in the field, um, you know, in terms of, of internal validity, um, I am, I am randomizing within these courses where they've already enrolled. So when I'm getting it at the causal effect of my intervention, that's, that's sort of not an issue, um, but, but there is a sort of generalizable question. Uh, about external validity and, and sort of who do these students represent and what would happen if we targeted students earlier on, um, you know, this is the point of the, of the pipeline that that I'm able to explore, um, but I think there are some really interesting questions, and I'll I'll maybe get to this more at the end about where these beliefs. From and and sort of how they might play out even earlier. Um, I, I think it's important to think about, uh, you know, maybe targeting beliefs before we get to this point, um, but that's, um, the, the sample I have is you're, you're totally right to be thinking of it as as people who have already selected into at least sort of an intro level of STEM. Just a reminder that once you have done your or complete your question, you can load your hands. I'm not sure if Tristan or Clement. No. OK, yeah, you can continue to dance no more hands. Great, um, OK, so that was all sort of descriptive evidence. So, so what do I actually do? What is my experimental intervention? So my intervention involves two different treatment arms. So in the first arm, I provide students with information about their relative performance in these STEM courses, and I'll, uh, I tell them how they're doing. Relative to their classmates as well as the performance of a typical STEM major, and I'll show you just in a second what what exactly this information looks like, but it corresponds to the two different measures that I previously went over. Um, I have a second treatment arm, which only a subset of higher performing students are eligible for. And so in the second treatment, they get all of the same information, but it's, it's paired with an explicit encouragement message to consider or stay in a STEM major. So the same information, but, but framed. In a more positive way, and, and this arm was designed based on sort of some general behavioral economic research suggesting that additional sort of nudges or salients might be necessary to move behavior, um, as well as some some work suggesting that encouragement might be especially important for women in these settings. So, what do these messages look like? Here, here's an example of the the just pure informational message. Uh, this is a sample for, for a student in the biology course. Um, and to sort of go through what this looks like, uh, the sort of key information in the intervention is, is contained here in this histogram, in this picture. So they see a number of things. So, first, they see the sort of full distribution of course scores, um, as of the middle of the semester, um, and at the point when they get this, uh, everyone has taken at least one exam, many have taken 2 in their courses, as well as any other assignments that they've completed. Um, and so I, within this distribution, I highlight their percentile, so sort of where they fall, uh, you're at the whatever it is percentile, um, as well as the second piece of information, which is the, the median grade for STEM majors. Um, and so that that information is sort of repeated again, both their, their raw score and their percentile. Um, Uh, as well as, again, sort of repeating the information about, uh, median grades, both for STEM majors, as well as for all students in the class, um, and for students in that subject to give more context about what grades mean. Uh, and then the message concludes with some basic information about, uh, scheduling, advising appointments, and, uh, course registration. So, you, you are, are likely wondering, right, I'm, I'm giving students information about, about their grades, about these courses, but, you know, what, what do they already know? Do they already know all of this? Um, and so it, it varies a bit by the course, but in general, uh, if, you know, you've ever been, been a college student, you, you know, you can, you already know. Or can sort of very easily see your raw score in the course, um, but you generally don't know your exact percentile, at least in, in this setting. So some of the instructors do provide a distribution of exam grades, but, but the students aren't able to calculate their exact percentile in the, in the full distribution, um. And in terms of sort of, uh, the distribution of, of grades, some of these courses are curved, so, you know, in, in theory, students sort of know the full distribution of grades. Um, there's also a way at Michigan to look up historical course medians online, um, but those are, are just sort of overall, they're not specific for a certain population like STEM majors, um, which, uh, you know, could be particularly important as you're thinking about whether to continue in SSTEM. Um, and then I'll just, just note, even though they have a lot of this information available to them, as I showed earlier, uh, they're, they're pretty inaccurate, right? Even though they have it, they, they have pretty substantial biases in, in both of the measures that I'm targeting. Um, and, and even for the information that is, is very readily available, so just the overall course median, um, less than a third of students can identify that. So there's, there's lots of room to, to sort of intervene on these beliefs. Um, and then I have, uh, the, the second treatment arm, right, which is, I'm calling the encouragement treatment message. It looks very similar to the informational message. It has all of the same content. This, um, this histogram, the information is exactly the same. But the, the differences are just that it's, it's framed in a much more positive rather than a neutral way. Um, so rather than telling them, here's how you're doing, uh, I tell them you're performing like a STEM major, um, and again, only high performing students are, are eligible for, for this treatment arm. Um, and then finally, I do have a, a control message where students, uh, get sort of the same basic information about advising, um, and course registration. Um, backpacking at Michigan is just what we call course registration, um, and I just, I remind them of their, their raw score in the course, um, and sort of My my reason for sending a control message was to hopefully partially mitigate any spillovers. So the idea is that, you know, your friend got a message about their grades, and you don't wonder, oh, why didn't I also get a message about my grade in this course. Um, so the, the intervention is delivered a single time. Um, it's delivered in the middle of the semester, and I, I targeted it to be around the time that they would be registering or making their decisions for courses for the next term. Um, they received these messages as emails. Uh, they could also view them online, uh, through a site that they, they used in these courses. Um, the, the majority of students, um, 83% of those who were sent the message did view it in some way. Um, that's likely an upper bound on how many actually sort of read it and internalized it, but, uh, that's sort of the best measure I have of, of take up. Um, in terms of the, the design, uh, this is an individual level treatment assignment, so I, I assign at the student level, um, but I, I stratify within, um, course, gender, and performance, uh, so whether they are above or below the median in their course at the, at the time of randomization. Um, and just one sort of Technical note, um, students, uh, you know, this is across 7 courses. Students could be taking 2 of these courses, possibly 3 at once. Um, and, and so if that was the case, I, I randomly pick a course for them to be in for the study, um, and they're, they're just not in the study for, for the other course, um, not even a control message, uh, so that I, I can just do this sort of at the unique student level. Um, and just to sort of summarize the, the design given these, these two different treatment arms, so I have my, my full sample of students. And based on whether they're below or above the course median, that determines what what treatment they can get. So for the, the lower performing students, the ones below the median in their course, um, they can be assigned to the control or the information only condition with equal probability. Um, and if, if a student is in the top half of their course, they can be assigned to the control, the information only, or the encouragement treatment with also equal probability. Um, in terms of how I analyze this, um, I'll just, I'll just mention upfront that I did pre-register this this project with the AEA's trial registry, um, and, and all of my analyses, including the covariates I use, were pre-specified, um, although I do have some exploratory outcomes that I can discuss at the end. Um, for the most part, I'm interested in gender differences, and so I'm gonna estimate male and female effects, um, separately, and I, I do that with a, with a single regression where I interact, uh, treatment with gender. Um, I control for a number of pre-treatment, demographic, and academic characteristics about these students. Um, I, I include strata fixed effects because I, I had a stratified design. Um, and I'm gonna show, uh, today I'm gonna show intent to treat effects, um, so the effect of being sent an intervention message. So, beta one is the intent to treat effect on men. Beta 1 plus beta 3 is the intent to treat effect on women. Um, but as, as I mentioned earlier, uh, 83% of students viewed the message, so, um, you could, you could scale up by, uh, by 1.2% if you wanted to get a treatment untreated effect. Um, and just one note, so, uh, this was, this is a multi-armed treatment with, uh, I had the pure information, and then the information plus encouragement arm. Um, in general, I don't actually detect any difference between the two treatments, either on students' beliefs or on their ultimate behavior. Um, and so for that reason, um, and sort of for simplicity and also for for power reasons, for most of what I'm presenting today, I combine the two treatment arms. So you should sort of think of this as a general effect of receiving information because all of the treated students receive the same set of information. Um, but for the, for the higher performing students, the effects are going to be an average of the, the information only, and the information plus encouragement, um, but I don't actually distinguish between those two. Um, OK, and this is probably a good, another good place to, to pause for questions before I get into what I found. OK, good. Um, yeah, Alvaro, go ahead, please. Hi Stephanie, uh, quick clarifying question for those of us who didn't go to undergrad school in the US. So here they, they register for courses in the next semester within, within like you were saying in mid-semester. So because otherwise you, if, if updating is occurring and occurs properly in terms of relative performance, then by the end of the semester you sort of know how you do. And if you're resistant to courses, then, then there is no, there shouldn't be, I mean, your confidence effect should, should. To mitigate, right? So, so here, I mean, I guess that if there is an issue here, it's going, it's coming from the fact that you're registering early when you don't have that much information for the, you know, upcoming, upcoming courses. Yeah, so, so thanks for that clarifying question. Um, yeah, so in general at, at Michigan at least, um, you register for the next term's courses in the middle of the, the current semester, so you have some information, you know, a bit limited on, on your performance up to that point, and that's what they're basing. These decisions on, um, and I, I targeted the intervention. It was sort of this, this trade-off, right? I did it in the middle of the semester so that there was enough information so that this wasn't a completely noisy signal, um, but I wanted to, to target it at the time of decision making so that it really had a chance to, to actually affect their behavior. I'm very quickly to follow up. So, and then when you were saying two additional credits at the beginning that you STEM credits that you take in, in the next semester, are you still able to find like actual The declaration of major, major declaration like later on, do you have enough time of observation to, to hear us, especially for the, the, the, the upcoming results that you're gonna show I guess. Yeah, so the answer is right now I do not because not enough time has passed. I only have sort of one full semester of data on these students, um, since I, I did this about a year ago, um, I will be getting that soon, but, uh, yeah, so eventually I will be able to look at their, their major declaration and ultimately what they, what degree they, they graduate with, um, and I'm, I'm of course interested in that, but in, in the short term, um, I can look just at their behavior in the next semester. Thank you. Thank you. OK, there is one more hands up. Uh, Anu, please. Hi, Stephanie. So you mentioned uh spillover effects. Do you actually estimate if there are spillover effects on control classmates within the same course or on students' friends maybe in other classes? Yeah, so I, I, I cannot directly estimate spillover effects. I, I think it's a sort of important thing to, to think about in this context. Um, I, I, I, I will maybe mention this at the end again as well, but I, I had initially planned a second round of this intervention that had to get canceled because of COVID and the university shutting down and, and all of that. Um, but in that round, I, I really want to get at this, and I had designed some survey questions to ask students about essentially their their networks and their peers and their friends and sort of who they studied with or discussed grades with in these courses so that I would be able to actually see, OK, do control students who have treated friends change their behavior differently than control students who don't have treated friends, for example, um, to actually get it at the spillover. But unfortunately, I wasn't able to collect that data and I can't directly estimate it. Um, I think, you know, if anything spillover in this case would would sort of bias me towards, towards zero. Thanks. OK, I don't see the, uh, yeah, Tom, please go ahead. So follow up on this, uh, Suppose that the extent to which you discuss your grades, your performance differs by gender. So then your, your, your issue is that it might, it's just not some common effect that biases one way or the other. It's, it can have a, a gender differentiated bias. So, how do you think, how do you want to think about this? Yeah, so, again, it's, it's possible, and I think it, given my results, um, you would worry that maybe the women particularly discuss their grades, and so that biases me towards, towards finding nothing for the women. Um, uh, again, I, I don't quite have the survey data to, to rule that out, um, although I did do some sort of early focus grouping, talking to students, um, early on to, to get a sense. Of of exactly this, right? Like, do you talk to each other about grades? If you got a message telling you your percentile in the class, would you discuss that with your peers? And I think the fact that these messages are very personalized, um, and students view grades generally as a pretty private thing, um, they're not necessarily likely to just forward an email or, or bring it up, um, but it, it is always possible. OK, good. So I don't see more hands up and please don't forget to load your hands in when you are with your question. Go ahead, Stephanie, please. Great. um OK, so I just sort of went over the the intervention. So what did it do? Um, I'm gonna start sort of, uh, first set of results is about the effect of the intervention on student beliefs. Did it, did it actually cause them to update their beliefs in the ways we might expect? Um, so again, I, I measure belief outcomes in, in two different measures very similar to what I went over before. Um, so the first is the, the error in their own percentile belief where I, I subtract their, their actual realized percentile from what they predict it will be, um, and so the outcome is the size of the error, um, and, and just one. Slight change from from what I did before. I'm, I'm now using the um the realized percentile percentile. I'm measuring that as their, their mid-semester performance because the final performance could itself be an outcome that's affected by treatment. So I use the the sort of pre-treatment measure of their actual performance. Um, the, the mid-semester performance is also the thing that I sort of directly tell them about. Um, the, uh, so if, if they have a positive error, that means they're overpredicting or overconfident in their performance, um, and a negative error means that they're underpredicting or underconfident about their percentile rank. Um, and then the, the second measure is, um, again, this measure of their, um, how accurate they are about the, the median grade for STEM majors, um, and whether they're overestimating that value. So, uh, I'm going to start with this measure of the error in a student's percentile belief, um, and, and first I'm just showing control means. So recall from those, those scatter plots earlier that the average student was somewhat overpredicting, uh, which is why both of these control means are positive, um, and men overpredict by more than women, right? The control mean is larger for men. Um, but then I estimate the treatment effect on these beliefs, um, these darker bars are showing the estimated treatment effect with a 95% confidence interval. Um, and so, you'll see here, um, I, I find no average effect on this error measure for men or women. But remember, uh, people have both positive and negative errors, and they can update in either direction. Um, so these, these null effects across sort of all men and all women are actually masking some very real changes in their beliefs. So, I further break this measure down by gender, but now also by performance. So recall that the um the lower performing students tended to be overconfident, right? So the, the below median students have positive errors, um, and especially the lower performing men are especially overconfident. Um, whereas the higher performing students were somewhat underconfident, so they had negative errors, um, the women were, were, were slightly more underconfident than the men. And so the effect of the treatment is that, uh, it causes men to update their beliefs in both directions, um, depending on where they are. So the lower performing, overconfident men update downwards, um, and so this, because they were overconfident, This, this is an accurate, uh, uh, revision of beliefs, whereas the higher performing, uh, somewhat under confident men update upwards, um, also by about 4 percentiles. Um, and the, the women's beliefs about their percentile don't seem to change. And so sort of the, the results, um, especially if we, if we focus on these, these lower performing students here, the, the change in the gap, uh, in, in the air in this belief closes by 4.5 percentiles or more than half. Uh, so the, the, the second measure I have is about what they believe the, the median grade is for STEM majors. And so now I'm looking at the proportion of students who are underestimating the STEM median. Um, so, you'll recall that men were more likely to underestimate that quantity to sort of set the bar lower than it truly was, um, by about 10% points compared to women. The effect of the intervention is that men correctly update that belief, um, so, so they correct that by 5% points, uh, and the result is that the, the gender gap in, in this measure, um, under predicting the median, closes by 4% points or about a third. Um, and so, so both of these, these changes so far, uh, mean that men are becoming less overconfident about their performance relative to others, right? They, they sort of updated about their percentile, and they're also updating about how, how others are doing. Um, and then the, the sort of flip side of this is the proportion who are overestimating the STEM median, um, and women were much more likely to do that in the control group, to, to set the bar too high. Uh, and the result of the intervention is that women correct that belief, um, by, again, about 5% points, and men don't change, and so the, the gap closes by about a third. So, to, to just sort of summarize the belief effects so far, um, I've shown that information provision causes men to correct beliefs about themselves, um, and particularly low performing men correct their overconfidence. Um, it causes both men and women to correct their beliefs about others, um, in a way where men are correcting their relative over. Confidence, women are correcting their relative underconfidence, um, and so the gender gaps in these beliefs close significantly. Um, and if it's true that beliefs affect behavior, then these changes would lead us to expect men to decrease their STEM persistence and possibly for women to increase. Um, So maybe before I, I show the, the final results, um, was there another question or two? Yeah, there are 2. Takki, go ahead, please. Um, hi, Stefani, this is Tabaki, and, uh, I was wondering actually, so, uh, I know that, uh, this is, uh, these results are, uh, it's not controlling for selection in this team, but I was just wondering if, uh, there is any selection, uh, if more high ability women relative to more high ability men. Select into STEM major and if high ability women have this, uh, tendency to underestimate themselves more compared to low ability women, if that could change your results, I mean, uh, it would be probably nice to see if this actually exists, that more high ability women relative to, uh, more high ability men actually select into STEM majors or it's something which doesn't exist, then, then I guess that wouldn't be too much of a concern. Um, So, you're, you're asking, um, if, if it's the case that that the, the women who end up in my sample are sort of like more prepared than the men because they, they have more positive selection? Right, yes, relatively in STEM, if more, uh more high ability women relative to men are kind of selecting their. Um, I don't find that. I think I took out the. Sort of all of the some some more of the descriptive statistics, but um in general they have similar performance, and if anything, the men have slightly higher um like math standardized test scores and that sort of thing, although the women have slightly higher overall GPA, um, but But that doesn't quite get it like compared to the, the, the students who don't select into STEM. Um, so I haven't, I haven't looked a ton on, on that selection margin since it's not, that's not what I'm, I'm targeting, um, but I think there is some, some other evidence that that, that could matter and, uh, and you, and you do want to target earlier on. Thank you. Uh, no, go ahead, please. Thanks. Is it my turn? OK, this is key. Sorry, I have one more clarifying question. The intervention, uh, stated the percentile distribution of the students related to the STEM major in the past. So, am I right that the grading policy of the University of Michigan is always on a curve? The students are all, uh, there's no like everybody gets A's, so the percentile distribution doesn't make sense. Or they, they always go through a curve that top 5% get A, the next 15% get B, the rest get C. Uh, there, there is some kind of underlying distribution that you have in mind for the intervention to work. Uh, yeah, so the sort of, the short answer is that it, it varies by these courses. So some of them have a strict curve, some of them don't necessarily stick to a curve, some of them have a curve that only works in one direction, right? They say like, we guarantee at least this many A's and B's. Um, I, I think, uh, You know, that this, this information is, is relative to the course that they are in, um, and, and sort of how they, uh, what the distribution of grades in that course means, right? I, I tell them like the the the median grade in bio or in econ 101 for STEM majors, so it's, it's, it's letting them compare where they are in that class, uh, to, to STEM majors who also took that class. Um, I, I can't necessarily speak to sort of, uh, maybe what it would look like in other settings. That's sort of an interesting question, but did that, did that get at your question? I, I, I'm just worrying that the distribution that people have in mind is not really like a bell shape. If professors give grades not in a bell shape, then you give, you give students information in terms of the percentiles may not make too much sense. Or do you give information on lesser grades? Perhaps that makes more sense, but I don't know. Yeah, so they, they sort of get both, right? Like in, in those, in those pictures, if, if you are here for that, right, they, they get the full distribution of the grades, and, and generally those look pretty bell shaped in in all of these courses, um, so they do see the full distribution, and then it points out their position in it, as well as the, the median, um, to sort of show them, you know, where, where the median falls relative to them. OK, thank you. OK, now, Ana, please. Uh, so Stephanie, there are a number of things that are floating in my head with these results. So one is overconfidence a bad thing? Uh, so here you see that the average student, both male and female, is overconfident. And the only reason there's a gender gap is because the female students are less overconfident than the male students. The situation would be very different if the gender gap was coming from the fact. That the female students were actually underconfident on average and the male students were on average overconfident. So I'm starting to see how, you know, the, the fact that the gender gap now reduces because, uh, the lower uh ability overconfident male students are primarily the ones who are adjusting. So I think it's good that underconfident, high ability or high performance males. Students also adjust upwards, but nothing is really going on with the female students. So I'm just trying to, so I'm curious to see what you find on student behavior, but would you be worried that if the overconfident, low performing male students actually have some negative impacts on them, uh, and maybe they would have actually done better if they continued to be a little bit overconfident. Yeah, so I'm, I'm gonna maybe defer this question because I have some discussion at the end, sort of getting at exactly this, sort of what it means for, for the women, what it means for the men who are discouraged and how we think about welfare, um, so, but, but please like raise your hand again. If I don't answer your question, um, but I, but I definitely want to discuss this because it's really important in thinking, like, sort of, was this a good intervention? Do we want to do this? Do we want to discourage certain people from, from STEM? So, I will come back to that. Sure, thanks. OK, go ahead, Stephanie, please. Great. Um, OK, so we saw these effects on beliefs that based on the way men updated and women updated, maybe we would expect men to take less STEM based on, on sort of correcting their overconfidence. So, uh, as I, as I mentioned earlier, I have a, I have a short term measure of field specialization or major choice, um, using the administrative data. Um, so my primary outcome here is the number of STEM credits that a student takes in the semester following the intervention, and as more time passes, I'll be able to measure their actual major declaration and the degree that they receive. So here's, here's the, the control students. So I, I mentioned briefly earlier that the, the gender difference between men and women is, is about 2 STEM credits, or men take about half of a STEM course more than women among the control group. The result of the intervention here is that men decrease their STEM credits by 0.28 credits, um, with, with no effect for women, um, and again, if to sort of help, help with that, that magnitude, um, you know, what does 0.28 credits mean? Um, so, you know, a single student can't take 0.28 fewer credits. This is an average, so this would be equivalent to about 120 men, or 7% of the treated men in my sample, taking one fewer 4 credit course. So you can sort of decide if you think that's large or small. I consider it fairly big for the sort of magnitude of the intervention. Um, and, and women, again, don't change their behavior, and so the result is that the gap in, in course taking, this two credit gap, closes by 10%. So, right to to sort of summarize the, the main results so far. I started by showing systematic gender differences in women's versus men's beliefs about relative performance in STEM, and so I showed that men overpredict their percentile rank by more. Um, men underestimate how STEM majors do, while women overestimate. And then I showed that a low-cost, light touch intervention narrowed gender gaps in beliefs by between 1/3 and 15%. Um, and specifically men updated about their own rank, as well as about others in a way that that corrected their overconfidence, um, and women updated about others, um, in a way that that updated their underconfidence. Um, and ultimately the intervention closed gaps in STEM persistence by 10%, um, but this happened because men took 3% fewer STEM credits, um, and no change for women. Um, so, since we have about 15 minutes left, um, I, I want to get into a little bit of, of this other stuff that I, that I have built in to, to dig into what's going on and get at some of the other questions. Um, So, uh, a couple of additional outcomes I look at to, to sort of dig into what's going on. So, um, I, I showed short term effects on course taking, but I'm ultimately interested in, in major choice, right? I, I don't observe it yet, but I can speculate, um, or use the data in a couple of ways to, to, to guess what might happen. So, the first I ask students directly in the survey, if they plan to major in STEM, so their subjective, um, major intention. Um, I also do an exercise, uh, where I, uh, I predict how much the changes in short-term course taking might translate into degree receipt, using a, a historical sample of students, and the, the, the relationship between course taking and major choice, and then the observed changes in course taking I see. And so both of those exercises, um, I find pretty small negative, statistically insignificant effects on on this possible longer term major choice for the for the men who are discouraged. Um, so, you know, I'll be able to measure this directly with with follow-up data, um, but, but it sort of implies that longer term effects are likely to be small. Um, you, you also might be wondering, you know, whether the intervention had any effect on students' effort or performance. Um, there's, there's a fair amount of, of literature about the effect of performance feedback on, on performance. Um, it, it could actually go either way, right? There might be a, a discouragement effect of receiving bad news, uh, but on the other hand, learning that you're sort of doing worse than you thought could, could sort of lead you to increase your effort to, to compensate. Um. I, I don't find any, any change in performance or effort on, um, on either the sort of final exam grade in the course, or the, the final course score. Um, and then finally, right, I, I showed that that men take less STEM, but, right, what, what do they take instead? Um, and so, you know, we might expect that that certain, uh, uh, They might, they might shift into other subjects, um, and that's sort of important for thinking about welfare. So, it, it seems like the, the men may have, have shifted into psychology and humanities courses instead, at least in the short term, um, but again, we'll, we'll sort of, um, I'll look at that in the long term as well. Um, and then, uh, you know, we, we also might expect that certain types of students' beliefs and behavior would be particularly susceptible to an information intervention. Um, so I also look at heterogeneity along a number of dimensions that we might expect to matter. So just sort of to quickly go over some of those, um. I, I look at the, uh, the initial size of the error that students had, as well as the direction based on that baseline survey to see if it's, um, if, if it's sort of the students who were the most biased to update the most, um, and, and that is what I find. So students with, with the largest initial errors update the most and, and in the correct direction. Um, so I, I. interpret that as, you know, students are updating in the direction of the truth. Um, I also find that, um, sort of the younger students, the first year students, um, those who haven't yet declared a major, and those who are already interested in STEM are the ones who change their behavior the most. So students that we might expect to be on the margin are the ones whose behavior changes. Um, and then finally, right, I, I have this, this set of STEM courses that, that actually look fairly different, right? We have, um, physics and engineering that are extremely male dominated, and then we have biology, which is actually female dominated. Um, I don't quite have the power to compare, uh, across class and across gender, um, but to try and get at this, I, I look at, um, how the effects vary by how male dominated the field is. Sort of the percent of women in the course, um, and, and it sort of seems like, uh, again, a bit, a bit noisy, but the men may have sort of adjusted their beliefs the most in the more male dominated fields. And so I interpret that as being consistent with, you know, men are more biased in the male dominated fields, possibly because of gender stereotypes, and they're therefore more susceptible to information. Um, all of these are, are a bit underpowered, as I, as I think I mentioned earlier, um, I had planned for, for twice the sample size, but unfortunately, uh, the second round was, was canceled. Um. OK, um, and then just a little bit of discussion, given, given time, I'm gonna skip the sort of context and the wider literature, but if you want to talk about other papers, I'm happy to. I'll get some of the sort of the other questions that have come up. Or, um, you know, sort of really fundamental questions that underlie this work and, and what I find, um, so the first is, is Stephanie, yeah, would you mind to take a question from Tuan because maybe it's related with the previous section. Sure, go ahead down, please. Yes, I, um, so, so I feel that, you know, looking at major choices is one aspect of, of decision making. And then there, there, there's another one that you, you talk about a little bit which is effort because if I realize my score is low, it, it might be due to some inner ability that might lead me to choose a different field. And also how much effort I'm going to put in the, in, in studying. And if we do this, you know, how much effort in my choice of measure will depend only on how much I'm going to adjust effort, but how much I also believe others are going to adjust. In terms of effort. So it's essential to me how, how, how, how you want to think about, about that margin, that kind of intensive margin of, of adjustment and, and the color is, is not finding an effect in your, in your test might be related to the fact that some people who decide to drop. Will decrease the threat while people who decide to stay will, will increase the effort level. So on that, it's unclear whether we should expect a positive or a negative. Yeah, so again, sorry, there's a bit of like interference. It was a bit hard to hear, but I, I think you were saying it's, it's, it's a bit hard to think about sort of what direction the effort response might be. Is, is that what you were saying? Um. No, I'm just saying one margin which is I think as important as the choice of the major or dropping in or not is how students who get information are going to adjust their effort into studying. And that aspect not only depends on their own action but what also they believe others are going to, others are going to put effort in. And so I, I, I'm not sure what religious form. Tests is, is, is going to, to, to, to correctly capture the impact of your intervention. Yeah, so, you know, I'm, I'm a bit limited. I, I have their, their final, um, their performance on the final exam in these courses, and then I have their final like raw score in the course. That's sort of my measure of, it's, it's not directly effort, it's sort of the, the output of effort and and some other factors, um, and I don't find, I don't find changes there, um, but if I was able to directly measure, say, you know, study hours or something like that, that would be interesting as well, but, um, I'm not able to. So Stephanie, there are 2 more questions. Do you want to take them now? Um, I, if it's OK, I might, I might push just so I can address some of these things which I think address other things that, that have come up, and then I'll leave some time at the end. Um, yeah, OK, sounds good, um. Yeah, so just, you know, sort of the the the fundamental questions that have come out of what I find, um, and, and one is sort of the formation of these beliefs or where the biases come from and why there are differences by gender in the first place. Um, that's a bit sort of outside the scope of what I do, but other work suggests that that students learn about their abilities through their prior academic performance in in both high school. School and college, um, and I, I find that myself, right? Like over time, in the absence of my intervention, students seem to be, be learning about themselves, um, but even controlling for all of these prior measures of performance doesn't explain the, the gender gap in beliefs. And, and so I think there's really a role for stereotypes in explaining where these beliefs come from. So there's work by by psychologists and, and behavioral. Economist about biased beliefs, how they form and how they persist, um, and in sort of like formal models of, of stereotypes, where without, you know, without full information, people use shortcuts, um, to, to, to form beliefs about groups or, or yourself as part of a group, um, and so exaggerated stereotypes like men being better at STEM can persist despite very small true differences. Um. And then I think, you know, another, another big question is, OK, if the beliefs about relative performance and underconfidence don't explain the women's decisions, like even updating their beliefs wasn't enough to change their STEM behavior, so, so what is, what sort of can move that decision for women? There's, um, you know, some sort of an explanation that often comes up is about comparative advantage, and if, uh, you know, even if they update their beliefs about STEM, the women are so much better in their non-STEM courses that it's not enough to sort of pull them into STEM. And um other work has, has, has suggested this, and I find some support for that in the data just given how much stronger women's performance is in non-STEM courses. Um, there's also some, some other work suggesting that non-academic factors like peer effects, role models, sort of information about the field, um, could affect women's decisions, that might matter for them. Um, it could also be that, you know, types of beliefs other than the, the precise ones that I targeted in this intervention might be the ones that really matter for women. Um, And then just, just to sort of, uh, get to, there was an earlier question that I that I punted, um, so I want to make sure I, I talk about this. So, uh, you know, the was, was this intervention good, essentially? How do we think about welfare, given that it, it works by discouraging some men from STEM? Um, and so there, there's a number of things that, that we want to think through and, and whether this was a, a welfare improving policy. Um, so, first is, you know, do these effects that I found on short term course taking persist to major declaration and degree, um, and, and if they do, right, what do men pursue instead, and what are the associated outcomes, um, with those other majors, um. It's also important to think about sort of the interaction between, uh, major choice and performance in a field, right? So, uh, is it actually better to be sort of a mediocre STEM major and and stay in STEM given the, the labor market returns, or is it better to be sort of a higher performing non-STEM major? Um, that's not actually something we really have much causal evidence on. Um. Another thing to consider, um, the, the information might have just shifted the timing of certain men leaving STEM, so they just exited STEM somewhat faster, which could be welfare improving for them. Um. And then there are a couple of important spillover effects that could be playing out. So, for example, um, if some, some of these majors in this setting are capacity constrained, and so men leaving could free up spots for others, could, could possibly, um, improve, you know, efficient sorting into these majors, um. And then finally, you know, we, we have a peer effects literature about the, the effects of, of having different types of students around you, right? So, it might be better just to have fewer men around for the women who are left behind. Um, there could also be peer effects of having sort of a different composition of peers who are somewhat, uh, less low achieving or less overconfident. Again, that could sort of the, the peer effects literature is a bit mixed on what direction we would expect. Um. OK. Uh, so I'll just wrap up, uh, in the last two minutes. And just sort of say that this this project represents the first experimental evidence from the field about biased performance beliefs and gender differences in STEM persistence and enrollment, so the results, sort of the way they played out. are a bit surprising and the welfare implications as I just went over, are somewhat ambiguous, but I'm advancing understanding of an important topic that we didn't have great causal evidence on before how students make decisions and how it varies by gender. Um, it's just as important to know what mechanisms are not important or what policies don't work for a given goal like retaining women in STEM. Um, if we're in, you know, a world with limited resources, we don't want to pour them all into something that, that won't work. Um, and this is important to know, given, you know, sort of the conversation we often hear about women, you know, women just need to lean in or advocate for themselves. Um, my work suggests that's not actually the, the solution. Um, so I'll just end by saying we need to continue to do careful research into additional mechanisms and policies that can improve diversity in STEM. Um, and I'll just say thank you and, and acknowledge that, um, I, I had some really fantastic research partners at the University of Michigan who helped me with this project. So thank you all. Um, if you have a question or two, I'm happy to stick around, um, before, before you jump off. Thank you so much.

showAllTranscripts

no

duration

PT1H13M22S

scene7File

worldbank/StephanieOwenCollege

scene7Domain

scene7FileAvs

worldbank/StephanieOwenCollege-AVS

title

StephanieOwenCollege

description

StephanieOwenCollege

showTimestampAndTranscript

yes