Policy Research Report Launch Event | Improving Effective Coverage in Health: Do Financial Incentives Work?

videoType

dynamic-media

videoDmUrl

keyFrameImage

timestamp

00:00 Excellent.

00:00 Thank you.

00:02 Well,

00:02 good morning,

00:03 good afternoon,

00:03 good evening,

00:04 everybody.

00:04 Uh,

00:05 my name is uh Dion Filmer.

00:07 I'm the director of the World Bank's Development Research Group.

00:09 And welcome to today's,

00:11 um,

00:12 which is the May edition of our policy research talk series.

00:16 I'd like to welcome our in-person audience as well as our

00:19 online audience both on Webex as well as on YouTube.

00:22 For those in the room,

00:23 we've been asked to please keep masks on unless speaking,

00:27 and the whole room is set up for social distancing,

00:29 so we're complying with all the rules and guidelines.

00:33 Um,

00:33 these policy research talks provide us with

00:35 an opportunity to present work coming out of

00:37 the research department here at the World Bank with the goal of sharing findings,

00:41 uh,

00:41 inside and outside

00:43 the department,

00:44 along with others outside the bank.

00:47 Today's policy research talk is a special occasion because

00:50 we're using it as an opportunity to launch uh,

00:53 our new policy research report,

00:56 Improving effective health coverage,

00:58 uh,

00:58 improving effective coverage in health.

01:00 Do financial incentives work?

01:03 The policy research reports are a product of the development research group

01:06 that have the aim of synthesizing the most

01:08 rigorous evidence available from within and beyond the bank

01:12 on important topics in development.

01:15 The reports benefit from our department's dual mandate of working closely with

01:18 bank operations on projects while also

01:20 producing cutting edge research on development.

01:24 This particular

01:25 report takes on an issue that's especially relevant in many low-income contexts.

01:30 It looks at the evidence on the impact of financial incentives in health

01:34 over 15 years and nearly 40 countries.

01:37 Two of the lead authors,

01:40 Damien De Walk

01:41 and Ishahi Kanpal,

01:43 will be

01:43 co-presenting today.

01:45 Damien is a lead economist

01:47 in our human development

01:49 team,

01:50 and his work is currently focused on the impact

01:51 of financial incentives on health and education outcomes.

01:55 Ishani is a senior economist in our poverty and inequality team.

02:00 And conducts research on financial incentives

02:02 like cash transfers to poor households

02:04 and pay for performance contracts with health workers.

02:08 We're grateful to have Norbert Shady and Feng Zhao

02:12 as discussants today.

02:13 Norbert is chief economist for Human Development at the World Bank Group.

02:17 He's published extensively in academic journals,

02:19 and his main research topic areas include early childhood development,

02:22 teacher quality,

02:23 cash transfer programs,

02:24 and the effect of economic

02:26 contractions on the accumulation of human capital.

02:29 Feng is the strategy operations and global engagement

02:33 practice manager here at the World Bank.

02:36 As World Bank practice manager,

02:37 he oversees the Bank Global Health engagement program

02:41 and has more than 20 years of experience in public health,

02:43 medicine,

02:44 economics,

02:45 and demography.

02:46 I'll ask uh Damien Nanni to speak for about 40 minutes,

02:50 after which we'll hear from Norbert and Fen

02:53 for approximately 5 to 10 minutes each.

02:55 We'll conclude the session with questions and answers from,

02:58 uh,

02:58 the in-person and online audiences.

03:01 If you have a question in person,

03:03 uh,

03:03 we'll ask you to come to the microphone

03:05 at the start of the Q&A.

03:07 If you're watching online and have a question,

03:09 please use the raised hand option in Webex,

03:12 and I will call on you.

03:14 Um,

03:14 if you're following on YouTube,

03:16 please submit your question in the chat,

03:18 and that'll get relayed to us.

03:20 Uh,

03:20 just a reminder,

03:21 uh,

03:22 we are recording

03:23 the session and if you're following online,

03:25 please do mute your microphone unless you're speaking.

03:28 So with that,

03:29 over to you,

03:29 Damien Anushani.

03:31 Thank you.

03:37 Um,

03:38 thank you very much,

03:39 uh,

03:42 uh,

03:42 and for all of you for coming,

03:43 uh,

03:44 today,

03:45 uh,

03:45 in person or,

03:47 uh,

03:47 following on Webex

03:49 on YouTube.

03:49 So it's a pleasure for

03:51 Ishani and myself to,

03:53 uh,

03:53 present you,

03:54 uh,

03:55 this report,

03:56 this,

03:56 uh,

03:56 uh,

03:57 policy research report

03:59 entitled Improving Effective Coverage in Health.

04:02 Do Financial incentives

04:04 Work?

04:07 So,

04:07 I'd like to start by saying this is uh the work of a large team

04:11 composed of colleagues in the research department at the World Bank as well as

04:15 in the health,

04:16 nutrition and population

04:18 uh global practice at the World Bank.

04:21 Um,

04:21 and,

04:22 uh,

04:22 in addition to which Shani and myself,

04:24 the team includes

04:25 Adam Wagstaff,

04:27 Jett Friedman,

04:28 Sven Nielsen,

04:29 Mauritz Piati Ffkirchen,

04:31 Anel Soutmann.

04:32 Gil Shapira

04:34 and Helene Vanderpool,

04:35 and recently we took this team picture outside of the bank,

04:38 but since uh Ellen is uh sitting in Dakar,

04:42 we had to uh Photoshop her

04:44 in the picture.

04:47 We also want to take this opportunity

04:50 to dedicate this report to the

04:53 memory and legacy of Adam Wagstaff.

04:57 He was the driving force behind this report.

05:01 He contributed numerous concepts and analysis.

05:05 And as you,

05:07 as many know,

05:08 um,

05:10 he untimely passed away

05:13 two years ago.

05:14 He still remains

05:16 an inspiration

05:18 for us,

05:18 but we are missing him.

05:25 So now I want to start with

05:27 the big picture that motivates

05:29 uh this report.

05:31 And

05:33 this big picture,

05:34 we call it a frustrating status quo,

05:36 status quo.

05:37 So we are going to illustrate this frustrating status quo with

05:41 um

05:43 the example of pregnancies.

05:46 And what we see around the world

05:48 is that pretty much everywhere,

05:50 everywhere,

05:51 there are high rates of antenatal care visits around the world.

05:55 That is.

05:56 Most women

05:58 who are pregnancy at least once.

06:01 A service provider,

06:02 they have an antenatal care check.

06:04 On this

06:06 map

06:06 you see that the dark blue is more than 90%.

06:10 The countries where more than 90% of women

06:12 have an

06:13 antenatal care check.

06:14 It's a lot of countries

06:15 and the other countries are also

06:17 in this darker shade of blue,

06:19 above 80% or 70%.

06:23 But despite that,

06:25 What we see

06:27 is that at the same time that there is

06:30 Widespread access to antenatal

06:33 antenatal care.

06:35 We still find

06:37 high maternal mortality rates

06:39 in many low income countries.

06:41 So this is

06:42 on this map you see maternal deaths

06:45 per 100,000 live births.

06:48 The dark blues are a country where maternal death is.

06:52 Very low

06:54 And the colors that are orange or

06:58 red show you countries

07:00 which are mainly known in West Africa or Central Africa,

07:04 but also

07:05 in Afghanistan where we still have

07:08 shockingly and stubbornly high

07:10 maternal

07:11 mortality rates.

07:14 And this is this is

07:16 true not only

07:18 for

07:19 maternal health,

07:20 but for a lot of

07:21 other health outcomes.

07:23 Indeed,

07:23 the Lancet Commission on High Quality

07:26 Systems conclude that most deaths in low income countries

07:31 are due to the poor quality of care

07:33 rather than the lack of access

07:35 to care.

07:36 There are still important gaps in equity in access to care.

07:40 And most of,

07:42 a lot of uh primary health care systems are still very fragile.

07:47 And so now as we

07:48 are hopefully

07:50 coming out of this pandemic,

07:52 and if we think about,

07:54 you know,

07:54 regaining

07:55 the gains that occurred before the pandemic,

07:57 we know

07:58 there's been

07:59 further decline or there's been decline during the pandemic.

08:02 And as we want

08:03 also to go beyond the gains that we had before the pandemic,

08:07 we think we really need

08:09 to focus on

08:10 quality of care.

08:11 So that's the motivation for this uh report.

08:15 And if we think about

08:17 quality of care,

08:19 one

08:20 problem that as economists we call the principal agent problem.

08:25 It's present in health,

08:26 but also it's common in many markets,

08:29 especially in the public sector.

08:30 What is the principal agent

08:32 problem

08:33 is that

08:34 You have

08:35 an employer,

08:36 in the case of health would be the Ministry of Health,

08:38 for example.

08:40 Who does not observe the performance of or effort by the worker.

08:44 In the example of health would be doctors or nurses

08:48 or in schools it would be teachers,

08:50 and that can lead

08:52 to shirking by the worker or lack of effort.

08:55 But at the same time,

08:56 the employer can observe the outcome.

09:00 But that outcome is not.

09:02 Determined entirely

09:04 by the effort of the worker,

09:07 we know that health outcome,

09:08 even with good treatment and

09:11 can

09:12 Go bad.

09:13 So that's the,

09:14 the main issue of the principal agent problem.

09:19 Faced with this

09:21 problem,

09:23 one solution

09:24 has been

09:25 performance pay.

09:27 It's like,

09:28 so paying for performance is,

09:29 is very common.

09:31 Um,

09:32 to deal with the principal agent problem,

09:34 and it basically

09:36 links

09:37 prices

09:38 to actions.

09:40 It's used

09:42 Across many public sector bureaucracies

09:45 with applications

09:47 in education,

09:48 infrastructure,

09:49 agriculture,

09:50 public-private partnership,

09:52 contracting with nonprofit,

09:53 and

09:53 in-house,

09:54 and we are going to focus

09:55 on the application in-house today.

09:58 The goal of this approach is to improve the efficiency

10:02 of public expenditure.

10:04 And as I said,

10:05 performance pay is based

10:06 on the quantity and the quality of service delivered,

10:11 and we'll we'll give some examples,

10:12 and we want to stress that

10:14 the introduction of performance pay

10:16 was a paradigmatic shift from input-based financing,

10:20 which was

10:21 basically the status quo before,

10:23 towards more transparency,

10:25 transparency and accountability.

10:28 Health

10:29 is one example,

10:30 an important one,

10:31 but there's a big push across sectors

10:34 for donor as well as domestic resources

10:38 to be shifted towards a performance pay uh mechanism.

10:44 So what we are going to look in this report is that

10:47 performance pay

10:49 and performance-based financing

10:51 abbreviated by

10:52 the acronym PBF Projects in Health.

10:56 Let me first

10:57 stress

10:58 that PBF.

11:00 Project performance-based project and performance.

11:03 They are not one and the same.

11:05 PBF are projects that actually

11:07 deploy a package intervention,

11:09 and that includes

11:11 performance pay indeed,

11:12 and that's one

11:13 admittedly crucial component of PBF packages,

11:16 but also other

11:18 components in that package.

11:20 More autonomy for the frontline health facilities,

11:23 they can decide

11:25 more about how they use their own budgets,

11:27 etc.

11:28 more transparency and accountability

11:31 and more community engagement,

11:32 and these are all

11:34 also related

11:35 to

11:37 substantial effects

11:38 on the financial management

11:42 of uh of the health system.

11:46 So these reforms,

11:48 these performance-based financing packages

11:52 have been

11:53 supported.

11:54 By

11:55 a very large

11:57 trust fund managed by the World Bank

11:59 called the Health Results Innovation Trust Fund.

12:03 It was mainly invested in performance-based financing approaches

12:07 for maternal and child healthcare

12:10 in low-income countries.

12:12 It spans 15 years

12:14 and 40 countries and one very strong.

12:17 element of that trust fund is that it not only supported

12:22 many projects but also

12:24 many

12:25 rigorous impact evaluations,

12:26 and we are building on the evidence from these rigorous impact evaluations

12:30 to

12:31 to produce this report.

12:34 It was also the opportunity for a sustained and fruitful collaboration

12:38 between the research department here at the World Bank and

12:41 the teams in the health sector at the World Bank.

12:44 So what have we learned from,

12:46 from these pilots?

12:47 That's what we are going to try to

12:49 answer

12:51 today.

12:52 So,

12:53 and,

12:53 you know,

12:54 I could,

12:55 and I've used that,

12:56 that

12:58 picture before.

12:59 Uh,

13:00 you know,

13:02 you all know this glass of water,

13:03 and if you're an optimist,

13:04 it's half full,

13:05 and if you are a pessimist,

13:06 it's half empty,

13:07 and when you have an impact evaluation,

13:09 you have some.

13:11 Uh,

13:11 significant outcome,

13:12 positive and significant outcomes

13:14 for some,

13:15 uh,

13:16 for some of your results,

13:17 but

13:18 some of the outcomes you were expecting some results

13:20 and you don't find them and you conclude mixed results

13:23 and,

13:24 and then you show this picture.

13:26 But what we want to do,

13:28 uh,

13:28 today is go beyond that.

13:30 We want to go beyond that um

13:33 cliche of the glass half full and half empty

13:36 and let you pick and decide whether

13:38 you are optimist or pessimist.

13:40 So

13:41 more specifically we want to look

13:43 at the content

13:45 um

13:46 of our findings

13:47 and what we are going to do in this report

13:50 is look at these five questions.

13:53 First,

13:53 what does the evidence say

13:56 about performance pay in low-income country health system?

13:59 We're going to assess the impacts on coverage

14:02 and effective coverage.

14:04 I will

14:05 give you a more precise definition of what we mean by effective coverage,

14:08 but basically think for it,

14:10 for not think of it as health coverage

14:13 with a minimum content and

14:15 quality,

14:16 and we are going to look at this for maternal and child health services.

14:21 Second question is,

14:22 what are the key,

14:23 key constraints to quality?

14:25 Which constraint to quality can be addressed by performance pay.

14:30 And can we realistically expect performance pay to improve quality,

14:34 and if yes,

14:35 by how much?

14:37 Third question,

14:38 can demand-side approaches improve coverage and effective coverage?

14:43 Of maternal and child health services.

14:46 PBF performance-based financing

14:48 is a supply side approach,

14:49 but we can also act on the demand side,

14:51 and we should also look at that.

14:54 Fourth question

14:55 is how do

14:57 this performance-based financing project compare to what we will call

15:01 and define shortly

15:03 direct facility financing abbreviated by DFF.

15:07 And finally,

15:08 what are the key lessons

15:09 that we can learn from this evidence for the design

15:12 of new health financing reforms?

15:16 Briefly,

15:17 uh,

15:18 giving you the snapshot of the geographic scope of the report,

15:21 as you can see it's quite

15:24 broad.

15:25 Even though for some of the uh section we are going

15:28 to focus on uh uh set of uh countries in sub-Saharan

15:34 Africa.

15:35 So let me go back to this

15:37 distinction.

15:39 Uh,

15:39 Between performance-based financing,

15:41 PBF

15:43 and

15:43 direct facility financing,

15:45 DFF,

15:46 and on the left

15:47 you have the four components of PBF

15:50 that I've already covered

15:52 performance pay,

15:53 operating budgets,

15:54 and autonomy,

15:56 transparency and accountability.

15:59 And uh community engagement.

16:01 Well,

16:02 if you want.

16:04 To know what is the effect it's basically the same.

16:07 With the exception of performance pay,

16:09 there is no mechanism

16:11 tying

16:12 the payment of the health facility and the health worker

16:15 to

16:16 the quantities and quality delivered,

16:18 but the three other components

16:20 are there.

16:22 With this in mind,

16:24 let me give you

16:25 a sneak peek at our key takeaways.

16:28 The first one

16:30 is that overall we find little evidence for impacts

16:34 for across the board

16:36 performance pay in under-resourced and unfinanced

16:39 health systems.

16:40 Instead,

16:41 we find that direct facility financing that I just defined

16:46 with autonomy and accountability

16:48 at the

16:49 level of the frontline facilities

16:52 can deliver many gains

16:54 at lower cost

16:55 and with relatively easier implementation.

16:58 So what we suggest is that,

17:00 is that for

17:02 teams working in,

17:02 in health reform is that before

17:05 designing health financing reform.

17:08 They should do

17:09 an assessment in 3 steps.

17:11 First,

17:12 assess coverage versus

17:14 effective coverage

17:16 for their indicator of interest to identify

17:19 where would lie the low hanging fruit

17:22 for performance pay.

17:25 Second,

17:26 assess the constraint to quality of care

17:28 that would ensure.

17:30 Uh,

17:31 that they are

17:33 in the locus of control for the of the frontline worker,

17:36 that is,

17:36 the frontline worker can actually

17:38 act

17:39 on these

17:40 barriers.

17:41 Third.

17:43 They should find

17:44 that sweet spot where baseline utilization should have

17:49 some room for improvement,

17:50 but not be so low

17:52 as to indicate that there are demand side barriers,

17:56 because when there are demand side barriers,

17:58 it would be best

18:00 to include demand side approach and so we suggest often

18:03 that it can be useful

18:05 to combine demand and supply side approaches.

18:08 And finally,

18:09 we suggest also to sequence intervention

18:12 and use performance pay strategically.

18:16 So now I'll

18:18 come back to this concept of

18:20 effective coverage and try to motivate it and

18:23 also give you

18:24 a more complete definition.

18:27 So what is effective coverage?

18:29 It is health coverage

18:31 that includes a minimum content and quality of care.

18:35 And you can think of it as the product of coverage

18:39 and quality.

18:40 We we've

18:41 tried to show that in that very simple

18:43 equation

18:44 where effective coverage is basically

18:47 the product of coverage,

18:49 that is the percentage of person in need of care

18:52 for a specific medical condition X

18:55 who actually

18:56 consult,

18:57 that's coverage times

18:59 the percentage of people

19:01 consulting

19:02 who get

19:03 relevant care.

19:05 In the report,

19:05 we operationalize effective coverage by investigating,

19:08 investigating,

19:09 sorry,

19:10 how to measure it using existing data.

19:12 What we do is use household survey.

19:15 From,

19:16 uh,

19:16 for 93 countries

19:19 and we show effective our effective coverage varies by country,

19:22 by wealth level

19:23 of the patient,

19:24 and also

19:26 across different

19:27 medical conditions.

19:28 We want to stress also that

19:31 When we look at quality of care,

19:32 we,

19:33 we,

19:33 we not only looked at insufficient care,

19:35 whether not enough is done,

19:37 but also at irrelevant care or overuse.

19:40 When something is done that should not be done,

19:42 either it's a waste or even it can be harmful.

19:45 So we look at both

19:47 issues

19:48 when we look at quality.

19:51 Let me give you

19:52 one example,

19:53 one illustration

19:54 of

19:55 effective coverage

19:57 using the example of antenatal care.

20:00 In this figure,

20:02 on the horizontal axis,

20:04 you have coverage going from 0 to 100,

20:07 and it's defined as the person of woman giving birth with at least one

20:11 antenatal care

20:13 visit.

20:14 On the vertical axis

20:15 with quality of care also going

20:18 between 0 and 100,

20:20 and that's among these women who got at least one antenatal care visit,

20:24 the percentage who had

20:26 correct treatment according to WHO guidelines,

20:30 which is defined as the percentage who had at least 4 visits.

20:35 One of these visits with a skilled provider

20:37 and for whom

20:39 blood pressure was taken and blood and urine samples were taken.

20:43 So

20:44 then we can

20:46 Draw

20:47 what we call effective contours,

20:49 lines of effective coverage,

20:50 and these are,

20:52 they are here for illustrative purpose,

20:54 but they are basically lines

20:55 that

20:56 show

20:57 different points of coverage and quality that gives you the same

21:01 effective coverage.

21:03 OK,

21:03 so let's look at 6 countries.

21:06 5 sorry in Africa

21:08 to see

21:10 what the data tells us if we look at the Republic of Congo.

21:14 We see

21:15 a high level of coverage.

21:17 Above 90%

21:19 and fairly high level of qualities,

21:22 around 80%,

21:23 that gives us something

21:26 around 70% of effective coverage.

21:28 Now if we look at the two next countries,

21:30 Cameroon and Nigeria,

21:32 coverage is somewhat

21:34 lower,

21:35 between 70 and 85%,

21:37 but quality is also markedly lower.

21:40 It's

21:41 around 60%,

21:42 and that brings us to a lower level

21:45 of effective coverage.

21:47 And then the next two countries,

21:49 the Democratic Republic of Congo and the Central African Republic,

21:53 they've,

21:53 if you look at it,

21:54 they have similar levels of coverage compared

21:57 to Cameroon and Nigeria,

21:58 but their level of quality

22:00 is much lower.

22:01 It's below 40%.

22:03 And as a result,

22:04 we see that effective coverage

22:07 is basically under

22:09 30.

22:10 So with that illustration,

22:12 I'll give the floor to Eshani.

22:15 Who is going to uh,

22:16 go more in detail into the quality of maternal and child health.

22:21 Thank you.

22:22 So as,

22:22 um,

22:23 as Damien sort of motivated,

22:24 there's this gap between the coverage of maternal and child

22:27 health services and the quality or the effective coverage,

22:30 pointing to serious problems with quality in many of the

22:33 contexts that we're gonna be talking about later today.

22:36 Um,

22:36 so what we do is

22:38 to try and understand,

22:39 you know,

22:39 how you improve quality.

22:40 As a first step,

22:41 we try to understand what the constraints of quality are.

22:44 In other words,

22:45 why the rate of relevant treatment,

22:47 uh,

22:47 is not 100%.

22:49 The literature often,

22:50 um,

22:51 points to

22:52 health worker effort or the lack thereof

22:55 as the binding constraint to quality in these sorts of settings.

22:58 Um,

22:59 if this is true,

23:00 that,

23:00 you know,

23:00 health workers are sort of,

23:01 uh,

23:01 leaving money on the table by not doing everything they could do,

23:05 and crucially,

23:06 if effort is observable and it's contractable,

23:09 uh,

23:09 then performance pay might make sense.

23:11 Performance pay to the health worker might in fact improve their effort.

23:15 But is that the case?

23:16 In fact,

23:17 you know,

23:17 if we think about it and we sort of systematically decompose quality,

23:20 what we find is that the constraints to quality

23:22 actually may lie at several different levels.

23:25 The first is that uh of the health facility or perhaps at a higher level

23:30 in the form of problems with equipment availability,

23:32 drug availability,

23:34 uh,

23:34 availability of supplies.

23:36 Now,

23:36 you might say this should be in the locus of control of the health facility,

23:39 but note that in,

23:40 in centralized systems,

23:41 you often have to go through centralized procurement

23:44 to address these sorts of issues.

23:45 So this might not even be in the locus of control of the facility,

23:48 leave alone the health worker.

23:49 It might lie further upstream in the medical system in the form of medical training,

23:53 problematic medical training.

23:54 So,

23:55 you know,

23:55 that's it with medical schools,

23:57 and there's typically a centralized syllabus of what's,

23:59 what's taught and how it's taught.

24:01 Uh,

24:02 there's,

24:02 of course,

24:02 issues of the public expenditure tracking system,

24:05 uh,

24:05 which can lead to,

24:06 if there are significant delays in,

24:08 uh,

24:08 salaries being disbursed,

24:09 problems of worker absenteeism.

24:11 So,

24:11 if,

24:11 you know,

24:12 if workers don't even show up at the health worker,

24:13 uh,

24:14 at the health facility.

24:15 Then,

24:15 of course,

24:16 there is

24:17 health worker effort,

24:18 which is the component

24:20 that is malleable to performance pay that's given to the health worker,

24:23 right?

24:25 And then finally,

24:26 there's potentially patient-driven demand,

24:28 which may combine or may not combine with the workers' effort,

24:32 but likely depends on the patient's uh socioeconomic status.

24:34 And,

24:34 and we'll touch upon all of these,

24:36 uh,

24:37 aspects at various points today,

24:39 but

24:39 The main,

24:40 the main message I want you to take away from this is that not all of the constraints

24:44 are in fact controlled by frontline facilities or workers.

24:48 And so there may be important limits at the very outset to what performance pay

24:52 that's designed to go to the health worker to improve their effort could

24:55 even achieve in these sorts of settings where there are competing constraints.

24:59 So to formalize our thinking,

25:01 uh,

25:01 what we do is,

25:02 uh,

25:02 use the no can-do gap framework to decompose constraints to quality

25:06 into the following three gaps.

25:08 The first is,

25:09 uh,

25:09 structural capacity gaps,

25:11 which,

25:11 uh,

25:12 which,

25:12 you know,

25:12 which tell us about what's available at the health facility

25:14 and for which we need data on health facility infrastructure,

25:17 so equipment,

25:17 drugs,

25:18 and supplies.

25:19 We need,

25:20 um,

25:21 knowledge gaps,

25:22 uh,

25:22 uh,

25:23 to assess knowledge gaps,

25:24 we need data,

25:25 uh,

25:25 assessing health workers' uh,

25:27 knowledge of clinical protocol for the types

25:29 of care that we're interested in studying.

25:31 Uh,

25:32 for this,

25:32 we typically use vignettes with health workers.

25:35 Um,

25:35 and then we can put these things together,

25:37 once we have direct observations of patient-provider interactions

25:41 to assess underused capacity,

25:43 sometimes also called no can-do gaps,

25:45 which is care that could have been provided,

25:47 but is not.

25:48 And this is again,

25:48 the component of care

25:50 that is uh malleable or amenable to performance pay directed to the health worker.

25:55 Um,

25:55 this is a data-intensive exercise,

25:57 as you can see,

25:57 but we can quantify these gaps for antenatal

26:00 care consultations in five Sub-Saharan African countries,

26:03 which are Cameroon,

26:04 the Central African Republic,

26:06 the Democratic Republic of Congo,

26:07 Nigeria,

26:08 and the Republic of Congo.

26:10 Um,

26:10 using this,

26:11 um,

26:12 this large HRITF impact evaluation portfolio that,

26:14 uh,

26:15 Damien has described,

26:16 we,

26:17 uh,

26:17 we can,

26:17 uh,

26:18 leverage really rich data,

26:19 sort of uniquely rich data on

26:21 the patient,

26:22 on the provider,

26:23 and on the facility,

26:24 characteristics of all of these three things,

26:26 as well as patient-provider interactions.

26:28 And what we do to benchmark the quality of care,

26:30 to say that this is what should have been done,

26:32 is use the WHO protocol for antenatal care.

26:35 Um,

26:35 and I'll,

26:35 I'll talk about that in,

26:37 um,

26:38 Uh,

26:38 a little bit more detail in the next slide,

26:40 but we'll,

26:40 uh,

26:40 one other thing I want to note is that we assess two different manifestations

26:44 of poor quality.

26:45 The first is underperformance,

26:46 which is where the health worker does only a share

26:49 of what she should be doing in,

26:50 uh,

26:51 a patient-provider,

26:52 uh,

26:52 consultation for antenatal care.

26:54 The second is overtreatment or irrelevant care,

26:57 that should not be provided.

26:59 Um,

26:59 and,

27:00 uh,

27:00 the,

27:00 the,

27:00 there are two interesting things to note here.

27:02 Um,

27:03 the first is that these coexist.

27:04 So,

27:04 at the same time,

27:05 health workers aren't doing everything that they're supposed to be doing,

27:07 and they're doing things that they're not supposed to be doing.

27:10 Uh,

27:10 and the other thing is,

27:11 uh,

27:12 that

27:12 some of the measures of irrelevant care,

27:14 which I'm really not gonna have time to talk about today,

27:16 but you can read about in the report,

27:18 are actually harmful to the,

27:19 the growing fetus,

27:20 and they're still being done,

27:21 uh,

27:22 in these settings.

27:23 OK.

27:24 So the first um key finding that I want

27:26 to show you here is that structural capacity gaps,

27:29 knowledge gaps,

27:29 and underused capacity

27:31 coexist

27:32 in these antenatal care consultations in all five of the countries

27:36 that we examine.

27:37 On your left here.

27:40 Left,

27:40 yes,

27:40 on your left here,

27:41 uh,

27:42 is a snapshot of the essential equipment,

27:45 drugs,

27:45 and supplies

27:46 that are needed to perform,

27:48 um,

27:49 that antenatal care visit as per protocol.

27:52 So on the Y-axis,

27:53 you have the,

27:53 um,

27:55 Some of the,

27:55 the,

27:56 uh,

27:56 the,

27:56 uh,

27:57 the physical infrastructure that's required,

27:59 and on the X-axis is the percent of facilities

28:01 that had this item in stock and correctly stored

28:03 on the day of the survey.

28:05 And the main thing to note here is that,

28:06 you know,

28:07 these are facilities,

28:07 frontline facilities that are supposed to and,

28:09 and do in fact provide antenatal care on a routine basis.

28:13 And yet even for

28:14 these basic uh elements of care,

28:16 um.

28:17 Uh,

28:17 elements of infrastructure,

28:18 um,

28:19 there's far from universal availability and you can,

28:22 uh,

28:22 you know,

28:22 look at the report for the full set of,

28:24 uh,

28:24 um,

28:25 structural capacity,

28:26 uh,

28:26 gaps.

28:27 On the right here is where we put things together.

28:30 I,

28:30 I'm not gonna go into knowledge gaps,

28:31 but you can again look at the report to see

28:33 how knowledge gaps do exist for every aspect of care,

28:35 but putting it all together on the right here

28:37 is underused capacity,

28:38 and what we find here,

28:39 again,

28:39 for just a snapshot of the actions,

28:42 uh,

28:42 but,

28:42 uh,

28:42 in the report,

28:43 uh,

28:43 we report,

28:44 um,

28:45 underuse capacity for all 16 actions on the WHO protocol.

28:48 The first thing to note is that yes,

28:50 there is underused capacity for every single aspect of,

28:53 of care,

28:53 for every single task that should be performed by the provider.

28:57 That said,

28:58 the highest

28:59 level of underused capacity anywhere is actually for counseling

29:02 for which you really don't need any infrastructure.

29:04 And there,

29:04 you know,

29:05 we have an average underuse capacity of 50%,

29:07 but

29:07 overall,

29:08 underused capacity only explains a third

29:10 of observed underperformance,

29:12 which is to say that 2/3 of poor quality

29:15 are not actually attributable to poor worker effort.

29:18 So there are many competing constraints in these settings,

29:20 and performance pay is therefore unlikely to be a silver bullet in isolation.

29:25 We also find that wealthier women are 3 times as likely

29:30 to receive high-quality care as,

29:31 as,

29:32 uh,

29:32 poorer women.

29:34 The part of this is driven by facility choice and so there's lots of within-country,

29:38 uh,

29:38 variation in facility quality and so,

29:40 shoring up,

29:41 you know,

29:41 the worst

29:42 facilities within a country is,

29:43 is,

29:43 is a likely,

29:44 uh,

29:45 is likely,

29:45 uh,

29:45 a worthy investment.

29:47 But we also have some evidence,

29:48 uh,

29:48 of within facility inequality,

29:50 which is consistent with,

29:51 although not the same thing as

29:53 discrimination by the,

29:54 by the provider,

29:54 but that we do see that,

29:56 uh,

29:56 which may point to the role of user

29:57 fees and patient information and interventions designed to,

30:00 uh,

30:01 to address these,

30:01 uh,

30:02 uh,

30:02 issues there.

30:04 But to summarize,

30:05 you know,

30:05 um,

30:06 what I want to leave you with for this section of the report

30:08 is that health system financing does appear to be a central challenge,

30:11 but it has to affect address still,

30:14 uh,

30:14 despite decades of sustained investments,

30:16 uh,

30:17 gaps in structural capacity,

30:19 in medical training,

30:20 and at the same time,

30:21 worker effort.

30:23 So,

30:23 to do so,

30:23 to design a sort of a robust health system,

30:26 uh,

30:26 financing reform,

30:27 you know,

30:28 we suggest doing things like tracking coverage versus effective coverage ex ante.

30:32 And you can use,

30:33 as Damien showed,

30:34 available household surveys to do this,

30:36 to also assess the constraints to quality

30:38 before along these lines,

30:39 this sort of a decomposition before designing

30:41 interventions that aim to improve quality.

30:44 So,

30:44 let's look at what sorts of interventions have

30:46 been designed to uh improve effective coverage.

30:50 You know,

30:50 so the literature and performance pays,

30:52 is quite well established,

30:53 uh,

30:53 in,

30:54 in the health sector,

30:55 uh,

30:55 with evidence from high-income countries.

30:57 Uh,

30:57 this,

30:58 these,

30:58 uh,

30:59 performance pay interventions,

31:00 which include the US's Medicaid system and

31:02 the United Kingdom's National Health System.

31:04 Um,

31:05 uh,

31:05 focus on clinical quality.

31:07 These are high-quality systems that have few other competing constraints,

31:10 uh,

31:10 other than work or effort that the performance pay

31:13 is attempting to address.

31:15 These are decentralized,

31:16 well-resourced systems.

31:17 Even so,

31:18 the literature is at best,

31:19 uh,

31:19 sort of mixed in terms of impacts of performance pay,

31:23 particularly

31:23 on clinical quality.

31:25 Furthermore,

31:26 what we're gonna be talking about today are applications

31:28 in a very different type of ah setting.

31:30 So,

31:31 low-income settings typically tend to be centralized.

31:33 They don't have operating budgets to facilities.

31:35 There's no autonomy over facility management and staffing,

31:38 and

31:39 the incentives,

31:40 as Damien has described in these performance-based financing programs,

31:44 uh,

31:45 pair,

31:46 uh,

31:46 incentives to the facility for,

31:48 uh,

31:48 structural improvements because that's an,

31:50 uh,

31:50 that's a,

31:51 that's a known constraint that was,

31:52 uh,

31:53 also,

31:53 uh,

31:53 attempted to be addressed with these interventions and at the same time,

31:57 worker incentives,

31:58 performance pay to the worker.

32:00 The early evidence of PBF projects from these settings,

32:02 from Rwanda and Burundi in particular,

32:04 was,

32:04 was quite promising.

32:07 Um,

32:10 Interesting.

32:10 I'm getting an error message here.

32:12 OK,

32:13 there we go.

32:14 Um,

32:15 so in terms of the key features of the PBF projects,

32:18 as,

32:18 as Damien described,

32:19 there are 4.

32:20 let me just give you a,

32:21 a concrete example of how that performance pay works.

32:23 So this is an example for

32:25 Nigeria.

32:25 Uh,

32:26 so it paid

32:27 $12 for every woman who gave birth in a,

32:29 in a health center,

32:31 $1.20 for an antenatal care visit,

32:33 and 80 cents for child immunization.

32:35 It also provided unconditional financing for everything

32:38 but staff remuneration to the facility and,

32:40 and gave the facilities autonomy

32:41 over how to spend that.

32:43 Um,

32:43 facilities could then,

32:45 um,

32:45 report,

32:46 uh,

32:46 performance on these purchase services typically every month,

32:49 and then payments were based,

32:50 uh,

32:51 on these reports.

32:52 In addition,

32:53 there was a third-party audit,

32:54 typically every quarter,

32:56 uh,

32:57 for,

32:57 uh,

32:57 of these,

32:58 uh,

32:58 of these reports.

32:59 And this,

32:59 you know,

32:59 this is expensive to implement because you need

33:01 the auditor to go to every frontline facility.

33:04 So,

33:04 for example,

33:04 in Nigeria,

33:05 1100 facilities participated in only the pilot.

33:08 Um,

33:09 and so,

33:09 you know,

33:09 at scale,

33:10 that's,

33:10 that's many,

33:11 many more facilities,

33:12 that's expensive,

33:12 that adds up,

33:13 but it also creates,

33:14 uh,

33:14 uh,

33:15 an opportunity for gaming or collusion.

33:17 And finally,

33:18 there's a community engagement.

33:20 Uh,

33:20 component wherein ward or village development

33:22 committees sometimes attend the facility management,

33:25 uh,

33:25 uh,

33:25 meetings.

33:27 The scope of evidence that we're gonna focus on

33:29 largely is uh from Sub-Saharan Africa for this component.

33:32 Uh,

33:32 and these are by and large,

33:34 the first generation of PBF projects that use this sort of

33:36 uh price checklist that I just gave you an example of.

33:39 But we do,

33:40 um,

33:41 spotlight evidence from high and middle-income countries

33:44 and later-stage PBF designs that I'll come to towards the end of my time today,

33:47 uh,

33:47 that use other approaches,

33:49 uh,

33:49 or,

33:49 you know,

33:49 combination of approaches to improve quality.

33:52 And again,

33:52 we're mostly gonna focus on maternal and child health,

33:54 but we do also touch upon curative care and non-communicable diseases,

33:58 with the hope of answering the question,

34:00 you know,

34:00 what is it that we learned from these donor-financed pilots

34:03 about at-scale health financing uh reform

34:06 that would use domestic resources?

34:08 To answer this,

34:09 the first comparison we are gonna look at is PBF versus business as usual.

34:13 Um,

34:14 I'm not really gonna go into the coverage impacts of PBF,

34:17 uh,

34:17 projects,

34:18 but,

34:18 um,

34:19 uh,

34:19 the recent Cochrane review by Sophie Witter and others,

34:22 uh,

34:23 finds that at least in,

34:24 in,

34:24 in most contexts,

34:26 there's some improvements in terms of simple coverage,

34:28 um,

34:29 although there is a lot of heterogeneity in the impacts here.

34:32 In terms of quality,

34:33 uh,

34:34 we're going to assess impacts on structural capacity and underused

34:37 capacity going back to that no can-do gap framework.

34:40 So let me show you structural capacity first.

34:42 Um,

34:43 but first,

34:43 let me spend a second describing these types of graphs because

34:46 I'm going to be showing you a few of these.

34:48 Um,

34:49 so,

34:49 on the Y-axis are the outcomes of interest,

34:51 above the dotted horizontal line are the summary indices.

34:55 So here we're gonna be looking at overall structural capacity

34:57 and then separately to understand issues of um supply chain,

35:01 we look at the,

35:01 the drugs index.

35:03 Um,

35:03 on the X-axis are the treatment impacts here,

35:06 PBF versus business as usual,

35:07 the dotted,

35:08 um,

35:08 vertical line at zero is zero impacts.

35:11 Increases or improvements in this case instructural capacity,

35:15 uh,

35:15 would lie to the,

35:16 uh,

35:16 right,

35:17 and,

35:17 uh,

35:18 decreases,

35:18 uh,

35:18 obviously to the left.

35:20 As you see here,

35:21 overall,

35:22 there are no,

35:22 for,

35:23 this is just an illustrative example from Cameroon and Nigeria,

35:25 but again,

35:26 you know,

35:26 go to the report,

35:27 look at the Cochrane Review,

35:29 uh,

35:29 for more detail on these sorts of impacts.

35:31 But,

35:31 um,

35:33 Uh,

35:33 in,

35:33 in general,

35:34 we do find,

35:35 at least on some aspects of quality,

35:37 uh,

35:38 some positive impacts,

35:39 sometimes fairly large positive impacts,

35:41 even if overall there isn't.

35:43 But then when we turn to underused capacity,

35:46 um,

35:47 the data for this are more limited.

35:49 Again,

35:49 this is a fairly data-intensive exercise and,

35:51 and we can only estimate these impacts for Cameroon and Nigeria again.

35:55 But

35:55 here,

35:56 remember that underused capacity is undesirable.

35:58 It's,

35:58 it's money left on the table.

36:00 Um,

36:00 so increases are actually problematic,

36:02 and yet,

36:02 that's what we find evidence of,

36:04 uh,

36:05 you know,

36:05 looking at,

36:05 again,

36:06 all of the,

36:07 uh,

36:07 aspects of care,

36:08 uh,

36:08 that the WHO says are necessary.

36:11 And,

36:11 and really no overall um reduction in underused capacity.

36:15 The one other thing I want to note is that,

36:16 yes,

36:17 these are only two countries,

36:18 but these are often viewed as two successful

36:20 instances of the uh early stage PBF design.

36:23 So despite that,

36:24 you know,

36:24 no uh large impacts of

36:27 um

36:28 Uh,

36:28 of performance pay.

36:30 So the bottom line here is,

36:32 you know,

36:32 performance pay in terms of quality is not a

36:34 silver bullet for work or effort in these settings,

36:36 and this is also consistent

36:38 with limited,

36:39 uh,

36:39 evidence of impacts of performance pay and clinical quality,

36:42 even with,

36:43 uh,

36:43 settings with fewer competing constraints.

36:46 And if anything,

36:46 the largest impacts we see

36:48 are for structural quality,

36:49 which may not be surprising given that

36:51 that's where we saw a lot of the

36:53 constraints upfront,

36:54 the quality improvements,

36:56 uh,

36:56 and,

36:56 and only a third of underperformance being tied to underused capacity.

37:00 But this then begs the question,

37:02 you know,

37:02 are there more cost-effective ways of financing structural improvements?

37:05 So,

37:06 let's turn to some other ways in which

37:07 financial incentives have been leveraged for effective coverage.

37:10 Um,

37:11 here,

37:11 we're gonna compare both coverage and effective coverage impacts

37:14 of PBF projects to conditional cash transfers and voucher,

37:16 uh,

37:17 programs on the demand side,

37:18 and direct facility financing on the supply side.

37:20 And,

37:21 and,

37:21 and I just want to note that,

37:22 you know,

37:23 as Damien's also said,

37:24 these are not necessarily

37:25 pure alternatives.

37:26 They often can and should be used in combination and the sequencing

37:30 also becomes really important.

37:32 Here's the geographic scope of the evidence for this part of the report.

37:35 Um,

37:36 a lot of the,

37:37 the PDF and DFF comparisons come from,

37:39 um,

37:41 sub-Saharan Africa,

37:42 but,

37:42 uh,

37:42 but by and large,

37:43 we do have,

37:44 um,

37:44 uh,

37:45 relatively wide global coverage.

37:47 So turning to the

37:49 comparison of PBF and demand side alternatives,

37:52 uh,

37:52 we draw on a meta-analysis

37:54 that pulls,

37:54 uh,

37:55 evidence from 52 programs from 30 countries.

37:58 Um,

37:58 and here on the right,

38:00 uh,

38:00 are the,

38:01 uh,

38:02 impact summarized in this graph.

38:03 So we look at PDF,

38:04 um,

38:05 Program impacts,

38:06 voucher programs,

38:06 and conditional cash transfer programs,

38:08 and the mean effect sizes are on the

38:10 Y axis,

38:11 um,

38:11 for modern family planning and facility delivery,

38:14 at least 4 antenatal care visits,

38:16 and at least 1 postnatal care visit.

38:18 Uh,

38:18 in terms of effective coverage,

38:19 we also look at maternal tetanus vaccination during antenatal care

38:23 and full child vaccination.

38:25 Um

38:26 On average,

38:27 it's true that,

38:29 you know,

38:29 all three types of financial incentives examined improve service coverage,

38:33 um,

38:34 and the,

38:34 uh,

38:35 the mean effect sizes vary between sort of 2% and

38:39 7%.

38:40 These are modest mean effect sizes,

38:42 right?

38:42 These are Context

38:43 where there's a lot of room for improvement.

38:45 And by themselves,

38:46 we're,

38:46 we're talking about fairly um modest improvements,

38:49 uh,

38:49 from any of these.

38:51 If anything,

38:51 if you compare,

38:52 for instance,

38:53 uh,

38:53 the impacts of PBF uh to voucher and CCT programs

38:56 for in facility delivery here in the light blue,

38:59 uh,

38:59 you know,

38:59 there may be some indications that CCTs and vouchers are actually more effective.

39:04 Again,

39:04 going back to sort of the,

39:06 the,

39:06 the constraints that are germane.

39:09 The third and last set of comparisons that we look

39:11 at is that of PBF and its uh supply side,

39:14 uh,

39:14 alternative,

39:15 uh,

39:15 which is uh DFF.

39:16 Um,

39:17 as Damien said,

39:17 the only difference between these two

39:19 is performance pay.

39:21 However,

39:22 the way in which transparency and accountability are operationalized

39:25 in DFF projects is also a bit different.

39:28 Um,

39:28 so,

39:28 in,

39:29 um,

39:30 DFF projects,

39:31 there isn't this reporting and auditing,

39:33 uh,

39:33 setup.

39:33 There's just a business plan or a dashboard typically

39:36 that facilities you.

39:36 So this is a much lighter touch,

39:38 less expensive,

39:39 easier to implement

39:40 without that,

39:41 uh,

39:41 risk of collusion and,

39:42 and gaming,

39:43 uh,

39:43 approach,

39:44 uh,

39:44 for,

39:45 uh,

39:45 transparency.

39:46 In the HRITF portfolio,

39:48 there are 5 projects that directly compare PBF and DFF approaches,

39:51 and we look at all of them.

39:53 Uh,

39:53 these are Cameroon,

39:54 Nigeria,

39:54 Rwanda,

39:54 Zambia,

39:55 and Zimbabwe,

39:56 3 of which Cameroon,

39:57 Nigeria,

39:57 and Zimbabwe also included,

39:59 uh,

39:59 business as usual arm,

40:00 so we can,

40:01 uh,

40:01 look at that comparison.

40:03 And then in 2 of these in Nigeria and Zambia,

40:05 I just want you to keep in mind when I show you the results on the next slide,

40:08 that PBF actually disbursed twice as much as DFF.

40:11 So it's not actually a 1 to 1 comparison.

40:13 That said,

40:14 here's what we find.

40:15 So,

40:15 on the Y-axis again are all of the um impacts of interest,

40:19 and there are a lot of impacts here,

40:20 just to show you that we looked

40:22 at a lot of different outcomes,

40:24 right?

40:24 We're not cherry-picking here.

40:25 Um,

40:26 but,

40:27 and we look at outcomes related to

40:28 effective coverage and then also simple coverage.

40:31 The only place where we,

40:32 uh,

40:33 find an impact of PBF

40:35 over DFF is for institutional deliveries.

40:38 And even some of that,

40:39 so these are,

40:39 you know,

40:39 the,

40:40 the significant impacts are the solid um markers.

40:44 Even there,

40:45 uh,

40:45 some of the gain,

40:46 although not all,

40:47 not necessarily in,

40:48 in some of the settings,

40:49 but

40:50 at least in a few of the countries,

40:51 some of the gain is,

40:52 uh,

40:53 partially explained by a switch from the private sector to the public sector.

40:57 It's being driven by an increase in institutional deliveries in the public sector.

41:02 Where we don't know the quality of the private sector,

41:04 and so this is sort of unclear in terms of impacts on effective coverage.

41:08 So just as I wrap up,

41:10 you know,

41:10 a couple of other considerations and approaches and,

41:13 and sort of strengthening program impacts and designing programs

41:16 that might be more effective in terms of,

41:17 uh,

41:18 quality gains.

41:19 Um,

41:20 one thing that we,

41:21 uh,

41:21 highlight both from the meta-analysis and the decomposition of effects

41:24 in Nigeria is the role of baseline coverage in,

41:26 in the catchment area.

41:27 So,

41:27 if

41:28 baseline coverage was really low,

41:30 then that might be indicative of,

41:32 uh,

41:32 demand side constraints,

41:33 uh,

41:33 for whatever reason,

41:34 it should be addressed through demand side approaches.

41:36 If it's very high,

41:37 it probably means there's room for improvement.

41:40 Uh,

41:40 there's little room for improvement,

41:42 and,

41:42 you know,

41:42 performance pay,

41:44 uh,

41:44 might not adequately,

41:45 um,

41:46 reimburse the,

41:47 the worker for the effort that's required.

41:50 We also know from the high-income country literature,

41:52 but also from uh an increasing number of

41:54 examples from the HRITF portfolio that performance pay,

41:57 if anything,

41:57 may work best at the hospital level,

42:00 so not at the primary level.

42:02 Um,

42:02 again,

42:02 a question of competing constraints.

42:05 Uh,

42:05 we know,

42:06 for example,

42:06 from one really sterling example in Afghanistan,

42:09 that private sector contracting

42:11 can be fruitful grounds for PBF and that's not what we talk about here,

42:14 we really focus on the public sector

42:15 in the report,

42:16 but this is an important thing to note.

42:18 And then finally,

42:19 you know,

42:19 we don't talk about cost-effectiveness.

42:21 Um,

42:22 uh,

42:22 there's limited evidence,

42:23 and this is sort of a little bit outside our wheelhouse,

42:25 but we do have,

42:27 uh,

42:27 aspects that we look at that are sort of

42:29 That have important implications for cost-effectiveness.

42:32 Uh,

42:32 the first is that there are complementarities and tasks that might generate

42:36 positive spillovers.

42:38 The theory for this actually goes back to Holmstrom and Milgram,

42:40 last year's,

42:41 uh,

42:41 Economics Nobel laureates,

42:42 but we don't see that theory being leveraged

42:44 in,

42:45 uh,

42:45 in the design of these performance pay contracts,

42:48 and they perhaps should be.

42:50 Um,

42:50 we also know from a,

42:51 a large body of literature,

42:53 including going to conditional cash transfers

42:54 and even payment for ecosystem services,

42:57 but also from this HRITF portfolio,

42:59 that prices sometimes,

43:00 uh,

43:01 serve to signal the importance of the task you're purchasing.

43:03 And so the level of the price might be less important than that signal.

43:07 Um,

43:07 also from the HRRTF portfolio,

43:09 we know the importance of using non-pecuniary tools

43:12 to amplify the impact of the pecuniary ones,

43:14 such as through the use of hand,

43:16 uh,

43:16 hands-on supervision,

43:17 uh,

43:18 for example,

43:18 through the anatomical mama,

43:19 momma Natalie and neo-Natalie models.

43:22 I'm gonna leave you with this bottom line for financial incentives,

43:24 and that is that financial incentives,

43:26 you know,

43:26 it is clear,

43:27 improve service utilization and at least some measures of quality,

43:30 but those impacts are small,

43:31 and they leave lots of room for improvement.

43:34 Um,

43:34 in under-resourced central,

43:36 uh,

43:36 centralized health systems,

43:37 performance pay,

43:38 we find

43:39 yields only modest additional impacts.

43:41 Um,

43:42 but flexible operating budgets and

43:44 associated accountability measures can deliver

43:47 significant gains by themselves without performance pay,

43:50 while avoiding

43:51 the,

43:51 the large additional costs and complexity of design

43:54 of performance pay interventions.

43:56 And with that,

43:56 Damien's gonna wrap up.

44:03 Thank you very much,

44:03 Ian and let me uh wrap up and try to focus on what

44:07 could be the operational implication of these findings.

44:11 So the key takeaways

44:12 we think are the following.

44:14 Overall little evidence for impacts for across the board performance pay,

44:19 especially in those under-resourced and unfinanced

44:23 health systems,

44:24 and we think that direct facility financing

44:27 with autonomy and accountability can deliver many gains

44:30 at lower cost

44:32 and with

44:33 easier implementation.

44:35 We note,

44:35 for example,

44:36 that performance pay make budgets unpredictable.

44:39 Both from the point of view of the government

44:42 as seen from the

44:43 point of view of the health facility.

44:46 And the cost,

44:47 the costs related to verification in performance pay,

44:50 which are necessary to enforce conditionality can often

44:54 be

44:54 quite substantial.

44:55 For example,

44:56 in Nigeria,

44:58 2/3 of the project administrative costs

45:01 were attributed to the effort for verification.

45:05 So it's true that

45:07 uh risk-based

45:09 system of verification could substantially reduce these costs,

45:12 but still

45:13 they remain important.

45:15 In general,

45:16 we find that performance pay is a complex intervention which,

45:19 especially in those settings,

45:21 can be

45:22 difficult to scale.

45:24 So now I'd like to finish with

45:26 a few action items,

45:27 and they

45:28 don't necessarily come from the evidence.

45:30 They are more like supported by the evidence.

45:32 They are informed by what has been happening on the ground over the last

45:35 15 years and that

45:37 we've observed that we were working with our colleagues in the health sector

45:41 in the design of the,

45:42 of these health financing reform.

45:44 So our suggestion would be

45:46 to start with,

45:47 if you think about starting a new

45:49 health financing reform,

45:51 to start by.

45:52 A set of assessments.

45:54 First,

45:55 assess using existing data

45:57 coverage versus effective coverage

46:00 to identify those low hanging fruit

46:02 for performance pay.

46:04 Also try to assess the gaps

46:06 in structural capacity

46:08 and in health worker knowledge.

46:11 And try to measure what we've called the no can do gap,

46:15 the difference between

46:16 what is available at the health facility in terms

46:18 of knowledge and equipment and what is actually done.

46:21 And we think the sweet spot for performance pay

46:26 is

46:27 where baseline utilization

46:29 has room for improvement.

46:30 It's not too high,

46:31 but it's not so low

46:33 as to be indicative of

46:35 demand side barriers.

46:37 Um,

46:37 which might best be

46:39 addressed by the demand side intervention

46:41 and also the sweet spot is also probably where there is

46:44 a high level of no can do gaps.

46:46 So

46:46 basically they are

46:48 underused capacity at the health facility.

46:51 We also think that sequencing is key

46:54 when you think about starting this new health financing reform.

46:59 We suggest 18 to 24 months at the beginning

47:02 after these assessments for demand side interventions

47:06 and

47:07 direct facility financing,

47:08 and then repeat the diagnostics exercise

47:11 and then introduce performance pay,

47:14 performance pay for select indicators.

47:16 We also think that emerging technologies

47:19 can reduce the implementation costs.

47:21 We are thinking about mobile money that can

47:24 that can send the funds directly to

47:26 the health facility and the health workers,

47:28 and in terms of accountability,

47:31 uh,

47:32 there are new apps,

47:33 etc.

47:33 that can develop portals and dashboards

47:36 to look at both

47:37 quantity,

47:40 quantity quality as well as payments.

47:43 Let me finish

47:45 with

47:46 5 key principles

47:48 for sustainable health financing reform.

47:50 Again,

47:51 those are not

47:52 necessarily new,

47:53 but I think

47:54 looking at the evidence

47:56 from our report,

47:57 it's important to highlight them again.

48:00 The first one

48:01 would be to send

48:02 funds to the front line to those health facilities serving people in the village

48:08 and neighborhoods.

48:09 Second one,

48:11 ensure transparency and accountability through

48:13 good facility financial management.

48:16 Third one,

48:18 when necessary,

48:19 introduce

48:20 routine supervision that should be supportive,

48:23 not punitive.

48:24 It's important also to introduce an output or performance orientation.

48:29 But we note that this can be

48:32 done through dashboards or portals and not necessarily

48:36 or not

48:37 always with

48:38 just performance pay.

48:40 And finally,

48:40 it's important to unify the payment system with one

48:45 system to push the payment

48:48 to the front lines.

48:49 With this,

48:50 we're looking forward to the discussion and thank you very much for your

48:55 attention.

49:02 Thank you,

49:03 Damien Ushani.

49:03 So now we'll talk,

49:05 turn straight to our discussions.

49:06 Uh,

49:06 first up will be Norbert,

49:08 uh,

49:08 here,

49:09 and then we have Feng who is online.

49:11 Uh,

49:12 we'll go in that order if that's OK.

49:13 Thank you.

49:15 Thanks,

49:15 Dion,

49:16 and uh thanks

49:18 uh Damien and Ianni for asking me to,

49:20 to comment on this.

49:21 Um,

49:21 it's great to be back uh here.

49:24 Um,

49:24 I've also read the report at great length as uh

49:27 more than once actually,

49:28 uh,

49:29 since I was one of the peer reviewers,

49:30 and some of my comments

49:32 are comments that I made at the time,

49:34 um,

49:34 and I just,

49:35 I want to reiterate some of them.

49:37 So this is clearly a very serious research product,

49:40 um,

49:41 in the best tradition of uh of PRRs I think.

49:44 Uh,

49:44 it brings together as much of the evidence

49:46 as is available on the effectiveness of a particular

49:50 kind of intervention,

49:51 and then tries to make sense of this evidence sort of more broadly,

49:54 not just the individual,

49:55 um,

49:55 the individual case studies.

49:57 It's obviously highly,

49:58 uh,

49:59 policy relevant.

50:00 Um,

50:00 so that that's,

50:01 that's really important.

50:03 And it's particularly policy relevant,

50:04 I think,

50:05 as we make the transition from sort of the focus

50:08 on the pandemic to sort of the more underlying,

50:11 if you want,

50:11 routine,

50:12 uh,

50:13 problems that,

50:13 uh,

50:14 health systems face in,

50:15 in many of the countries that we work on.

50:16 I,

50:17 I sat in during the spring meetings that many of the,

50:19 uh,

50:20 conversations we had with the ministers of finance and ministers of

50:24 health and education and so on.

50:25 Certainly in the conversations on health,

50:27 uh,

50:27 in,

50:27 in,

50:27 in sub-Saharan African countries,

50:29 there was often a feeling that

50:31 Um,

50:31 we as the bank were

50:33 perhaps too concerned with things related to the

50:35 pandemic and not concerned enough with some of the

50:38 underlying health issues,

50:40 um,

50:40 that,

50:41 uh,

50:41 that these countries face.

50:42 So I think this is really an important report in that regard as well.

50:47 I think there's,

50:48 there's so much that is useful in this report,

50:50 so,

50:50 um,

50:50 I'm gonna just highlight a few things,

50:53 um.

50:54 Uh,

50:55 and this list is by no means meant to be the full list by any means.

50:59 Uh,

51:00 I think the descriptive results on,

51:02 on effective coverage are,

51:03 are extremely useful,

51:04 the ones that are there for 150 some countries.

51:07 I think of those

51:08 kind of like what the WDR for education.

51:10 With lays

51:12 showing that

51:13 it's not just coverage that is important,

51:14 but effective coverage.

51:16 It doesn't make much sense to have coverage if people don't receive care that is

51:19 useful in some ways,

51:20 just as with education it doesn't make much sense to have

51:22 kids brought to school if they don't learn at least something.

51:25 So I think that is really,

51:26 really useful.

51:27 I like these effective coverage contours of the authors present,

51:31 which sort of make really clear the point that

51:33 you can increase effective coverage either by increasing coverage,

51:37 or by increasing quality or some combination of the two.

51:41 I will say that these contours are sort of,

51:43 uh,

51:44 are,

51:45 I mean,

51:45 as the authors well know,

51:46 this is not a point that they don't know,

51:48 but they're,

51:48 but they're silent on the fact that two different points

51:52 on the same contour could have very different distributional implications.

51:55 So you could basically

51:57 You know,

51:57 have coverage for a very small fraction of the population

52:00 with very high quality,

52:02 and you would be on the same point as having lower quality,

52:05 but broader coverage.

52:07 So I think it's just something

52:08 to keep in mind when we sort of make sense of these effective coverage contours.

52:13 Um.

52:15 The results to my mind are,

52:16 are,

52:16 are quite sobering.

52:18 Um,

52:18 you know,

52:19 the authors argue that PBF is no,

52:22 no silver bullet.

52:23 I know they want to get away from this glass half empty,

52:26 glass half full analogy,

52:28 and I understand.

52:30 Um,

52:30 but on the other hand,

52:31 for me,

52:31 and I don't think it's only because I'm

52:34 not the most optimistic person in the room,

52:36 but I think it's hard if you want for me not to come away from this,

52:40 saying that it looks to me more like the glass half empty.

52:43 Uh,

52:43 than the glass half full.

52:45 Um,

52:46 I think we should take these results very seriously.

52:48 I mean,

52:48 we're all searching for solutions to really difficult,

52:51 uh,

52:51 development problems,

52:53 um,

52:53 and we really want,

52:55 uh,

52:55 things to work,

52:56 and sometimes things that we think have to work,

52:59 and that really they have to work.

53:00 Uh,

53:01 well,

53:02 maybe they don't work,

53:03 and,

53:03 and maybe,

53:04 maybe we get carried away,

53:05 uh,

53:06 by our enthusiasm,

53:07 uh,

53:07 somewhat.

53:08 I also like the fact that in the report,

53:10 I,

53:10 I very much like the no can do framework,

53:13 uh,

53:13 jokingly,

53:13 I when I first started telling somebody else about this,

53:16 they,

53:16 they,

53:16 they thought I was saying no can do,

53:18 like no can do.

53:19 And maybe that is,

53:20 uh,

53:20 maybe that is one of the messages,

53:22 no can't do this stuff.

53:23 I mean,

53:23 you know,

53:23 really,

53:24 it's too complicated,

53:24 no can do,

53:25 but no,

53:26 in a,

53:26 on a more serious level,

53:28 um,

53:28 I,

53:28 I think it's really useful to try to identify what,

53:31 where the constraints are to having more effective,

53:34 more.

53:34 Effective uh

53:36 coverage.

53:37 And I think what the authors point out that many

53:40 of the problems and potential solutions are things that don't,

53:43 don't depend really on

53:45 health worker effort that they're not really in the

53:47 locus of control of that particular health worker.

53:49 They don't have the right

53:51 materials or infrastructure in the health center or the or the training and

53:56 education that they received wasn't adequate.

53:57 And so these are things that are not going to be solved by

54:00 by trying to

54:01 more effort and and the authors,

54:03 they don't cover it here,

54:04 but they go at in some length

54:06 in the report about,

54:07 I mean,

54:07 this to me,

54:08 the staggering levels of coverage uh or or treatment

54:13 that is ineffective and potentially harmful.

54:15 I mean,

54:16 these are huge numbers.

54:17 I had no idea,

54:18 and surely that shows just my ignorance,

54:20 but I,

54:21 but if that is my ignorance,

54:22 it probably is an ignorance shared by many others,

54:24 but the levels are really,

54:25 really big,

54:26 and the fact that in some cases,

54:28 The performance-based financing makes this worse

54:31 is definitely a source of concerns.

54:35 Um,

54:36 can you guys put up slide 41 for a second?

54:38 Because,

54:39 uh,

54:39 now comes the,

54:40 the,

54:40 the,

54:40 the part of where I,

54:41 where I sort of like wonder a little bit about some of the

54:44 conclusions that,

54:45 that we draw,

54:46 only if it's possible,

54:47 if it's,

54:48 so that's slide 41.

54:49 So I think this is a really important slide,

54:51 because really at the end,

54:52 what the authors conclude is that uh

54:55 Uh,

54:56 uh,

54:57 direct frontline financing,

54:58 DFF,

54:59 um,

54:59 can have the same

55:01 effects

55:02 as PBF,

55:03 and in some sense that maybe this is what we should be doing instead of PBF.

55:08 But really,

55:08 what I see there is that DFF had no impacts either relative to business as usual.

55:14 So here again,

55:15 you look at the coefficients,

55:16 if you want to look at,

55:18 you know,

55:18 which ones are the the DFF versus none.

55:21 So those are The Little squares,

55:24 blue squares,

55:25 and I don't,

55:25 I see as many,

55:26 just about negative as positive,

55:28 and,

55:28 and no significant results.

55:30 So it's hard for me to conclude from this.

55:32 I mean,

55:32 I understand here we are in the glass half empty,

55:35 glass half full,

55:36 and we're saying,

55:36 well,

55:37 you know,

55:38 PBF is a problem.

55:39 It's too complicated,

55:40 you know,

55:41 it,

55:41 it,

55:42 it may give us the wrong incentives,

55:44 but I don't see much evidence here

55:46 that at least in these,

55:47 uh,

55:48 in,

55:48 in these countries.

55:49 DFF,

55:50 so this direct transfer of resources to the frontline workers with accountability

55:55 and,

55:56 and so on,

55:57 that this actually improved outcomes.

55:59 Um,

55:59 it seems to me that it's,

56:01 it's a bit speculative to say that it improved outcomes,

56:04 unless we're somehow

56:06 thinking that,

56:06 you know,

56:07 as this thing gets

56:09 institutionalized,

56:10 if you want,

56:11 sort of over time,

56:12 you know,

56:13 people get used to it and this improves outlook,

56:14 but here in this figure.

56:17 I don't really see any evidence of improved outcomes either

56:20 from PBF or

56:22 from DFF,

56:24 and I think that's,

56:24 I think we want to be honest with ourselves about that as well.

56:28 Um,

56:30 Couple of more points.

56:31 Uh,

56:32 the authors point out that,

56:33 uh,

56:34 the early results from PBF from,

56:36 uh,

56:36 uh,

56:36 Rwanda and Burundi were encouraging,

56:39 and that's really interesting to me.

56:40 So why were those encouraging and,

56:41 and the rest,

56:42 and the rest not?

56:43 Um,

56:44 it,

56:44 it,

56:45 it,

56:45 uh,

56:45 I mean,

56:46 I don't know the answer,

56:47 and,

56:47 and,

56:47 uh,

56:48 maybe somebody else knows the answer,

56:49 but it seems like again,

56:51 because we saw those as encouraging,

56:52 we sort of went in.

56:53 You know,

56:54 fully into this approach,

56:57 and were then disappointed.

56:58 So some reflection about why those were encouraging and the rest were not,

57:02 I think,

57:03 would be really useful.

57:05 These are based,

57:06 I'm

57:07 reflecting now on the on the

57:08 presentation more than on the report,

57:10 but I think that would be useful.

57:12 Uh,

57:13 a few,

57:13 uh,

57:14 sort of qualifications.

57:15 Um,

57:16 I think,

57:16 uh,

57:17 the the authors well acknowledged that,

57:19 uh,

57:19 the focus is mainly on maternal and child health,

57:21 and it's just something to keep in mind

57:23 that we're talking of maternal and child health,

57:25 not lots of other things,

57:26 uh,

57:26 in,

57:27 in health.

57:28 Um,

57:28 we're also not talking directly of health outcomes.

57:30 Um,

57:31 we're talking about the use and the quality of these services.

57:34 I think all of us have some doubts about

57:36 how directly the outcomes are linked to the use,

57:38 and so it's just,

57:39 again,

57:40 it's something to keep in mind.

57:42 I don't find the comparison with CCTs.

57:44 I've said this to the authors before,

57:45 and it's fine,

57:46 but I don't find the comparison with CCTs all that

57:48 persuasive because it really is a very different geographic sample.

57:52 Uh,

57:52 when we're looking at CCTs,

57:54 the evidence is largely from middle income

57:57 countries,

57:57 especially in Latin America,

57:59 and the evidence from these PBF and DFF pilots is,

58:02 is mainly,

58:03 uh,

58:04 from,

58:04 from sub-Saharan Africa,

58:05 mainly,

58:05 I,

58:05 I should say,

58:06 at least the results that are stressed in the presentation.

58:08 So I don't find that comparison as

58:10 As

58:11 as persuasive,

58:12 but overall,

58:14 I think this is really an important report.

58:16 Um,

58:17 I think we all need to reflect in our different role in the bank.

58:21 We need to work in our conversations with policymakers.

58:24 I think we all need to reflect.

58:25 Uh,

58:25 on,

58:26 on what messages we want to take away from this,

58:29 um,

58:30 I found,

58:31 I learned a ton,

58:32 uh,

58:32 reading it,

58:32 and it leaves me with a lot of questions about where we should go next,

58:36 uh,

58:36 in,

58:36 in,

58:37 in these kinds of interventions.

58:38 But,

58:38 uh,

58:38 thank you very much again,

58:40 to,

58:40 to everybody for asking me to be a reviewer and then for providing comments.

58:44 Thanks.

58:46 Thanks Norbert for those reflections.

58:48 Um,

58:49 we will have a chance to go back to the team to maybe pick up on some of them,

58:53 um,

58:54 uh,

58:54 but right now I'd like to turn to Defeng who's online,

58:57 um,

58:58 for,

58:58 uh,

58:58 his,

58:59 uh,

58:59 reactions and comments.

59:00 Thank you.

59:03 Thanks,

59:04 Tia.

59:04 Can you hear me?

59:11 Hello Dean,

59:12 can you hear me and see me?

59:14 Uh,

59:14 we can hear you very faintly.

59:15 I don't know if it's on our side or your side.

59:20 Hello,

59:21 is it better?

59:22 OK,

59:22 we can hear you in the room,

59:24 fine,

59:24 so.

59:26 OK,

59:26 great.

59:27 OK,

59:27 perfect.

59:28 Thank you.

59:28 Let me start.

59:30 Very happy to be here.

59:32 Um,

59:32 I first want to echo,

59:34 uh,

59:35 Norbert's point.

59:36 This is,

59:36 uh,

59:38 to me,

59:38 a high impact

59:40 report.

59:41 It has a lot of policy implications,

59:44 um,

59:46 not only to this particular

59:48 area,

59:48 but also to

59:49 our health,

59:51 um,

59:51 program,

59:52 uh,

59:52 as a whole.

59:53 So I,

59:54 I really want to,

59:55 um,

59:56 commend the team

59:58 for getting this done and get the report in place.

1:00:01 I would have hoped and

1:00:04 this report could

1:00:05 have come

1:00:06 much,

1:00:07 much earlier,

1:00:08 but

1:00:09 Nevertheless,

1:00:10 it has brought a lot of policy thinking and policy reflections.

1:00:14 So

1:00:14 let me,

1:00:14 let me say that upfront.

1:00:17 And this is also a topic very close to my heart.

1:00:20 Um,

1:00:21 I'm actually on leave,

1:00:22 I'm on my way to Michigan.

1:00:25 to accompany my kids for his sports.

1:00:27 Um,

1:00:28 but when Moor asked me,

1:00:29 I immediately said yes,

1:00:31 um,

1:00:32 because,

1:00:33 um,

1:00:34 When the RPF took off,

1:00:36 um,

1:00:37 I was a frontline TTL.

1:00:39 I was based in Ethiopia.

1:00:40 I can replay

1:00:42 some of the conversation.

1:00:43 I think it's relevant here.

1:00:44 Let,

1:00:44 let me just maybe

1:00:46 put this,

1:00:46 uh,

1:00:46 as my first point.

1:00:47 But before that,

1:00:48 I want to say I'm,

1:00:50 I'm very happy to say many colleagues have joined

1:00:53 and they are

1:00:54 more qualified than me,

1:00:55 uh,

1:00:56 in these areas.

1:00:57 I,

1:00:57 I'm also eager to,

1:01:00 to hear their comments uh during the Q&A time.

1:01:03 Um,

1:01:04 The first question,

1:01:05 the first reflection I want to

1:01:07 put on the table is really,

1:01:09 as I said,

1:01:09 from

1:01:10 a frontline TTO point of view.

1:01:13 When I was based in Ethiopia when this program sort of took off,

1:01:18 um,

1:01:18 we were very excited,

1:01:20 um,

1:01:21 because we all tired about

1:01:23 the stagnation of the poor health outcome

1:01:26 and we have been working on input-based financing for a while.

1:01:30 We need something new,

1:01:32 um,

1:01:33 something exciting.

1:01:35 Uh,

1:01:35 when this came out and together with a

1:01:37 lot of analytical underpinning and case studies,

1:01:39 it's not,

1:01:40 you know,

1:01:41 there's no,

1:01:42 no,

1:01:42 um,

1:01:43 Uh,

1:01:44 no science behind it.

1:01:46 Um,

1:01:47 so we were very excited,

1:01:48 brought a lot of energy in our policy dialogue,

1:01:51 um,

1:01:51 and together with a lot of workshops and taking place in Africa

1:01:56 and,

1:01:56 and our financing,

1:01:58 there's a trust fund,

1:01:59 large size of trust fund also behind it.

1:02:01 Um,

1:02:02 but once we take on the,

1:02:04 um,

1:02:04 the,

1:02:05 the policy dialogue with,

1:02:06 with Minister of Health,

1:02:08 Minister of Finance,

1:02:10 and we run into a few

1:02:12 You know,

1:02:14 Realistic issues.

1:02:16 I remember the Minister of Health

1:02:19 at that time was Doctor Tetras,

1:02:20 now head of WHO basically told us

1:02:23 Uh,

1:02:24 Ethiopia graduate 200 doctors per year.

1:02:27 This way,

1:02:28 we have,

1:02:29 um,

1:02:30 by

1:02:31 directing your financing towards

1:02:33 some health coverage targets will not resolve,

1:02:37 uh,

1:02:37 the fact that we don't have enough doctors,

1:02:39 right?

1:02:39 So that's,

1:02:40 that's the constraint,

1:02:42 um,

1:02:42 they are facing.

1:02:43 So just,

1:02:43 just give that one example,

1:02:46 um,

1:02:47 that may be generated to 1.15.

1:02:49 Um,

1:02:50 we know,

1:02:51 um,

1:02:52 you know,

1:02:52 money cannot

1:02:54 resolve all the health issues.

1:02:57 Um,

1:02:57 the,

1:02:58 the solution often lie in a package of,

1:03:01 um,

1:03:02 factors,

1:03:03 sometimes beyond

1:03:04 health sector.

1:03:06 But because we are working on financing,

1:03:08 financing institution,

1:03:10 inevitably we,

1:03:12 we looking for solution toward that direction.

1:03:15 So that's,

1:03:16 that's maybe

1:03:17 the value of this report really got us thinking.

1:03:20 I,

1:03:20 I really like some of the conclusion of the report

1:03:23 and some of the,

1:03:24 uh,

1:03:24 the message make me pause quite a bit to,

1:03:27 to reflect,

1:03:28 but nevertheless,

1:03:29 I think this report has really

1:03:32 Hit the heart

1:03:33 of our,

1:03:33 our core sort of health um

1:03:36 program and,

1:03:37 and make us think about

1:03:39 uh what's,

1:03:39 what's going to work,

1:03:40 uh,

1:03:41 how we can best help our clients.

1:03:43 So that's one message I want to share.

1:03:46 The second one is really

1:03:48 um a forward looking

1:03:52 um

1:03:52 angle.

1:03:54 Um,

1:03:57 So I,

1:03:58 you know,

1:03:59 now the RBF um were

1:04:03 Performance-based financing or some other terminology,

1:04:07 um,

1:04:07 the incentive we provide has evolved into a different phase.

1:04:11 So we have

1:04:12 an instrument called P4R.

1:04:15 Um,

1:04:15 many of the,

1:04:17 this type of initiative has uh integrated into

1:04:21 The overall health financing program.

1:04:23 For example,

1:04:23 many insurance agencies have

1:04:26 quality based measure,

1:04:28 um,

1:04:29 you know,

1:04:29 for them to contracting out services

1:04:31 and a lot of those performance financing element also embedded into,

1:04:36 into that regard.

1:04:38 So we are actually in a new phase.

1:04:41 So that's why I want to

1:04:43 encourage the team to think more

1:04:45 in terms of the next step.

1:04:46 I,

1:04:47 I share Norbert's view.

1:04:50 Um,

1:04:51 Let's go beyond the comparison between

1:04:55 RPF,

1:04:56 DDF,

1:04:56 or CCT.

1:04:57 There's many other options out there.

1:05:00 Um,

1:05:00 I like some of the comments thing.

1:05:02 I will put a false dichotomy

1:05:04 in there.

1:05:05 Um,

1:05:06 to me,

1:05:07 um,

1:05:07 think about the instrument of P4R will be much more meaningful

1:05:12 for our,

1:05:13 uh,

1:05:13 internal audience than

1:05:15 compared with those different options.

1:05:17 Sometimes the,

1:05:18 the solution is really the package of them.

1:05:20 It's not really one or another.

1:05:22 Um,

1:05:23 I

1:05:24 demand

1:05:25 demand,

1:05:25 I,

1:05:25 I,

1:05:26 the,

1:05:27 the slide the last.

1:05:29 Last slides were the 2nd to the last slide on the sequencing.

1:05:33 I,

1:05:33 I,

1:05:33 I question that.

1:05:35 Uh,

1:05:35 that also leading the philosophy of,

1:05:38 you know,

1:05:38 have

1:05:39 low hanging fruit,

1:05:41 have analytical work,

1:05:42 but eventually land on performance pay.

1:05:45 So maybe that's not the sequence we want to put forward.

1:05:48 Um,

1:05:49 I know maybe you,

1:05:50 you have different thinking,

1:05:51 but I just

1:05:52 question the sequence

1:05:54 still uh working towards

1:05:56 um performance pay.

1:05:58 This is something maybe we,

1:05:59 we can,

1:05:59 we can think about it on the way forward.

1:06:02 Recently,

1:06:02 we have um

1:06:04 a conversation with

1:06:08 There's a lot of comments from colleagues from the HMP

1:06:12 TTLs.

1:06:13 One comment is the capacity building,

1:06:15 um,

1:06:16 component.

1:06:18 Which is not the P4R,

1:06:20 but it's really uh a TA component to get come together with is so critical,

1:06:24 will help to address some of those uh bottlenecks and constraints

1:06:29 uh in the system.

1:06:30 The next point I want to put

1:06:32 on the table is really about our decision making.

1:06:36 And this morning in our HD forum,

1:06:39 I think a few colleagues,

1:06:40 Xiang Nan,

1:06:41 and China,

1:06:42 uh,

1:06:42 and

1:06:43 Norbert,

1:06:44 uh,

1:06:44 you know,

1:06:44 have put a session really on lesson learned

1:06:47 for COVID.

1:06:48 Uh,

1:06:48 there's a lot of quick decision,

1:06:50 um,

1:06:51 have been made,

1:06:52 um,

1:06:53 um,

1:06:54 for COVID response,

1:06:56 uh,

1:06:56 without really

1:06:57 solid,

1:06:58 um,

1:06:58 evaluation or data or,

1:07:00 or scientific evidence,

1:07:02 but we have to make it up,

1:07:03 right?

1:07:04 So to,

1:07:04 to move forward.

1:07:06 Um,

1:07:07 that

1:07:07 sort of

1:07:09 also related to this conversation,

1:07:11 right?

1:07:11 So the decision was made at the time.

1:07:14 Uh,

1:07:14 move forward,

1:07:16 um,

1:07:16 pushing for performance pay,

1:07:19 um,

1:07:20 but

1:07:21 What's the science and,

1:07:22 and,

1:07:23 and evidence behind it at the time and what's the evaluation

1:07:27 which can help us

1:07:28 uh make the adjustment.

1:07:30 Um,

1:07:31 I want to challenge all of us.

1:07:33 Um,

1:07:34 I,

1:07:35 I hope that this study

1:07:37 You know,

1:07:37 if we've done many years earlier,

1:07:39 we will really help,

1:07:41 help everybody.

1:07:42 This is nobody's fault.

1:07:43 Um,

1:07:44 but I want to challenge two things.

1:07:46 One is

1:07:47 Um,

1:07:48 have we learned from this?

1:07:50 Uh,

1:07:51 do we have,

1:07:52 um,

1:07:52 early investment on data,

1:07:54 on science,

1:07:55 on research,

1:07:56 which will help us,

1:07:58 um,

1:07:58 to make some decisions

1:08:00 in real time.

1:08:02 Uh,

1:08:02 this first one.

1:08:03 Second,

1:08:04 have we embedded evaluation,

1:08:07 um,

1:08:08 quickly into our ongoing,

1:08:10 uh,

1:08:10 operations?

1:08:11 In COVID,

1:08:12 I appreciate IEG quickly put

1:08:15 Put together some evaluation for our response,

1:08:18 which

1:08:18 have provide a lot of

1:08:20 uh insights.

1:08:21 But

1:08:22 um

1:08:22 I hope we have

1:08:23 done in many areas.

1:08:26 Uh,

1:08:27 lucky in this area.

1:08:28 Um,

1:08:29 the next point I want to make,

1:08:31 um,

1:08:32 is we shouldn't oversell.

1:08:35 The service

1:08:36 coverage,

1:08:37 um,

1:08:38 contribution to better health outcomes.

1:08:41 We know house is complicated.

1:08:42 It's multiple

1:08:44 production function.

1:08:45 Even you'll have

1:08:47 perfect

1:08:48 effective coverage,

1:08:49 you have perfect quality,

1:08:51 you probably still cannot get

1:08:53 um

1:08:53 the desired

1:08:54 house outcome.

1:08:55 Let's,

1:08:56 let's be careful because the,

1:08:57 the social determinants,

1:08:59 the,

1:08:59 the other determinants of health outcome.

1:09:02 So let's be mindful.

1:09:03 I see some of the languages they make me worried.

1:09:06 Um,

1:09:07 so that's,

1:09:08 um,

1:09:09 another point I want to make.

1:09:11 Um,

1:09:11 looking forward,

1:09:12 I really want to encourage we all

1:09:15 think about those,

1:09:16 uh,

1:09:17 issues.

1:09:17 Um,

1:09:18 I do feel

1:09:19 some of the strong elements,

1:09:21 um,

1:09:22 the principle

1:09:23 demand has laid out.

1:09:25 I,

1:09:25 I really like them.

1:09:27 So

1:09:27 financing to front line,

1:09:29 um,

1:09:31 Those,

1:09:31 you know,

1:09:32 maybe by itself,

1:09:34 um,

1:09:34 if it's not necessarily lead to performance pay,

1:09:38 it will make a big,

1:09:39 big difference.

1:09:40 Maybe the solution will stop there.

1:09:42 Um,

1:09:42 so let's,

1:09:43 that's why I question the sequence.

1:09:45 Um,

1:09:46 just,

1:09:47 um,

1:09:48 I guess my point is,

1:09:50 is,

1:09:50 let's be mindful about the package of services,

1:09:54 solution we put forward.

1:09:55 Um,

1:09:57 many colleagues have mentioned the silver bullet argument,

1:10:00 um,

1:10:01 but in reality,

1:10:02 I see inevitably sometimes we bring a solution.

1:10:06 Before,

1:10:07 uh,

1:10:07 we see the reality.

1:10:09 So that's something I take home.

1:10:11 I feel,

1:10:11 um,

1:10:12 it's the really the value of the reports

1:10:14 to reflect on those important issues.

1:10:17 Then,

1:10:17 let me stop here.

1:10:18 I have more to say,

1:10:19 but let me stop here.

1:10:20 I want to hear from,

1:10:21 uh,

1:10:22 the audiences also.

1:10:27 Thanks,

1:10:27 Fang for those comments,

1:10:29 both very specifically to this analysis and this report,

1:10:32 but also some of the big,

1:10:33 bigger challenges I think you,

1:10:35 you're posing out for,

1:10:36 for,

1:10:37 for the,

1:10:37 the HNP sector as a whole and probably even beyond.

1:10:42 Um,

1:10:43 so,

1:10:44 We have a couple of,

1:10:45 we have about a little bit more than 10 minutes left.

1:10:48 I propose we,

1:10:49 there are a couple of people raised some questions in

1:10:51 the chat that I don't think were necessarily answered.

1:10:53 Um,

1:10:54 so I might call on just a couple of people who put in questions.

1:10:58 Um,

1:10:58 can I ask you to please keep your questions to sort of under,

1:11:01 under,

1:11:02 you know,

1:11:02 under a minute,

1:11:03 and then we'll give the chance to the team to,

1:11:05 to kind of respond both to the discussions and to these specific questions,

1:11:09 and then if we have some time,

1:11:10 we can do another round.

1:11:12 Um.

1:11:12 So with that,

1:11:13 I'd like to call in order just to give you a bit of a heads up

1:11:16 on Adana,

1:11:18 Emre and Guri Frits.

1:11:20 Um,

1:11:21 can I call on you each in order to please unmute and ask a question,

1:11:26 uh,

1:11:26 again,

1:11:27 keeping it as brief as possible,

1:11:28 and then,

1:11:28 um,

1:11:29 then we'll,

1:11:30 uh,

1:11:30 turn it back to the team.

1:11:32 So with that,

1:11:32 uh,

1:11:33 Ana,

1:11:34 are you able to unmute and ask a question?

1:11:39 Oh,

1:11:39 OK,

1:11:40 Adena has left,

1:11:41 so can we go to Emery?

1:11:46 Is Emery still there?

1:11:49 I am.

1:11:50 Just one moment,

1:11:50 please.

1:11:53 Sorry,

1:11:53 I was in two meetings at the same time.

1:11:55 Yeah,

1:11:55 sorry.

1:11:55 So my,

1:11:56 uh,

1:11:56 yeah,

1:11:56 my question was regarding,

1:11:58 you know,

1:11:58 we're treating PBF,

1:12:01 um,

1:12:01 as essentially unitary,

1:12:03 that PBF is a single intervention,

1:12:05 but of course PBF is extremely different and

1:12:08 looks very different depending on the setting.

1:12:10 The level of the incentive is different.

1:12:12 The methodology is different.

1:12:14 The way payments are

1:12:15 made can be different.

1:12:16 It should be different.

1:12:17 And even within a certain project

1:12:20 as it progresses year to year,

1:12:22 typically we make methodological changes to adjust

1:12:25 to how the incentives that we're,

1:12:27 you know,

1:12:28 that we're trying to push are working.

1:12:30 We'll increase amounts,

1:12:31 we'll change incentives,

1:12:32 etc.

1:12:33 So I mean,

1:12:34 of course,

1:12:34 by it's by design this type of analysis can't.

1:12:38 Capture that,

1:12:39 but I think it's worth noting

1:12:41 that when we're comparing PBF,

1:12:43 we're looking at different things.

1:12:44 So

1:12:45 I think,

1:12:46 and I think this this was answered a little bit in the chat already saying that,

1:12:49 well,

1:12:49 you know,

1:12:50 the effect sizes aren't

1:12:52 necessarily all that different,

1:12:54 though it did seem that there were fairly wide confidence intervals,

1:12:57 but overall there aren't huge effects.

1:13:00 But I think still the question shouldn't be PBF,

1:13:02 yes or no,

1:13:04 but rather PBF,

1:13:05 you know,

1:13:05 when.

1:13:06 Where and how and how does it potentially work

1:13:09 in complement

1:13:11 with other interventions as well and my experience on this is,

1:13:14 you know,

1:13:14 at least the intimate experiences from one country in Cote d'Ivoire

1:13:18 where it worked quite well

1:13:20 and the country actually adopted it into their own

1:13:23 national strategy and started using it as the way

1:13:25 they reimburse at least part of facilities and also

1:13:28 integrated it with their social health insurance system.

1:13:32 So because there were

1:13:33 Payment mechanisms set up,

1:13:34 verification mechanisms set up,

1:13:35 bank accounts set up.

1:13:37 They said that they could actually use this

1:13:39 to reimburse facilities for using social health insurance.

1:13:43 So just to say the externalities from this can be huge as well

1:13:47 and probably aren't,

1:13:48 you know,

1:13:48 of course can't be evaluated in a study like this,

1:13:51 but are worth

1:13:52 considering when we start to come up with hard conclusions as to,

1:13:55 you know,

1:13:55 whether we should do PBF or not or other things.

1:13:58 Thank you.

1:14:01 Thank you.

1:14:01 uh,

1:14:01 Gary,

1:14:02 are you still there?

1:14:05 And able to unmute

1:14:09 Yes,

1:14:09 I am.

1:14:10 Can you,

1:14:11 can you hear me?

1:14:13 Sure,

1:14:13 and just in the interest of time,

1:14:14 can I ask you to be very brief?

1:14:16 Sorry,

1:14:16 thank you.

1:14:18 Go ahead very brief,

1:14:19 uh.

1:14:20 Good.

1:14:20 Uh,

1:14:21 I congratulate the,

1:14:22 uh,

1:14:22 the authors of the report.

1:14:23 I'm looking forward to read it.

1:14:25 I know it's a lot of work in it.

1:14:26 A couple of comments.

1:14:28 So one comment is,

1:14:29 I was always surprised about sort of this effort of trying to

1:14:33 calculate the effect of performance pay on the results.

1:14:36 We know from the international literature is the way you pay doctors and nurses

1:14:41 influences their behavior,

1:14:42 right?

1:14:43 So nothing new under the sun.

1:14:45 Um,

1:14:46 one comment on the Nigeria experience,

1:14:48 the PBF versus DFF.

1:14:49 Actually,

1:14:50 DFF is not a term

1:14:51 used outside Nigeria.

1:14:53 It's a Nigerian intervention.

1:14:55 One comment on that intervention is that

1:14:57 the impact relation showed that

1:14:59 PBF,

1:15:00 although it was twice as expensive as compared to DFF,

1:15:04 saved more than twice as many years of life as DFF

1:15:07 in the same period of time.

1:15:09 I mean,

1:15:09 the cost effectiveness was similar,

1:15:11 right?

1:15:11 The cost utility analysis is similar,

1:15:13 but this means that

1:15:14 basically you save more than twice as many years of life

1:15:17 as compared to DFF.

1:15:19 That doesn't come out of this,

1:15:20 uh,

1:15:20 this

1:15:21 mega analysis.

1:15:23 Thank you.

1:15:26 Thank you.

1:15:26 And,

1:15:27 uh,

1:15:27 before turning it back to the team,

1:15:28 let me call on Oyebola who's in the room here.

1:15:31 Thank you.

1:15:31 Thank you.

1:15:32 Great presentation.

1:15:33 Just wanted to see

1:15:36 if you had any frontline worker

1:15:39 insights you could share with us about this no can do,

1:15:42 um,

1:15:42 because you're seeing that at the same time they're overdoing

1:15:45 some things that are wrong and underdoing other things,

1:15:47 so it doesn't seem like it's a time

1:15:49 constraint

1:15:50 just.

1:15:51 It's been really good to hear some frontline,

1:15:52 um,

1:15:53 like the conversation with the minister in Ethiopia and just

1:15:56 hearing what people think is their own real constraints,

1:15:58 and so it'd be great to have any anecdotal evidence,

1:16:01 even you could share on that.

1:16:02 Thank you.

1:16:06 Thanks,

1:16:07 Deminiani,

1:16:07 over to you.

1:16:14 Thanks a lot for all the comments.

1:16:15 I,

1:16:15 I don't know if we can

1:16:17 uh address them all.

1:16:19 Maybe one thing I'd like to,

1:16:21 to start is,

1:16:22 uh,

1:16:22 follow up on Emery uh

1:16:24 uh

1:16:25 comment

1:16:27 because

1:16:28 we,

1:16:29 I just want to emphasize what he says and I,

1:16:31 I hope we,

1:16:32 we convey that message.

1:16:33 This report is not about PBF,

1:16:36 uh,

1:16:36 yes or no,

1:16:37 it's precisely about

1:16:40 when,

1:16:40 where,

1:16:41 And how,

1:16:42 as,

1:16:42 as um

1:16:44 Eray emphasized,

1:16:45 so I mean,

1:16:45 I know sometimes it's easy to simplify,

1:16:48 but

1:16:48 we really

1:16:50 hope to convey the message that

1:16:52 It,

1:16:53 there,

1:16:54 there are

1:16:55 times and indicators when PBF

1:16:58 can be,

1:16:58 uh,

1:17:00 um,

1:17:01 extremely,

1:17:02 uh,

1:17:02 useful,

1:17:03 um,

1:17:04 or Ebola,

1:17:05 some anecdotal dividends,

1:17:07 I,

1:17:08 from my observation when I've been and I've been for in many health facilities.

1:17:13 It's not,

1:17:14 I don't think the main constraint is time,

1:17:16 so it's more like maybe a routine of

1:17:19 not doing things or doing things.

1:17:21 There is also

1:17:22 an issue that that uh we've not touched too much in this presentation,

1:17:26 but it's in the report.

1:17:27 It's sometimes some of the things that are done

1:17:31 by health workers

1:17:33 are

1:17:34 not irrelevant,

1:17:35 but they are demanded by the patient,

1:17:37 so that that's another issue,

1:17:38 you know,

1:17:39 um,

1:17:40 so.

1:17:41 Um,

1:17:43 Um,

1:17:44 Ishani,

1:17:44 you want to take the Nigeria questions?

1:17:47 Sure,

1:17:47 um,

1:17:48 I wanted to address a few things.

1:17:49 Uh,

1:17:50 one is to just remind people that this is a 290 page book,

1:17:53 and there is,

1:17:54 Norbert,

1:17:55 even in your multiple readings,

1:17:56 um,

1:17:57 I,

1:17:57 I think

1:17:58 there,

1:17:58 there's some of the answers to your questions,

1:18:00 for example,

1:18:00 on DFF impacts lie in the report.

1:18:03 Um,

1:18:04 on,

1:18:04 you know,

1:18:04 Fang's point about

1:18:07 Just,

1:18:07 just to push back a little bit on,

1:18:09 you know,

1:18:09 why this didn't come up sooner,

1:18:11 uh,

1:18:11 we have been presenting these results as they've come together.

1:18:14 For instance,

1:18:14 the polled analysis,

1:18:15 the meta-analysis,

1:18:16 going back to the last HD week that was in person,

1:18:18 so I

1:18:20 believe

1:18:20 the summer of 2019.

1:18:22 Um,

1:18:22 and of course,

1:18:22 we've worked with the country teams as,

1:18:24 as Damien described for,

1:18:26 um,

1:18:26 approximately 15 years now.

1:18:29 Um.

1:18:30 Emery's point on the heterogeneity PBF,

1:18:33 there was also an excellent question raised in the chat about,

1:18:36 um,

1:18:37 by REM,

1:18:37 I think,

1:18:38 on,

1:18:38 um,

1:18:39 The importance of uh timely HM HMIS data.

1:18:42 Uh,

1:18:43 these are all things that we talk about in some depth in the report,

1:18:46 so please do look at it.

1:18:48 Um,

1:18:48 and Emery,

1:18:49 also your point about the externalities of the system level,

1:18:51 I think is really well taken.

1:18:52 This is a,

1:18:53 a,

1:18:53 a,

1:18:54 a thing that we acknowledge in the report that we can't really touch upon,

1:18:57 uh,

1:18:58 in the timeline of most impact evaluations,

1:19:00 right?

1:19:00 We're talking about perhaps

1:19:02 decadal impacts on the health system.

1:19:04 Um.

1:19:06 Oh,

1:19:06 Ebola,

1:19:07 on your frontline anecdotes,

1:19:08 you know,

1:19:08 there are many,

1:19:09 uh,

1:19:10 there are some qualitative studies that I can point

1:19:13 you to that are also cited in the report that

1:19:15 uh bring together a wealth of evidence,

1:19:17 but I think sometimes,

1:19:18 Uh,

1:19:19 just in sort of

1:19:21 economics,

1:19:22 uh,

1:19:22 terms,

1:19:22 there is just inattention,

1:19:24 right,

1:19:24 uh,

1:19:25 by the agent.

1:19:25 There's a lot going on.

1:19:27 Jishnu's work shows how,

1:19:28 uh,

1:19:29 there's a complex,

1:19:30 uh,

1:19:30 interaction,

1:19:31 I think,

1:19:31 with,

1:19:32 um,

1:19:33 uh,

1:19:33 uh,

1:19:34 facility infrastructure

1:19:36 with individual level incentives

1:19:38 and intrinsic motivation.

1:19:39 Ken Leonard's work shows this as well,

1:19:41 um,

1:19:41 as does Maoj Mohannan.

1:19:42 So,

1:19:42 uh,

1:19:43 all things to,

1:19:44 uh,

1:19:44 keep in mind.

1:19:45 And I

1:19:46 do want to come back to,

1:19:47 to Yuri's point about Nigeria and DFF.

1:19:50 Um,

1:19:51 Yuri,

1:19:51 I'll,

1:19:52 I'll see your,

1:19:52 uh,

1:19:53 Nothing New Under the Sun from Ecclesias and,

1:19:55 and quote you,

1:19:55 Pink Floyd.

1:19:56 Um,

1:19:57 you know,

1:19:57 I feel a little bit like we've had this conversation many times and we're like

1:20:00 two lost souls going around in a fishbowl.

1:20:02 At this point,

1:20:03 um,

1:20:04 you know,

1:20:05 as regards the,

1:20:06 the impact on,

1:20:08 um,

1:20:09 Um,

1:20:11 The lives saved.

1:20:12 That depends on a particular

1:20:15 assumption-driven

1:20:17 estimation that we are not entirely comfortable with,

1:20:20 I think on this team

1:20:21 about,

1:20:22 The quality that essentially is about the quality of public facilities,

1:20:27 uh,

1:20:27 which is often,

1:20:28 um,

1:20:29 uh,

1:20:29 unobserved.

1:20:30 And when we have detailed data as we do in this report

1:20:34 about

1:20:35 the quality of public facilities,

1:20:36 I think some of those assumptions seem quite heroic to us and,

1:20:39 and specifically for those who know the literature,

1:20:42 you know,

1:20:42 we're talking about the list tool here,

1:20:44 um,

1:20:44 and,

1:20:45 and so that's why we sort of stay away from,

1:20:47 uh,

1:20:47 some of that literature.

1:20:53 Thank you,

1:20:53 um,

1:20:55 We have

1:20:56 2 more minutes.

1:20:57 I'm gonna give

1:20:58 Ben a chance to come in.

1:21:00 Ben Levinson,

1:21:01 he's had his hand up.

1:21:02 Um,

1:21:03 uh,

1:21:03 Ben,

1:21:03 do you want to come in with a,

1:21:04 with a brief question?

1:21:06 There are just,

1:21:07 I,

1:21:07 you know,

1:21:07 uh,

1:21:08 first of all,

1:21:08 uh,

1:21:09 really intuitive force by,

1:21:10 by the team.

1:21:11 This is,

1:21:11 this is great.

1:21:12 And,

1:21:13 you know,

1:21:13 one thing that would be really helpful is to

1:21:15 see what are the lessons learned in terms of doing

1:21:18 these impact evaluations.

1:21:20 I think a lot of people will be skeptical,

1:21:22 but from my perspective,

1:21:23 it,

1:21:23 it,

1:21:23 you know,

1:21:24 it's been really valuable,

1:21:25 um,

1:21:26 you know,

1:21:26 and,

1:21:27 and,

1:21:27 you know,

1:21:27 really helpful.

1:21:28 I guess the real thing is that,

1:21:30 uh,

1:21:30 You know,

1:21:31 we,

1:21:32 most of the work that we've done

1:21:33 on PBF has been in the public sector,

1:21:35 and so,

1:21:37 you know,

1:21:38 the,

1:21:39 you know,

1:21:39 maybe it doesn't work so well in the public sector,

1:21:41 but it could work quite differently in the private sector,

1:21:44 which is why we see such difference,

1:21:46 differences in

1:21:48 provider performance when those providers are,

1:21:50 are,

1:21:51 are private

1:21:52 or non-salaried civil servants.

1:21:54 It'd be interesting to know what you think about this.

1:21:56 Thanks.

1:22:04 Uh

1:22:05 We are out of time.

1:22:06 I do want to give you a chance to react to that,

1:22:08 to that private if you have any thoughts on that private sector question.

1:22:15 Um,

1:22:16 so,

1:22:16 uh,

1:22:17 you know,

1:22:17 this is a conversation that I think Benjamin and I've had before.

1:22:20 I think the evidence is

1:22:23 suggestive that,

1:22:24 um,

1:22:27 Both patients and providers react differently to,

1:22:31 uh,

1:22:31 the particular financial incentive structure at place in the private sector,

1:22:36 um,

1:22:36 and that's,

1:22:36 I suppose I'll just end with,

1:22:37 that's a,

1:22:38 the topic for another report,

1:22:39 another 290 page report.

1:22:43 No,

1:22:43 but I,

1:22:43 I guess we should just be upfront.

1:22:46 It is mainly about

1:22:48 public,

1:22:48 the public sector and our report,

1:22:50 uh,

1:22:50 that,

1:22:52 uh.

1:22:55 Thank you.

1:22:55 I do realize there's some people who have their hands up.

1:22:57 There's some questions that I would have hoped to come to,

1:23:00 but we have run out of time,

1:23:01 and I do want to respect all your,

1:23:02 your other obligations.

1:23:03 So with that,

1:23:04 um,

1:23:05 let me thank the presenters today for a really interesting

1:23:08 presentation

1:23:09 of the highlights of the report,

1:23:10 and as Ishani mentioned,

1:23:12 there,

1:23:12 there's a lot more detail in the report,

1:23:13 and I would encourage all of you to go and check it out,

1:23:16 um.

1:23:17 Also to recognize some of the members of the team in the room

1:23:20 and as well as the members of the team who,

1:23:22 who couldn't make it here today,

1:23:24 some of who I think I saw a couple of,

1:23:26 well,

1:23:26 definitely saw a couple of them online.

1:23:28 Uh,

1:23:28 congratulations to all of them and please join me in giving them a hand,

1:23:31 uh,

1:23:32 this report.

1:23:35 So.

1:23:37 Thank you everyone.

showAllTimestamps

no

transcript

Excellent. Thank you. Well, good morning, good afternoon, good evening, everybody. Uh, my name is uh Dion Filmer. I'm the director of the World Bank's Development Research Group. And welcome to today's, um, which is the May edition of our policy research talk series. I'd like to welcome our in-person audience as well as our online audience both on Webex as well as on YouTube. For those in the room, we've been asked to please keep masks on unless speaking, and the whole room is set up for social distancing, so we're complying with all the rules and guidelines. Um, these policy research talks provide us with an opportunity to present work coming out of the research department here at the World Bank with the goal of sharing findings, uh, inside and outside the department, along with others outside the bank. Today's policy research talk is a special occasion because we're using it as an opportunity to launch uh, our new policy research report, Improving effective health coverage, uh, improving effective coverage in health. Do financial incentives work? The policy research reports are a product of the development research group that have the aim of synthesizing the most rigorous evidence available from within and beyond the bank on important topics in development. The reports benefit from our department's dual mandate of working closely with bank operations on projects while also producing cutting edge research on development. This particular report takes on an issue that's especially relevant in many low-income contexts. It looks at the evidence on the impact of financial incentives in health over 15 years and nearly 40 countries. Two of the lead authors, Damien De Walk and Ishahi Kanpal, will be co-presenting today. Damien is a lead economist in our human development team, and his work is currently focused on the impact of financial incentives on health and education outcomes. Ishani is a senior economist in our poverty and inequality team. And conducts research on financial incentives like cash transfers to poor households and pay for performance contracts with health workers. We're grateful to have Norbert Shady and Feng Zhao as discussants today. Norbert is chief economist for Human Development at the World Bank Group. He's published extensively in academic journals, and his main research topic areas include early childhood development, teacher quality, cash transfer programs, and the effect of economic contractions on the accumulation of human capital. Feng is the strategy operations and global engagement practice manager here at the World Bank. As World Bank practice manager, he oversees the Bank Global Health engagement program and has more than 20 years of experience in public health, medicine, economics, and demography. I'll ask uh Damien Nanni to speak for about 40 minutes, after which we'll hear from Norbert and Fen for approximately 5 to 10 minutes each. We'll conclude the session with questions and answers from, uh, the in-person and online audiences. If you have a question in person, uh, we'll ask you to come to the microphone at the start of the Q&A. If you're watching online and have a question, please use the raised hand option in Webex, and I will call on you. Um, if you're following on YouTube, please submit your question in the chat, and that'll get relayed to us. Uh, just a reminder, uh, we are recording the session and if you're following online, please do mute your microphone unless you're speaking. So with that, over to you, Damien Anushani. Thank you. Um, thank you very much, uh, uh, and for all of you for coming, uh, today, uh, in person or, uh, following on Webex on YouTube. So it's a pleasure for Ishani and myself to, uh, present you, uh, this report, this, uh, uh, policy research report entitled Improving Effective Coverage in Health. Do Financial incentives Work? So, I'd like to start by saying this is uh the work of a large team composed of colleagues in the research department at the World Bank as well as in the health, nutrition and population uh global practice at the World Bank. Um, and, uh, in addition to which Shani and myself, the team includes Adam Wagstaff, Jett Friedman, Sven Nielsen, Mauritz Piati Ffkirchen, Anel Soutmann. Gil Shapira and Helene Vanderpool, and recently we took this team picture outside of the bank, but since uh Ellen is uh sitting in Dakar, we had to uh Photoshop her in the picture. We also want to take this opportunity to dedicate this report to the memory and legacy of Adam Wagstaff. He was the driving force behind this report. He contributed numerous concepts and analysis. And as you, as many know, um, he untimely passed away two years ago. He still remains an inspiration for us, but we are missing him. So now I want to start with the big picture that motivates uh this report. And this big picture, we call it a frustrating status quo, status quo. So we are going to illustrate this frustrating status quo with um the example of pregnancies. And what we see around the world is that pretty much everywhere, everywhere, there are high rates of antenatal care visits around the world. That is. Most women who are pregnancy at least once. A service provider, they have an antenatal care check. On this map you see that the dark blue is more than 90%. The countries where more than 90% of women have an antenatal care check. It's a lot of countries and the other countries are also in this darker shade of blue, above 80% or 70%. But despite that, What we see is that at the same time that there is Widespread access to antenatal antenatal care. We still find high maternal mortality rates in many low income countries. So this is on this map you see maternal deaths per 100,000 live births. The dark blues are a country where maternal death is. Very low And the colors that are orange or red show you countries which are mainly known in West Africa or Central Africa, but also in Afghanistan where we still have shockingly and stubbornly high maternal mortality rates. And this is this is true not only for maternal health, but for a lot of other health outcomes. Indeed, the Lancet Commission on High Quality Systems conclude that most deaths in low income countries are due to the poor quality of care rather than the lack of access to care. There are still important gaps in equity in access to care. And most of, a lot of uh primary health care systems are still very fragile. And so now as we are hopefully coming out of this pandemic, and if we think about, you know, regaining the gains that occurred before the pandemic, we know there's been further decline or there's been decline during the pandemic. And as we want also to go beyond the gains that we had before the pandemic, we think we really need to focus on quality of care. So that's the motivation for this uh report. And if we think about quality of care, one problem that as economists we call the principal agent problem. It's present in health, but also it's common in many markets, especially in the public sector. What is the principal agent problem is that You have an employer, in the case of health would be the Ministry of Health, for example. Who does not observe the performance of or effort by the worker. In the example of health would be doctors or nurses or in schools it would be teachers, and that can lead to shirking by the worker or lack of effort. But at the same time, the employer can observe the outcome. But that outcome is not. Determined entirely by the effort of the worker, we know that health outcome, even with good treatment and can Go bad. So that's the, the main issue of the principal agent problem. Faced with this problem, one solution has been performance pay. It's like, so paying for performance is, is very common. Um, to deal with the principal agent problem, and it basically links prices to actions. It's used Across many public sector bureaucracies with applications in education, infrastructure, agriculture, public-private partnership, contracting with nonprofit, and in-house, and we are going to focus on the application in-house today. The goal of this approach is to improve the efficiency of public expenditure. And as I said, performance pay is based on the quantity and the quality of service delivered, and we'll we'll give some examples, and we want to stress that the introduction of performance pay was a paradigmatic shift from input-based financing, which was basically the status quo before, towards more transparency, transparency and accountability. Health is one example, an important one, but there's a big push across sectors for donor as well as domestic resources to be shifted towards a performance pay uh mechanism. So what we are going to look in this report is that performance pay and performance-based financing abbreviated by the acronym PBF Projects in Health. Let me first stress that PBF. Project performance-based project and performance. They are not one and the same. PBF are projects that actually deploy a package intervention, and that includes performance pay indeed, and that's one admittedly crucial component of PBF packages, but also other components in that package. More autonomy for the frontline health facilities, they can decide more about how they use their own budgets, etc. more transparency and accountability and more community engagement, and these are all also related to substantial effects on the financial management of uh of the health system. So these reforms, these performance-based financing packages have been supported. By a very large trust fund managed by the World Bank called the Health Results Innovation Trust Fund. It was mainly invested in performance-based financing approaches for maternal and child healthcare in low-income countries. It spans 15 years and 40 countries and one very strong. element of that trust fund is that it not only supported many projects but also many rigorous impact evaluations, and we are building on the evidence from these rigorous impact evaluations to to produce this report. It was also the opportunity for a sustained and fruitful collaboration between the research department here at the World Bank and the teams in the health sector at the World Bank. So what have we learned from, from these pilots? That's what we are going to try to answer today. So, and, you know, I could, and I've used that, that picture before. Uh, you know, you all know this glass of water, and if you're an optimist, it's half full, and if you are a pessimist, it's half empty, and when you have an impact evaluation, you have some. Uh, significant outcome, positive and significant outcomes for some, uh, for some of your results, but some of the outcomes you were expecting some results and you don't find them and you conclude mixed results and, and then you show this picture. But what we want to do, uh, today is go beyond that. We want to go beyond that um cliche of the glass half full and half empty and let you pick and decide whether you are optimist or pessimist. So more specifically we want to look at the content um of our findings and what we are going to do in this report is look at these five questions. First, what does the evidence say about performance pay in low-income country health system? We're going to assess the impacts on coverage and effective coverage. I will give you a more precise definition of what we mean by effective coverage, but basically think for it, for not think of it as health coverage with a minimum content and quality, and we are going to look at this for maternal and child health services. Second question is, what are the key, key constraints to quality? Which constraint to quality can be addressed by performance pay. And can we realistically expect performance pay to improve quality, and if yes, by how much? Third question, can demand-side approaches improve coverage and effective coverage? Of maternal and child health services. PBF performance-based financing is a supply side approach, but we can also act on the demand side, and we should also look at that. Fourth question is how do this performance-based financing project compare to what we will call and define shortly direct facility financing abbreviated by DFF. And finally, what are the key lessons that we can learn from this evidence for the design of new health financing reforms? Briefly, uh, giving you the snapshot of the geographic scope of the report, as you can see it's quite broad. Even though for some of the uh section we are going to focus on uh uh set of uh countries in sub-Saharan Africa. So let me go back to this distinction. Uh, Between performance-based financing, PBF and direct facility financing, DFF, and on the left you have the four components of PBF that I've already covered performance pay, operating budgets, and autonomy, transparency and accountability. And uh community engagement. Well, if you want. To know what is the effect it's basically the same. With the exception of performance pay, there is no mechanism tying the payment of the health facility and the health worker to the quantities and quality delivered, but the three other components are there. With this in mind, let me give you a sneak peek at our key takeaways. The first one is that overall we find little evidence for impacts for across the board performance pay in under-resourced and unfinanced health systems. Instead, we find that direct facility financing that I just defined with autonomy and accountability at the level of the frontline facilities can deliver many gains at lower cost and with relatively easier implementation. So what we suggest is that, is that for teams working in, in health reform is that before designing health financing reform. They should do an assessment in 3 steps. First, assess coverage versus effective coverage for their indicator of interest to identify where would lie the low hanging fruit for performance pay. Second, assess the constraint to quality of care that would ensure. Uh, that they are in the locus of control for the of the frontline worker, that is, the frontline worker can actually act on these barriers. Third. They should find that sweet spot where baseline utilization should have some room for improvement, but not be so low as to indicate that there are demand side barriers, because when there are demand side barriers, it would be best to include demand side approach and so we suggest often that it can be useful to combine demand and supply side approaches. And finally, we suggest also to sequence intervention and use performance pay strategically. So now I'll come back to this concept of effective coverage and try to motivate it and also give you a more complete definition. So what is effective coverage? It is health coverage that includes a minimum content and quality of care. And you can think of it as the product of coverage and quality. We we've tried to show that in that very simple equation where effective coverage is basically the product of coverage, that is the percentage of person in need of care for a specific medical condition X who actually consult, that's coverage times the percentage of people consulting who get relevant care. In the report, we operationalize effective coverage by investigating, investigating, sorry, how to measure it using existing data. What we do is use household survey. From, uh, for 93 countries and we show effective our effective coverage varies by country, by wealth level of the patient, and also across different medical conditions. We want to stress also that When we look at quality of care, we, we, we not only looked at insufficient care, whether not enough is done, but also at irrelevant care or overuse. When something is done that should not be done, either it's a waste or even it can be harmful. So we look at both issues when we look at quality. Let me give you one example, one illustration of effective coverage using the example of antenatal care. In this figure, on the horizontal axis, you have coverage going from 0 to 100, and it's defined as the person of woman giving birth with at least one antenatal care visit. On the vertical axis with quality of care also going between 0 and 100, and that's among these women who got at least one antenatal care visit, the percentage who had correct treatment according to WHO guidelines, which is defined as the percentage who had at least 4 visits. One of these visits with a skilled provider and for whom blood pressure was taken and blood and urine samples were taken. So then we can Draw what we call effective contours, lines of effective coverage, and these are, they are here for illustrative purpose, but they are basically lines that show different points of coverage and quality that gives you the same effective coverage. OK, so let's look at 6 countries. 5 sorry in Africa to see what the data tells us if we look at the Republic of Congo. We see a high level of coverage. Above 90% and fairly high level of qualities, around 80%, that gives us something around 70% of effective coverage. Now if we look at the two next countries, Cameroon and Nigeria, coverage is somewhat lower, between 70 and 85%, but quality is also markedly lower. It's around 60%, and that brings us to a lower level of effective coverage. And then the next two countries, the Democratic Republic of Congo and the Central African Republic, they've, if you look at it, they have similar levels of coverage compared to Cameroon and Nigeria, but their level of quality is much lower. It's below 40%. And as a result, we see that effective coverage is basically under 30. So with that illustration, I'll give the floor to Eshani. Who is going to uh, go more in detail into the quality of maternal and child health. Thank you. So as, um, as Damien sort of motivated, there's this gap between the coverage of maternal and child health services and the quality or the effective coverage, pointing to serious problems with quality in many of the contexts that we're gonna be talking about later today. Um, so what we do is to try and understand, you know, how you improve quality. As a first step, we try to understand what the constraints of quality are. In other words, why the rate of relevant treatment, uh, is not 100%. The literature often, um, points to health worker effort or the lack thereof as the binding constraint to quality in these sorts of settings. Um, if this is true, that, you know, health workers are sort of, uh, leaving money on the table by not doing everything they could do, and crucially, if effort is observable and it's contractable, uh, then performance pay might make sense. Performance pay to the health worker might in fact improve their effort. But is that the case? In fact, you know, if we think about it and we sort of systematically decompose quality, what we find is that the constraints to quality actually may lie at several different levels. The first is that uh of the health facility or perhaps at a higher level in the form of problems with equipment availability, drug availability, uh, availability of supplies. Now, you might say this should be in the locus of control of the health facility, but note that in, in centralized systems, you often have to go through centralized procurement to address these sorts of issues. So this might not even be in the locus of control of the facility, leave alone the health worker. It might lie further upstream in the medical system in the form of medical training, problematic medical training. So, you know, that's it with medical schools, and there's typically a centralized syllabus of what's, what's taught and how it's taught. Uh, there's, of course, issues of the public expenditure tracking system, uh, which can lead to, if there are significant delays in, uh, salaries being disbursed, problems of worker absenteeism. So, if, you know, if workers don't even show up at the health worker, uh, at the health facility. Then, of course, there is health worker effort, which is the component that is malleable to performance pay that's given to the health worker, right? And then finally, there's potentially patient-driven demand, which may combine or may not combine with the workers' effort, but likely depends on the patient's uh socioeconomic status. And, and we'll touch upon all of these, uh, aspects at various points today, but The main, the main message I want you to take away from this is that not all of the constraints are in fact controlled by frontline facilities or workers. And so there may be important limits at the very outset to what performance pay that's designed to go to the health worker to improve their effort could even achieve in these sorts of settings where there are competing constraints. So to formalize our thinking, uh, what we do is, uh, use the no can-do gap framework to decompose constraints to quality into the following three gaps. The first is, uh, structural capacity gaps, which, uh, which, you know, which tell us about what's available at the health facility and for which we need data on health facility infrastructure, so equipment, drugs, and supplies. We need, um, knowledge gaps, uh, uh, to assess knowledge gaps, we need data, uh, assessing health workers' uh, knowledge of clinical protocol for the types of care that we're interested in studying. Uh, for this, we typically use vignettes with health workers. Um, and then we can put these things together, once we have direct observations of patient-provider interactions to assess underused capacity, sometimes also called no can-do gaps, which is care that could have been provided, but is not. And this is again, the component of care that is uh malleable or amenable to performance pay directed to the health worker. Um, this is a data-intensive exercise, as you can see, but we can quantify these gaps for antenatal care consultations in five Sub-Saharan African countries, which are Cameroon, the Central African Republic, the Democratic Republic of Congo, Nigeria, and the Republic of Congo. Um, using this, um, this large HRITF impact evaluation portfolio that, uh, Damien has described, we, uh, we can, uh, leverage really rich data, sort of uniquely rich data on the patient, on the provider, and on the facility, characteristics of all of these three things, as well as patient-provider interactions. And what we do to benchmark the quality of care, to say that this is what should have been done, is use the WHO protocol for antenatal care. Um, and I'll, I'll talk about that in, um, Uh, a little bit more detail in the next slide, but we'll, uh, one other thing I want to note is that we assess two different manifestations of poor quality. The first is underperformance, which is where the health worker does only a share of what she should be doing in, uh, a patient-provider, uh, consultation for antenatal care. The second is overtreatment or irrelevant care, that should not be provided. Um, and, uh, the, the, there are two interesting things to note here. Um, the first is that these coexist. So, at the same time, health workers aren't doing everything that they're supposed to be doing, and they're doing things that they're not supposed to be doing. Uh, and the other thing is, uh, that some of the measures of irrelevant care, which I'm really not gonna have time to talk about today, but you can read about in the report, are actually harmful to the, the growing fetus, and they're still being done, uh, in these settings. OK. So the first um key finding that I want to show you here is that structural capacity gaps, knowledge gaps, and underused capacity coexist in these antenatal care consultations in all five of the countries that we examine. On your left here. Left, yes, on your left here, uh, is a snapshot of the essential equipment, drugs, and supplies that are needed to perform, um, that antenatal care visit as per protocol. So on the Y-axis, you have the, um, Some of the, the, uh, the, uh, the physical infrastructure that's required, and on the X-axis is the percent of facilities that had this item in stock and correctly stored on the day of the survey. And the main thing to note here is that, you know, these are facilities, frontline facilities that are supposed to and, and do in fact provide antenatal care on a routine basis. And yet even for these basic uh elements of care, um. Uh, elements of infrastructure, um, there's far from universal availability and you can, uh, you know, look at the report for the full set of, uh, um, structural capacity, uh, gaps. On the right here is where we put things together. I, I'm not gonna go into knowledge gaps, but you can again look at the report to see how knowledge gaps do exist for every aspect of care, but putting it all together on the right here is underused capacity, and what we find here, again, for just a snapshot of the actions, uh, but, uh, in the report, uh, we report, um, underuse capacity for all 16 actions on the WHO protocol. The first thing to note is that yes, there is underused capacity for every single aspect of, of care, for every single task that should be performed by the provider. That said, the highest level of underused capacity anywhere is actually for counseling for which you really don't need any infrastructure. And there, you know, we have an average underuse capacity of 50%, but overall, underused capacity only explains a third of observed underperformance, which is to say that 2/3 of poor quality are not actually attributable to poor worker effort. So there are many competing constraints in these settings, and performance pay is therefore unlikely to be a silver bullet in isolation. We also find that wealthier women are 3 times as likely to receive high-quality care as, as, uh, poorer women. The part of this is driven by facility choice and so there's lots of within-country, uh, variation in facility quality and so, shoring up, you know, the worst facilities within a country is, is, is a likely, uh, is likely, uh, a worthy investment. But we also have some evidence, uh, of within facility inequality, which is consistent with, although not the same thing as discrimination by the, by the provider, but that we do see that, uh, which may point to the role of user fees and patient information and interventions designed to, uh, to address these, uh, uh, issues there. But to summarize, you know, um, what I want to leave you with for this section of the report is that health system financing does appear to be a central challenge, but it has to affect address still, uh, despite decades of sustained investments, uh, gaps in structural capacity, in medical training, and at the same time, worker effort. So, to do so, to design a sort of a robust health system, uh, financing reform, you know, we suggest doing things like tracking coverage versus effective coverage ex ante. And you can use, as Damien showed, available household surveys to do this, to also assess the constraints to quality before along these lines, this sort of a decomposition before designing interventions that aim to improve quality. So, let's look at what sorts of interventions have been designed to uh improve effective coverage. You know, so the literature and performance pays, is quite well established, uh, in, in the health sector, uh, with evidence from high-income countries. Uh, this, these, uh, performance pay interventions, which include the US's Medicaid system and the United Kingdom's National Health System. Um, uh, focus on clinical quality. These are high-quality systems that have few other competing constraints, uh, other than work or effort that the performance pay is attempting to address. These are decentralized, well-resourced systems. Even so, the literature is at best, uh, sort of mixed in terms of impacts of performance pay, particularly on clinical quality. Furthermore, what we're gonna be talking about today are applications in a very different type of ah setting. So, low-income settings typically tend to be centralized. They don't have operating budgets to facilities. There's no autonomy over facility management and staffing, and the incentives, as Damien has described in these performance-based financing programs, uh, pair, uh, incentives to the facility for, uh, structural improvements because that's an, uh, that's a, that's a known constraint that was, uh, also, uh, attempted to be addressed with these interventions and at the same time, worker incentives, performance pay to the worker. The early evidence of PBF projects from these settings, from Rwanda and Burundi in particular, was, was quite promising. Um, Interesting. I'm getting an error message here. OK, there we go. Um, so in terms of the key features of the PBF projects, as, as Damien described, there are 4. let me just give you a, a concrete example of how that performance pay works. So this is an example for Nigeria. Uh, so it paid $12 for every woman who gave birth in a, in a health center, $1.20 for an antenatal care visit, and 80 cents for child immunization. It also provided unconditional financing for everything but staff remuneration to the facility and, and gave the facilities autonomy over how to spend that. Um, facilities could then, um, report, uh, performance on these purchase services typically every month, and then payments were based, uh, on these reports. In addition, there was a third-party audit, typically every quarter, uh, for, uh, of these, uh, of these reports. And this, you know, this is expensive to implement because you need the auditor to go to every frontline facility. So, for example, in Nigeria, 1100 facilities participated in only the pilot. Um, and so, you know, at scale, that's, that's many, many more facilities, that's expensive, that adds up, but it also creates, uh, uh, an opportunity for gaming or collusion. And finally, there's a community engagement. Uh, component wherein ward or village development committees sometimes attend the facility management, uh, uh, meetings. The scope of evidence that we're gonna focus on largely is uh from Sub-Saharan Africa for this component. Uh, and these are by and large, the first generation of PBF projects that use this sort of uh price checklist that I just gave you an example of. But we do, um, spotlight evidence from high and middle-income countries and later-stage PBF designs that I'll come to towards the end of my time today, uh, that use other approaches, uh, or, you know, combination of approaches to improve quality. And again, we're mostly gonna focus on maternal and child health, but we do also touch upon curative care and non-communicable diseases, with the hope of answering the question, you know, what is it that we learned from these donor-financed pilots about at-scale health financing uh reform that would use domestic resources? To answer this, the first comparison we are gonna look at is PBF versus business as usual. Um, I'm not really gonna go into the coverage impacts of PBF, uh, projects, but, um, uh, the recent Cochrane review by Sophie Witter and others, uh, finds that at least in, in, in most contexts, there's some improvements in terms of simple coverage, um, although there is a lot of heterogeneity in the impacts here. In terms of quality, uh, we're going to assess impacts on structural capacity and underused capacity going back to that no can-do gap framework. So let me show you structural capacity first. Um, but first, let me spend a second describing these types of graphs because I'm going to be showing you a few of these. Um, so, on the Y-axis are the outcomes of interest, above the dotted horizontal line are the summary indices. So here we're gonna be looking at overall structural capacity and then separately to understand issues of um supply chain, we look at the, the drugs index. Um, on the X-axis are the treatment impacts here, PBF versus business as usual, the dotted, um, vertical line at zero is zero impacts. Increases or improvements in this case instructural capacity, uh, would lie to the, uh, right, and, uh, decreases, uh, obviously to the left. As you see here, overall, there are no, for, this is just an illustrative example from Cameroon and Nigeria, but again, you know, go to the report, look at the Cochrane Review, uh, for more detail on these sorts of impacts. But, um, Uh, in, in general, we do find, at least on some aspects of quality, uh, some positive impacts, sometimes fairly large positive impacts, even if overall there isn't. But then when we turn to underused capacity, um, the data for this are more limited. Again, this is a fairly data-intensive exercise and, and we can only estimate these impacts for Cameroon and Nigeria again. But here, remember that underused capacity is undesirable. It's, it's money left on the table. Um, so increases are actually problematic, and yet, that's what we find evidence of, uh, you know, looking at, again, all of the, uh, aspects of care, uh, that the WHO says are necessary. And, and really no overall um reduction in underused capacity. The one other thing I want to note is that, yes, these are only two countries, but these are often viewed as two successful instances of the uh early stage PBF design. So despite that, you know, no uh large impacts of um Uh, of performance pay. So the bottom line here is, you know, performance pay in terms of quality is not a silver bullet for work or effort in these settings, and this is also consistent with limited, uh, evidence of impacts of performance pay and clinical quality, even with, uh, settings with fewer competing constraints. And if anything, the largest impacts we see are for structural quality, which may not be surprising given that that's where we saw a lot of the constraints upfront, the quality improvements, uh, and, and only a third of underperformance being tied to underused capacity. But this then begs the question, you know, are there more cost-effective ways of financing structural improvements? So, let's turn to some other ways in which financial incentives have been leveraged for effective coverage. Um, here, we're gonna compare both coverage and effective coverage impacts of PBF projects to conditional cash transfers and voucher, uh, programs on the demand side, and direct facility financing on the supply side. And, and, and I just want to note that, you know, as Damien's also said, these are not necessarily pure alternatives. They often can and should be used in combination and the sequencing also becomes really important. Here's the geographic scope of the evidence for this part of the report. Um, a lot of the, the PDF and DFF comparisons come from, um, sub-Saharan Africa, but, uh, but by and large, we do have, um, uh, relatively wide global coverage. So turning to the comparison of PBF and demand side alternatives, uh, we draw on a meta-analysis that pulls, uh, evidence from 52 programs from 30 countries. Um, and here on the right, uh, are the, uh, impact summarized in this graph. So we look at PDF, um, Program impacts, voucher programs, and conditional cash transfer programs, and the mean effect sizes are on the Y axis, um, for modern family planning and facility delivery, at least 4 antenatal care visits, and at least 1 postnatal care visit. Uh, in terms of effective coverage, we also look at maternal tetanus vaccination during antenatal care and full child vaccination. Um On average, it's true that, you know, all three types of financial incentives examined improve service coverage, um, and the, uh, the mean effect sizes vary between sort of 2% and 7%. These are modest mean effect sizes, right? These are Context where there's a lot of room for improvement. And by themselves, we're, we're talking about fairly um modest improvements, uh, from any of these. If anything, if you compare, for instance, uh, the impacts of PBF uh to voucher and CCT programs for in facility delivery here in the light blue, uh, you know, there may be some indications that CCTs and vouchers are actually more effective. Again, going back to sort of the, the, the constraints that are germane. The third and last set of comparisons that we look at is that of PBF and its uh supply side, uh, alternative, uh, which is uh DFF. Um, as Damien said, the only difference between these two is performance pay. However, the way in which transparency and accountability are operationalized in DFF projects is also a bit different. Um, so, in, um, DFF projects, there isn't this reporting and auditing, uh, setup. There's just a business plan or a dashboard typically that facilities you. So this is a much lighter touch, less expensive, easier to implement without that, uh, risk of collusion and, and gaming, uh, approach, uh, for, uh, transparency. In the HRITF portfolio, there are 5 projects that directly compare PBF and DFF approaches, and we look at all of them. Uh, these are Cameroon, Nigeria, Rwanda, Zambia, and Zimbabwe, 3 of which Cameroon, Nigeria, and Zimbabwe also included, uh, business as usual arm, so we can, uh, look at that comparison. And then in 2 of these in Nigeria and Zambia, I just want you to keep in mind when I show you the results on the next slide, that PBF actually disbursed twice as much as DFF. So it's not actually a 1 to 1 comparison. That said, here's what we find. So, on the Y-axis again are all of the um impacts of interest, and there are a lot of impacts here, just to show you that we looked at a lot of different outcomes, right? We're not cherry-picking here. Um, but, and we look at outcomes related to effective coverage and then also simple coverage. The only place where we, uh, find an impact of PBF over DFF is for institutional deliveries. And even some of that, so these are, you know, the, the significant impacts are the solid um markers. Even there, uh, some of the gain, although not all, not necessarily in, in some of the settings, but at least in a few of the countries, some of the gain is, uh, partially explained by a switch from the private sector to the public sector. It's being driven by an increase in institutional deliveries in the public sector. Where we don't know the quality of the private sector, and so this is sort of unclear in terms of impacts on effective coverage. So just as I wrap up, you know, a couple of other considerations and approaches and, and sort of strengthening program impacts and designing programs that might be more effective in terms of, uh, quality gains. Um, one thing that we, uh, highlight both from the meta-analysis and the decomposition of effects in Nigeria is the role of baseline coverage in, in the catchment area. So, if baseline coverage was really low, then that might be indicative of, uh, demand side constraints, uh, for whatever reason, it should be addressed through demand side approaches. If it's very high, it probably means there's room for improvement. Uh, there's little room for improvement, and, you know, performance pay, uh, might not adequately, um, reimburse the, the worker for the effort that's required. We also know from the high-income country literature, but also from uh an increasing number of examples from the HRITF portfolio that performance pay, if anything, may work best at the hospital level, so not at the primary level. Um, again, a question of competing constraints. Uh, we know, for example, from one really sterling example in Afghanistan, that private sector contracting can be fruitful grounds for PBF and that's not what we talk about here, we really focus on the public sector in the report, but this is an important thing to note. And then finally, you know, we don't talk about cost-effectiveness. Um, uh, there's limited evidence, and this is sort of a little bit outside our wheelhouse, but we do have, uh, aspects that we look at that are sort of That have important implications for cost-effectiveness. Uh, the first is that there are complementarities and tasks that might generate positive spillovers. The theory for this actually goes back to Holmstrom and Milgram, last year's, uh, Economics Nobel laureates, but we don't see that theory being leveraged in, uh, in the design of these performance pay contracts, and they perhaps should be. Um, we also know from a, a large body of literature, including going to conditional cash transfers and even payment for ecosystem services, but also from this HRITF portfolio, that prices sometimes, uh, serve to signal the importance of the task you're purchasing. And so the level of the price might be less important than that signal. Um, also from the HRRTF portfolio, we know the importance of using non-pecuniary tools to amplify the impact of the pecuniary ones, such as through the use of hand, uh, hands-on supervision, uh, for example, through the anatomical mama, momma Natalie and neo-Natalie models. I'm gonna leave you with this bottom line for financial incentives, and that is that financial incentives, you know, it is clear, improve service utilization and at least some measures of quality, but those impacts are small, and they leave lots of room for improvement. Um, in under-resourced central, uh, centralized health systems, performance pay, we find yields only modest additional impacts. Um, but flexible operating budgets and associated accountability measures can deliver significant gains by themselves without performance pay, while avoiding the, the large additional costs and complexity of design of performance pay interventions. And with that, Damien's gonna wrap up. Thank you very much, Ian and let me uh wrap up and try to focus on what could be the operational implication of these findings. So the key takeaways we think are the following. Overall little evidence for impacts for across the board performance pay, especially in those under-resourced and unfinanced health systems, and we think that direct facility financing with autonomy and accountability can deliver many gains at lower cost and with easier implementation. We note, for example, that performance pay make budgets unpredictable. Both from the point of view of the government as seen from the point of view of the health facility. And the cost, the costs related to verification in performance pay, which are necessary to enforce conditionality can often be quite substantial. For example, in Nigeria, 2/3 of the project administrative costs were attributed to the effort for verification. So it's true that uh risk-based system of verification could substantially reduce these costs, but still they remain important. In general, we find that performance pay is a complex intervention which, especially in those settings, can be difficult to scale. So now I'd like to finish with a few action items, and they don't necessarily come from the evidence. They are more like supported by the evidence. They are informed by what has been happening on the ground over the last 15 years and that we've observed that we were working with our colleagues in the health sector in the design of the, of these health financing reform. So our suggestion would be to start with, if you think about starting a new health financing reform, to start by. A set of assessments. First, assess using existing data coverage versus effective coverage to identify those low hanging fruit for performance pay. Also try to assess the gaps in structural capacity and in health worker knowledge. And try to measure what we've called the no can do gap, the difference between what is available at the health facility in terms of knowledge and equipment and what is actually done. And we think the sweet spot for performance pay is where baseline utilization has room for improvement. It's not too high, but it's not so low as to be indicative of demand side barriers. Um, which might best be addressed by the demand side intervention and also the sweet spot is also probably where there is a high level of no can do gaps. So basically they are underused capacity at the health facility. We also think that sequencing is key when you think about starting this new health financing reform. We suggest 18 to 24 months at the beginning after these assessments for demand side interventions and direct facility financing, and then repeat the diagnostics exercise and then introduce performance pay, performance pay for select indicators. We also think that emerging technologies can reduce the implementation costs. We are thinking about mobile money that can that can send the funds directly to the health facility and the health workers, and in terms of accountability, uh, there are new apps, etc. that can develop portals and dashboards to look at both quantity, quantity quality as well as payments. Let me finish with 5 key principles for sustainable health financing reform. Again, those are not necessarily new, but I think looking at the evidence from our report, it's important to highlight them again. The first one would be to send funds to the front line to those health facilities serving people in the village and neighborhoods. Second one, ensure transparency and accountability through good facility financial management. Third one, when necessary, introduce routine supervision that should be supportive, not punitive. It's important also to introduce an output or performance orientation. But we note that this can be done through dashboards or portals and not necessarily or not always with just performance pay. And finally, it's important to unify the payment system with one system to push the payment to the front lines. With this, we're looking forward to the discussion and thank you very much for your attention. Thank you, Damien Ushani. So now we'll talk, turn straight to our discussions. Uh, first up will be Norbert, uh, here, and then we have Feng who is online. Uh, we'll go in that order if that's OK. Thank you. Thanks, Dion, and uh thanks uh Damien and Ianni for asking me to, to comment on this. Um, it's great to be back uh here. Um, I've also read the report at great length as uh more than once actually, uh, since I was one of the peer reviewers, and some of my comments are comments that I made at the time, um, and I just, I want to reiterate some of them. So this is clearly a very serious research product, um, in the best tradition of uh of PRRs I think. Uh, it brings together as much of the evidence as is available on the effectiveness of a particular kind of intervention, and then tries to make sense of this evidence sort of more broadly, not just the individual, um, the individual case studies. It's obviously highly, uh, policy relevant. Um, so that that's, that's really important. And it's particularly policy relevant, I think, as we make the transition from sort of the focus on the pandemic to sort of the more underlying, if you want, routine, uh, problems that, uh, health systems face in, in many of the countries that we work on. I, I sat in during the spring meetings that many of the, uh, conversations we had with the ministers of finance and ministers of health and education and so on. Certainly in the conversations on health, uh, in, in, in sub-Saharan African countries, there was often a feeling that Um, we as the bank were perhaps too concerned with things related to the pandemic and not concerned enough with some of the underlying health issues, um, that, uh, that these countries face. So I think this is really an important report in that regard as well. I think there's, there's so much that is useful in this report, so, um, I'm gonna just highlight a few things, um. Uh, and this list is by no means meant to be the full list by any means. Uh, I think the descriptive results on, on effective coverage are, are extremely useful, the ones that are there for 150 some countries. I think of those kind of like what the WDR for education. With lays showing that it's not just coverage that is important, but effective coverage. It doesn't make much sense to have coverage if people don't receive care that is useful in some ways, just as with education it doesn't make much sense to have kids brought to school if they don't learn at least something. So I think that is really, really useful. I like these effective coverage contours of the authors present, which sort of make really clear the point that you can increase effective coverage either by increasing coverage, or by increasing quality or some combination of the two. I will say that these contours are sort of, uh, are, I mean, as the authors well know, this is not a point that they don't know, but they're, but they're silent on the fact that two different points on the same contour could have very different distributional implications. So you could basically You know, have coverage for a very small fraction of the population with very high quality, and you would be on the same point as having lower quality, but broader coverage. So I think it's just something to keep in mind when we sort of make sense of these effective coverage contours. Um. The results to my mind are, are, are quite sobering. Um, you know, the authors argue that PBF is no, no silver bullet. I know they want to get away from this glass half empty, glass half full analogy, and I understand. Um, but on the other hand, for me, and I don't think it's only because I'm not the most optimistic person in the room, but I think it's hard if you want for me not to come away from this, saying that it looks to me more like the glass half empty. Uh, than the glass half full. Um, I think we should take these results very seriously. I mean, we're all searching for solutions to really difficult, uh, development problems, um, and we really want, uh, things to work, and sometimes things that we think have to work, and that really they have to work. Uh, well, maybe they don't work, and, and maybe, maybe we get carried away, uh, by our enthusiasm, uh, somewhat. I also like the fact that in the report, I, I very much like the no can do framework, uh, jokingly, I when I first started telling somebody else about this, they, they, they thought I was saying no can do, like no can do. And maybe that is, uh, maybe that is one of the messages, no can't do this stuff. I mean, you know, really, it's too complicated, no can do, but no, in a, on a more serious level, um, I, I think it's really useful to try to identify what, where the constraints are to having more effective, more. Effective uh coverage. And I think what the authors point out that many of the problems and potential solutions are things that don't, don't depend really on health worker effort that they're not really in the locus of control of that particular health worker. They don't have the right materials or infrastructure in the health center or the or the training and education that they received wasn't adequate. And so these are things that are not going to be solved by by trying to more effort and and the authors, they don't cover it here, but they go at in some length in the report about, I mean, this to me, the staggering levels of coverage uh or or treatment that is ineffective and potentially harmful. I mean, these are huge numbers. I had no idea, and surely that shows just my ignorance, but I, but if that is my ignorance, it probably is an ignorance shared by many others, but the levels are really, really big, and the fact that in some cases, The performance-based financing makes this worse is definitely a source of concerns. Um, can you guys put up slide 41 for a second? Because, uh, now comes the, the, the, the part of where I, where I sort of like wonder a little bit about some of the conclusions that, that we draw, only if it's possible, if it's, so that's slide 41. So I think this is a really important slide, because really at the end, what the authors conclude is that uh Uh, uh, direct frontline financing, DFF, um, can have the same effects as PBF, and in some sense that maybe this is what we should be doing instead of PBF. But really, what I see there is that DFF had no impacts either relative to business as usual. So here again, you look at the coefficients, if you want to look at, you know, which ones are the the DFF versus none. So those are The Little squares, blue squares, and I don't, I see as many, just about negative as positive, and, and no significant results. So it's hard for me to conclude from this. I mean, I understand here we are in the glass half empty, glass half full, and we're saying, well, you know, PBF is a problem. It's too complicated, you know, it, it, it may give us the wrong incentives, but I don't see much evidence here that at least in these, uh, in, in these countries. DFF, so this direct transfer of resources to the frontline workers with accountability and, and so on, that this actually improved outcomes. Um, it seems to me that it's, it's a bit speculative to say that it improved outcomes, unless we're somehow thinking that, you know, as this thing gets institutionalized, if you want, sort of over time, you know, people get used to it and this improves outlook, but here in this figure. I don't really see any evidence of improved outcomes either from PBF or from DFF, and I think that's, I think we want to be honest with ourselves about that as well. Um, Couple of more points. Uh, the authors point out that, uh, the early results from PBF from, uh, uh, Rwanda and Burundi were encouraging, and that's really interesting to me. So why were those encouraging and, and the rest, and the rest not? Um, it, it, it, uh, I mean, I don't know the answer, and, and, uh, maybe somebody else knows the answer, but it seems like again, because we saw those as encouraging, we sort of went in. You know, fully into this approach, and were then disappointed. So some reflection about why those were encouraging and the rest were not, I think, would be really useful. These are based, I'm reflecting now on the on the presentation more than on the report, but I think that would be useful. Uh, a few, uh, sort of qualifications. Um, I think, uh, the the authors well acknowledged that, uh, the focus is mainly on maternal and child health, and it's just something to keep in mind that we're talking of maternal and child health, not lots of other things, uh, in, in health. Um, we're also not talking directly of health outcomes. Um, we're talking about the use and the quality of these services. I think all of us have some doubts about how directly the outcomes are linked to the use, and so it's just, again, it's something to keep in mind. I don't find the comparison with CCTs. I've said this to the authors before, and it's fine, but I don't find the comparison with CCTs all that persuasive because it really is a very different geographic sample. Uh, when we're looking at CCTs, the evidence is largely from middle income countries, especially in Latin America, and the evidence from these PBF and DFF pilots is, is mainly, uh, from, from sub-Saharan Africa, mainly, I, I should say, at least the results that are stressed in the presentation. So I don't find that comparison as As as persuasive, but overall, I think this is really an important report. Um, I think we all need to reflect in our different role in the bank. We need to work in our conversations with policymakers. I think we all need to reflect. Uh, on, on what messages we want to take away from this, um, I found, I learned a ton, uh, reading it, and it leaves me with a lot of questions about where we should go next, uh, in, in, in these kinds of interventions. But, uh, thank you very much again, to, to everybody for asking me to be a reviewer and then for providing comments. Thanks. Thanks Norbert for those reflections. Um, we will have a chance to go back to the team to maybe pick up on some of them, um, uh, but right now I'd like to turn to Defeng who's online, um, for, uh, his, uh, reactions and comments. Thank you. Thanks, Tia. Can you hear me? Hello Dean, can you hear me and see me? Uh, we can hear you very faintly. I don't know if it's on our side or your side. Hello, is it better? OK, we can hear you in the room, fine, so. OK, great. OK, perfect. Thank you. Let me start. Very happy to be here. Um, I first want to echo, uh, Norbert's point. This is, uh, to me, a high impact report. It has a lot of policy implications, um, not only to this particular area, but also to our health, um, program, uh, as a whole. So I, I really want to, um, commend the team for getting this done and get the report in place. I would have hoped and this report could have come much, much earlier, but Nevertheless, it has brought a lot of policy thinking and policy reflections. So let me, let me say that upfront. And this is also a topic very close to my heart. Um, I'm actually on leave, I'm on my way to Michigan. to accompany my kids for his sports. Um, but when Moor asked me, I immediately said yes, um, because, um, When the RPF took off, um, I was a frontline TTL. I was based in Ethiopia. I can replay some of the conversation. I think it's relevant here. Let, let me just maybe put this, uh, as my first point. But before that, I want to say I'm, I'm very happy to say many colleagues have joined and they are more qualified than me, uh, in these areas. I, I'm also eager to, to hear their comments uh during the Q&A time. Um, The first question, the first reflection I want to put on the table is really, as I said, from a frontline TTO point of view. When I was based in Ethiopia when this program sort of took off, um, we were very excited, um, because we all tired about the stagnation of the poor health outcome and we have been working on input-based financing for a while. We need something new, um, something exciting. Uh, when this came out and together with a lot of analytical underpinning and case studies, it's not, you know, there's no, no, um, Uh, no science behind it. Um, so we were very excited, brought a lot of energy in our policy dialogue, um, and together with a lot of workshops and taking place in Africa and, and our financing, there's a trust fund, large size of trust fund also behind it. Um, but once we take on the, um, the, the policy dialogue with, with Minister of Health, Minister of Finance, and we run into a few You know, Realistic issues. I remember the Minister of Health at that time was Doctor Tetras, now head of WHO basically told us Uh, Ethiopia graduate 200 doctors per year. This way, we have, um, by directing your financing towards some health coverage targets will not resolve, uh, the fact that we don't have enough doctors, right? So that's, that's the constraint, um, they are facing. So just, just give that one example, um, that may be generated to 1.15. Um, we know, um, you know, money cannot resolve all the health issues. Um, the, the solution often lie in a package of, um, factors, sometimes beyond health sector. But because we are working on financing, financing institution, inevitably we, we looking for solution toward that direction. So that's, that's maybe the value of this report really got us thinking. I, I really like some of the conclusion of the report and some of the, uh, the message make me pause quite a bit to, to reflect, but nevertheless, I think this report has really Hit the heart of our, our core sort of health um program and, and make us think about uh what's, what's going to work, uh, how we can best help our clients. So that's one message I want to share. The second one is really um a forward looking um angle. Um, So I, you know, now the RBF um were Performance-based financing or some other terminology, um, the incentive we provide has evolved into a different phase. So we have an instrument called P4R. Um, many of the, this type of initiative has uh integrated into The overall health financing program. For example, many insurance agencies have quality based measure, um, you know, for them to contracting out services and a lot of those performance financing element also embedded into, into that regard. So we are actually in a new phase. So that's why I want to encourage the team to think more in terms of the next step. I, I share Norbert's view. Um, Let's go beyond the comparison between RPF, DDF, or CCT. There's many other options out there. Um, I like some of the comments thing. I will put a false dichotomy in there. Um, to me, um, think about the instrument of P4R will be much more meaningful for our, uh, internal audience than compared with those different options. Sometimes the, the solution is really the package of them. It's not really one or another. Um, I demand demand, I, I, the, the slide the last. Last slides were the 2nd to the last slide on the sequencing. I, I, I question that. Uh, that also leading the philosophy of, you know, have low hanging fruit, have analytical work, but eventually land on performance pay. So maybe that's not the sequence we want to put forward. Um, I know maybe you, you have different thinking, but I just question the sequence still uh working towards um performance pay. This is something maybe we, we can, we can think about it on the way forward. Recently, we have um a conversation with There's a lot of comments from colleagues from the HMP TTLs. One comment is the capacity building, um, component. Which is not the P4R, but it's really uh a TA component to get come together with is so critical, will help to address some of those uh bottlenecks and constraints uh in the system. The next point I want to put on the table is really about our decision making. And this morning in our HD forum, I think a few colleagues, Xiang Nan, and China, uh, and Norbert, uh, you know, have put a session really on lesson learned for COVID. Uh, there's a lot of quick decision, um, have been made, um, um, for COVID response, uh, without really solid, um, evaluation or data or, or scientific evidence, but we have to make it up, right? So to, to move forward. Um, that sort of also related to this conversation, right? So the decision was made at the time. Uh, move forward, um, pushing for performance pay, um, but What's the science and, and, and evidence behind it at the time and what's the evaluation which can help us uh make the adjustment. Um, I want to challenge all of us. Um, I, I hope that this study You know, if we've done many years earlier, we will really help, help everybody. This is nobody's fault. Um, but I want to challenge two things. One is Um, have we learned from this? Uh, do we have, um, early investment on data, on science, on research, which will help us, um, to make some decisions in real time. Uh, this first one. Second, have we embedded evaluation, um, quickly into our ongoing, uh, operations? In COVID, I appreciate IEG quickly put Put together some evaluation for our response, which have provide a lot of uh insights. But um I hope we have done in many areas. Uh, lucky in this area. Um, the next point I want to make, um, is we shouldn't oversell. The service coverage, um, contribution to better health outcomes. We know house is complicated. It's multiple production function. Even you'll have perfect effective coverage, you have perfect quality, you probably still cannot get um the desired house outcome. Let's, let's be careful because the, the social determinants, the, the other determinants of health outcome. So let's be mindful. I see some of the languages they make me worried. Um, so that's, um, another point I want to make. Um, looking forward, I really want to encourage we all think about those, uh, issues. Um, I do feel some of the strong elements, um, the principle demand has laid out. I, I really like them. So financing to front line, um, Those, you know, maybe by itself, um, if it's not necessarily lead to performance pay, it will make a big, big difference. Maybe the solution will stop there. Um, so let's, that's why I question the sequence. Um, just, um, I guess my point is, is, let's be mindful about the package of services, solution we put forward. Um, many colleagues have mentioned the silver bullet argument, um, but in reality, I see inevitably sometimes we bring a solution. Before, uh, we see the reality. So that's something I take home. I feel, um, it's the really the value of the reports to reflect on those important issues. Then, let me stop here. I have more to say, but let me stop here. I want to hear from, uh, the audiences also. Thanks, Fang for those comments, both very specifically to this analysis and this report, but also some of the big, bigger challenges I think you, you're posing out for, for, for the, the HNP sector as a whole and probably even beyond. Um, so, We have a couple of, we have about a little bit more than 10 minutes left. I propose we, there are a couple of people raised some questions in the chat that I don't think were necessarily answered. Um, so I might call on just a couple of people who put in questions. Um, can I ask you to please keep your questions to sort of under, under, you know, under a minute, and then we'll give the chance to the team to, to kind of respond both to the discussions and to these specific questions, and then if we have some time, we can do another round. Um. So with that, I'd like to call in order just to give you a bit of a heads up on Adana, Emre and Guri Frits. Um, can I call on you each in order to please unmute and ask a question, uh, again, keeping it as brief as possible, and then, um, then we'll, uh, turn it back to the team. So with that, uh, Ana, are you able to unmute and ask a question? Oh, OK, Adena has left, so can we go to Emery? Is Emery still there? I am. Just one moment, please. Sorry, I was in two meetings at the same time. Yeah, sorry. So my, uh, yeah, my question was regarding, you know, we're treating PBF, um, as essentially unitary, that PBF is a single intervention, but of course PBF is extremely different and looks very different depending on the setting. The level of the incentive is different. The methodology is different. The way payments are made can be different. It should be different. And even within a certain project as it progresses year to year, typically we make methodological changes to adjust to how the incentives that we're, you know, that we're trying to push are working. We'll increase amounts, we'll change incentives, etc. So I mean, of course, by it's by design this type of analysis can't. Capture that, but I think it's worth noting that when we're comparing PBF, we're looking at different things. So I think, and I think this this was answered a little bit in the chat already saying that, well, you know, the effect sizes aren't necessarily all that different, though it did seem that there were fairly wide confidence intervals, but overall there aren't huge effects. But I think still the question shouldn't be PBF, yes or no, but rather PBF, you know, when. Where and how and how does it potentially work in complement with other interventions as well and my experience on this is, you know, at least the intimate experiences from one country in Cote d'Ivoire where it worked quite well and the country actually adopted it into their own national strategy and started using it as the way they reimburse at least part of facilities and also integrated it with their social health insurance system. So because there were Payment mechanisms set up, verification mechanisms set up, bank accounts set up. They said that they could actually use this to reimburse facilities for using social health insurance. So just to say the externalities from this can be huge as well and probably aren't, you know, of course can't be evaluated in a study like this, but are worth considering when we start to come up with hard conclusions as to, you know, whether we should do PBF or not or other things. Thank you. Thank you. uh, Gary, are you still there? And able to unmute Yes, I am. Can you, can you hear me? Sure, and just in the interest of time, can I ask you to be very brief? Sorry, thank you. Go ahead very brief, uh. Good. Uh, I congratulate the, uh, the authors of the report. I'm looking forward to read it. I know it's a lot of work in it. A couple of comments. So one comment is, I was always surprised about sort of this effort of trying to calculate the effect of performance pay on the results. We know from the international literature is the way you pay doctors and nurses influences their behavior, right? So nothing new under the sun. Um, one comment on the Nigeria experience, the PBF versus DFF. Actually, DFF is not a term used outside Nigeria. It's a Nigerian intervention. One comment on that intervention is that the impact relation showed that PBF, although it was twice as expensive as compared to DFF, saved more than twice as many years of life as DFF in the same period of time. I mean, the cost effectiveness was similar, right? The cost utility analysis is similar, but this means that basically you save more than twice as many years of life as compared to DFF. That doesn't come out of this, uh, this mega analysis. Thank you. Thank you. And, uh, before turning it back to the team, let me call on Oyebola who's in the room here. Thank you. Thank you. Great presentation. Just wanted to see if you had any frontline worker insights you could share with us about this no can do, um, because you're seeing that at the same time they're overdoing some things that are wrong and underdoing other things, so it doesn't seem like it's a time constraint just. It's been really good to hear some frontline, um, like the conversation with the minister in Ethiopia and just hearing what people think is their own real constraints, and so it'd be great to have any anecdotal evidence, even you could share on that. Thank you. Thanks, Deminiani, over to you. Thanks a lot for all the comments. I, I don't know if we can uh address them all. Maybe one thing I'd like to, to start is, uh, follow up on Emery uh uh comment because we, I just want to emphasize what he says and I, I hope we, we convey that message. This report is not about PBF, uh, yes or no, it's precisely about when, where, And how, as, as um Eray emphasized, so I mean, I know sometimes it's easy to simplify, but we really hope to convey the message that It, there, there are times and indicators when PBF can be, uh, um, extremely, uh, useful, um, or Ebola, some anecdotal dividends, I, from my observation when I've been and I've been for in many health facilities. It's not, I don't think the main constraint is time, so it's more like maybe a routine of not doing things or doing things. There is also an issue that that uh we've not touched too much in this presentation, but it's in the report. It's sometimes some of the things that are done by health workers are not irrelevant, but they are demanded by the patient, so that that's another issue, you know, um, so. Um, Um, Ishani, you want to take the Nigeria questions? Sure, um, I wanted to address a few things. Uh, one is to just remind people that this is a 290 page book, and there is, Norbert, even in your multiple readings, um, I, I think there, there's some of the answers to your questions, for example, on DFF impacts lie in the report. Um, on, you know, Fang's point about Just, just to push back a little bit on, you know, why this didn't come up sooner, uh, we have been presenting these results as they've come together. For instance, the polled analysis, the meta-analysis, going back to the last HD week that was in person, so I believe the summer of 2019. Um, and of course, we've worked with the country teams as, as Damien described for, um, approximately 15 years now. Um. Emery's point on the heterogeneity PBF, there was also an excellent question raised in the chat about, um, by REM, I think, on, um, The importance of uh timely HM HMIS data. Uh, these are all things that we talk about in some depth in the report, so please do look at it. Um, and Emery, also your point about the externalities of the system level, I think is really well taken. This is a, a, a, a thing that we acknowledge in the report that we can't really touch upon, uh, in the timeline of most impact evaluations, right? We're talking about perhaps decadal impacts on the health system. Um. Oh, Ebola, on your frontline anecdotes, you know, there are many, uh, there are some qualitative studies that I can point you to that are also cited in the report that uh bring together a wealth of evidence, but I think sometimes, Uh, just in sort of economics, uh, terms, there is just inattention, right, uh, by the agent. There's a lot going on. Jishnu's work shows how, uh, there's a complex, uh, interaction, I think, with, um, uh, uh, facility infrastructure with individual level incentives and intrinsic motivation. Ken Leonard's work shows this as well, um, as does Maoj Mohannan. So, uh, all things to, uh, keep in mind. And I do want to come back to, to Yuri's point about Nigeria and DFF. Um, Yuri, I'll, I'll see your, uh, Nothing New Under the Sun from Ecclesias and, and quote you, Pink Floyd. Um, you know, I feel a little bit like we've had this conversation many times and we're like two lost souls going around in a fishbowl. At this point, um, you know, as regards the, the impact on, um, Um, The lives saved. That depends on a particular assumption-driven estimation that we are not entirely comfortable with, I think on this team about, The quality that essentially is about the quality of public facilities, uh, which is often, um, uh, unobserved. And when we have detailed data as we do in this report about the quality of public facilities, I think some of those assumptions seem quite heroic to us and, and specifically for those who know the literature, you know, we're talking about the list tool here, um, and, and so that's why we sort of stay away from, uh, some of that literature. Thank you, um, We have 2 more minutes. I'm gonna give Ben a chance to come in. Ben Levinson, he's had his hand up. Um, uh, Ben, do you want to come in with a, with a brief question? There are just, I, you know, uh, first of all, uh, really intuitive force by, by the team. This is, this is great. And, you know, one thing that would be really helpful is to see what are the lessons learned in terms of doing these impact evaluations. I think a lot of people will be skeptical, but from my perspective, it, it, you know, it's been really valuable, um, you know, and, and, you know, really helpful. I guess the real thing is that, uh, You know, we, most of the work that we've done on PBF has been in the public sector, and so, you know, the, you know, maybe it doesn't work so well in the public sector, but it could work quite differently in the private sector, which is why we see such difference, differences in provider performance when those providers are, are, are private or non-salaried civil servants. It'd be interesting to know what you think about this. Thanks. Uh We are out of time. I do want to give you a chance to react to that, to that private if you have any thoughts on that private sector question. Um, so, uh, you know, this is a conversation that I think Benjamin and I've had before. I think the evidence is suggestive that, um, Both patients and providers react differently to, uh, the particular financial incentive structure at place in the private sector, um, and that's, I suppose I'll just end with, that's a, the topic for another report, another 290 page report. No, but I, I guess we should just be upfront. It is mainly about public, the public sector and our report, uh, that, uh. Thank you. I do realize there's some people who have their hands up. There's some questions that I would have hoped to come to, but we have run out of time, and I do want to respect all your, your other obligations. So with that, um, let me thank the presenters today for a really interesting presentation of the highlights of the report, and as Ishani mentioned, there, there's a lot more detail in the report, and I would encourage all of you to go and check it out, um. Also to recognize some of the members of the team in the room and as well as the members of the team who, who couldn't make it here today, some of who I think I saw a couple of, well, definitely saw a couple of them online. Uh, congratulations to all of them and please join me in giving them a hand, uh, this report. So. Thank you everyone.

showAllTranscripts

no

duration

PT1H23M40S

scene7File

worldbank/PRRLaunchEvent_Trim

scene7Domain

scene7FileAvs

worldbank/PRRLaunchEvent_Trim-AVS

title

PRRLaunchEvent Trim

description

PRRLaunchEvent Trim

showTimestampAndTranscript

yes