Advancing the science of delivery: A proposed impact evaluation of inspection regimes in health care and their impact on patient safety standards

A growing body of experimental evidence points to the effectiveness of audits and monitoring (accompanied by rewards or sanctions) in improving the performance of service providers. In Kenya, where the government is working to expand health care access and improve quality, policymakers are interested in understanding what’s most effective and sustainable for strengthening patient safety standards in both public and private facilities. This evaluation, implemented with the Ministry of Health, will pilot a program that has the potential to be scaled up within the government’s existing system, providing valuable evidence for future policymaking in Kenya.

Problem
Health clinics in Kenya have limited budgets and safety standards are low. Regulations that do exist are poorly enforced, making it difficult to ensure health care quality and patient safety.

Intervention
The Government of Kenya, with the World Bank, is testing different inspection approaches to improving compliance with patient safety standards in both public and private facilities.  The pilot started in November 2016 in three counties -  Kakamega, Kilifu and Meru – and covers around 1,100 healthcare providers. Impacts will be assessed after one year.

Evaluation design
A randomized control trial is being used to evaluate the relative effectiveness of three different approaches to health clinic inspections. In the first treatment arm, health facilities will be inspected more than once by regional teams of full-time inspectors trained and supervised by the Ministry of Health and regulatory bodies, with warnings and sanctions for non-compliance. In the second treatment arm, health facilities will face the same inspections and possible sanctions as in the first treatment arm, but inspection results will also be prominently posted on the health facility’s door. Clinics in this treatment arm will receive a detailed checklist of procedures they are expected to follow and they can ask for a re-inspection if they want to change their grade. Health facilities in the control group will continue with “business as usual” inspections, which happen rarely.

Measurement
Compliance with safety and quality standards will be measured through unannounced visits by standardized patients (which will appear as real patients to healthcare providers), direct observations of patient-provider interactions, and administrative data.

Impacts to date on policy decisions and program design 

• Baseline survey. Checklists to measure adherence to standards used in baseline measurement led the Government of Kenya to make two wide reaching changes in how health facility quality is measured:
  • The government collaborated with the evaluation team on a new regulatory framework (Cap 242, Legal Notice No. 46 on March 21, 2016) for monitoring private and public health facilities to promote patient safety standards. For more information, see this case study.
  • The Ministry of Health, local councils, and regulatory boards collaborated with the evaluation team to create indicators appropriate for monitoring patient safety in Kenya.

Intervention and evaluation

The project will operate in three Kenyan counties—Meru in the central region, Kakamega in the Western region, and Kilifi on the coast. These three counties were specifically chosen because they contain a combination of different types of health markets, ranging from stand-alone clinics to dense markets.  Across the three counties, there are around 1300 health facilities, representing nearly 10 percent of all health facilities in the country.

The facilities will be grouped into “health markets,” geographical clusters where facilities are in close proximity to each other. “Markets” of facilities will be divided into two treatment arms – each with around 430 health facilities divided into around 91 markets -- and a control group with the same number of markets and clinics. Both groups of facilities will be inspected according to a pre-determined checklist to determine patient safety standards. Researchers will then compare results with those from the control group, which will continue to receive the usual low-probability inspections as mandated by the country’s laws.

The first treatment arm will receive high-intensity inspections, with warnings and sanctions for non-compliant facilities. A health facility will be inspected with 100% probability more than once, and the inspection will be conducted in an electronic format by a group of trained inspectors working full time, and based locally. The inspections will have specific and standardized warnings and sanctions imposed on facilities according to their overall JHIC score.

The second treatment arm will receive high-intensity inspections with public disclosure of patient safety performance and aims to empower patients to make informed choices of health facilities based on their patient safety record. At the end of the inspection, each facility will be assigned a letter-grade -- provisionally A through D -- that will be prominently posted on the health facility door. Fraud control mechanisms will be put in place to ensure that clinics do not alter the information on the clinic door. The inspection results will be reported to the relevant boards and councils and they are able to apply warnings and sanctions to the clinics. There will be a dissemination campaign for patients to introduce them to the warnings and sanctions. At the same time, clinics will receive a detailed checklist and they can ask for a re-inspection if they want to change their grade.

All public and private health facilities in Kakamega, Kilifi, and Meru counties will be randomly assigned to one of the three groups.

The evaluation will specifically measure quality, patient safety, demand for services and prices for facilities across the three groups. This will include (1) Adherence to a checklist developed by the regulators and includes indicators of quality and patient safety related to protocols, infrastructure, and equipment; (2) Adherence to patient safety practices related to infection prevention and control; (3) Adherence to case-specific checklists of essential and recommended care for four medical cases, and (4) Prevalence of substandard medicines. Demand for health services will be measured through patient flow records from health facilities and administrative data, and prices will be measured through patient exit surveys. Measurement will include direct observations in the form of surveyors dressed as patients who are recording patient safety – for example, did the doctor wash his hands before seeing the patient -- and the accuracy of the doctors’ diagnosis and treatment given.

Results from varying facilities in “markets” will also be used to determine whether increased competition between facilities in the same geographic area affects patient safety standards. The evaluation team will also collect data on sub-groups both by facilities and patients, including gender.

Policy impact

The results will help the Kenyan Ministry of Health craft more effective health care policies that increase both accountability and competition to boost patient safety throughout the country. The evidence will also be useful to policy makers in other low income countries who are looking for innovative ways to provide higher quality health care to their citizens. Results will help health care practitioners develop a set of tools and instruments that can be broadly used in surveys and inspections across diverse low-income settings.