Tracking the Pandemic's Impact on Families from a Distance

How much money have you spent in the past seven days? Have you skipped any meals in the past week? Is your child crying or talking more than usual? 

How inclined would you be to answer these questions from a person you had never met before who knocked on your door? What if they came from someone calling your phone?

Getting objective answers — and checking to make sure they reflect the reality — is one of the biggest challenges faced by surveyors who gather the data that the World Bank uses to assess, for example, changes in poverty levels. The challenge got harder in early 2020 with the onset of COVID.

Lockdowns and social-distancing requirements made in-person surveys impossible in most places, triggering a fast-paced, unparalleled shift in how data is collected around the world. The surveys were typically done face-to-face by hundreds of surveyors — or enumerators, as these professionals are technically called – hired by World Bank partners across the world. They had to quickly migrate to phone calls, much more impersonal – and, well, suspicious to many. 

“The transition was not small,” recalls Kevin Isiche, an enumerator in Busia, a small town in western Kenya. “The first challenge has to do with trust. In person, you introduce yourself, you have a badge. On the phone, some people think you’re a con. It’s much harder to build a rapport.” 

The difficulties imposed by COVID-19 on tracking welfare were not limited to the field. Those ensuring that the methodology was sound and analyzing the survey results to draw a picture of the pandemic’s impact on households faced unprecedented challenges as well. 

The World Bank had been conducting phone surveys for years, but the method was limited to special situations, when mobility or safety was an issue. They had been deployed in places affected by conflict or by epidemics such as the Ebola in West Africa, for example. COVID-19 gave researchers an opportunity to roll out phone surveys on a big scale — helping them understand their limitations and come up with ways to address these shortcomings.

“COVID pushed us to use phone surveys because it was the only way of getting socio-economic data from households,” says World Bank Senior Economist Utz Pape, who’s been tracking progress and setbacks in poverty reduction efforts for eight years. Many national statistical offices and survey practitioners that worked in partnership with the World Bank had limited experience in conducting phone interviews, and implementation guidelines were lacking.

Since the start of the pandemic, the World Bank has done phone surveys in more than 83 countries, compiling them in a dashboard that lets users see results across 16 topics and 155 indicators. 

Pape says that one of the main challenges was related to representing the population accurately. In face-to-face surveys, households are randomly selected — but, of course, not everyone has a phone. And when people do, what if they don’t answer the call? What if the phone is out of batteries? And who guarantees that the person answering is whom they claim to be?

Substantial changes had to be made to questionnaires, which for household surveys would focus on families’ spending and could take more than three hours, often lasting several days. Now surveys could not take more than 20 or 30 minutes, to minimize the risk of people just hanging up.

The work required some statistical techniques — a methodology called imputation that was around before COVID-19 but hadn’t been deployed yet over the phone, explains Pape. “We've refined some methodologies so that they can be applied in the context of phone surveys to estimate impacts of shocks like COVID-19 on livelihoods.” 

In the field, research companies had to quickly remap surveyors and help them acquire new technical skills to ensure the work would yield high-quality data. Quality checks had to be beefed up, to ensure that surveyors, now working from their homes, were doing the work effectively. Some interviews are now recorded, and team managers can listen in.

Adapting methodologies to a new reality imposed by COVID is crucial because tracking welfare has never been so important. The pandemic has disrupted economic activity in virtually every country, but impacts have varied within economies and between countries. Understanding which groups have been more affected and what kind of assistance has worked can help inform policy responses to support recovery and build resilience.

The successful rollout of phone surveys on a global basis means that the World Bank will probably keep using this kind of survey as a complement to the more accurate, in-depth household surveys, even when surveyors are able to visit households again, says Pape. This could lead to shorter intervals between the release of poverty estimates, currently at around two to three years.

“I think a lot of colleagues now feel much more comfortable with saying something statistically about a population, even though the data come from a phone survey, and that's a big development in itself,” says Pape.

The feeling of contributing to a positive outcome at a moment of such distress is shared by Isiche. He’s been doing surveys for nearly three years and hopes to continue working in the area. 

“Maybe through this work I'll be able to get in touch with people and at least have one positive impact on society on a broader perspective,” he says from his backyard in Busia, noting that after seven rounds of phone surveys since April 2020 — several of them to the same people — some respondents already recognize his voice when they pick up the phone.