Statistical validation of a large-scale web survey during the COVID-19 pandemic in India

Abhiyan Mothram¹, Soumya Roy Chowdhury², Santanu Pramanik³

1. LEAD at Krea University, New Delhi, India
2. IIT Gandhinagar and IIIT Vadodara, Gujarat, India
3. LEAD at Krea University, New Delhi, India

OPEN  ACCESS

PUBLISHED: 31 December 2024

CITATION: MOTHERAM, Abhinav; CHOWDHURY, Soumi Roy; PRAMANIK, Santanu. Statistical validation of a large-scale web survey during the COVID-19 pandemic in India. Medical Research Archives,Available at: <https://esmed.org/MRA/mra/article/view/6223>. 

COPYRIGHT: © 2025 European Society of Medicine. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

 ISSN 2375-1924

Abstract

There was an overwhelming demand for data to respond to the economic and health emergencies during the COVID-19 pandemic. This led to the rapid expansion of data collection such as mobile and web surveys to the forefront, which was not the case before in many low and middle-income countries. In this case report, we present the statistical validation of a large-scale web survey conducted in India to understand the COVID-19 trends and impact. The survey was designed to provide valuable information to help monitor and forecast how COVID-19 may be spreading.

Keywords

COVID-19, web survey, data validation, India, health emergency

Introduction

Face-to-face surveys as a method of data collection have been restricted due to the pandemic. Furthermore, fear of contracting infection and non-pharmaceutical interventions such as physical distancing and mobility restrictions to contain the spread of infection made it infeasible to continue data collection with this mode during the pandemic. At the same time, there was an overwhelming demand for data to respond to economic and health emergencies, this drove the remote modes of data collection such as mobile and web surveys to come to the forefront.

Facebook surveys are available for people aged 18 years and above in many countries. CTIS is the largest survey of its kind in India, and it has been designed to provide timely information on the COVID-19 pandemic. The survey was designed to collect data on COVID-19-related symptoms, experience with COVID-19 tests, contacts with others, mental health and economic security, disruptions in routine health services, vaccination status, vaccine hesitancy, and other related topics.

Methods

The COVID-19 TRENDS and IMPACT SURVEY: A unique data source in challenging times. In the COVID-19 Trends and Impact Survey (CTIS), over 1,000 respondents randomly selected from the sampling frame, FB users around the world were invited to self-report COVID-19-related symptoms, experience with COVID-19 tests, contacts with others, mental health and economic security, disruptions in routine health services, vaccination status, vaccine hesitancy, and other related topics. The CTIS data presents a unique opportunity to explore the spatial-temporal variation in COVID-19 related symptoms and experiences.

COVID-19 Illness

Since experiencing symptoms is often a precursor to getting more seriously ill or going to the hospital, the original rationale of CTIS was to help produce a weekly forecast of the hospitalization rates, as well as an early indicator of where the outbreak is growing and where the curve is being successfully flattened. Several studies on symptom tracking and forecasting COVID-19 cases have emerged using CTIS data.

At the state level, we observe a similar convergence of estimates over time between CTIS and official estimates as reported in the media. Here vaccine uptake is measured as percentage of adult population receiving at least one dose of vaccination. Due to the missing data on some days, to address that concern, we plot the 7-day moving average of CTIS trends versus the official state level vaccination for the select large states (determined by population sizes).

Results

Results

BIAS IN VACCINE UPTAKE ESTIMATES

Figure 1 presents the CTIS and official estimates of at least one dose and two doses of vaccination at the national level. The official number of doses are scaled to the eligible target group of people 18 years or above using the Census projections for the year 2021 published by the Government of India. The scaling factor, as plotted in the right panel, is defined as the ratio of CTIS vaccine uptake estimates to vaccine uptake in administrative data scaled to the adult population. Similar to Bradley et al. (2021), we observe the vaccine uptake as per CTIS to be higher than in the administrative data.

While the daily estimates during the initial months of vaccination roll out diverge greatly, the divergence between CTIS and official estimates decreases over time as more adults in the general population are vaccinated. The scaling factors of excess vaccination uptake in CTIS for at least one dose and two doses in March 2021 stood at 8 and 120 respectively. By April–May 2021, the excess vaccine uptake fell to 2 and 15 and continued the downward trend until December 2021.

In Figure 3, we present the state-level estimates of respondents who have reported anosmia as a symptom in the selected states by place of residence. Similar to CLI trends, we see a major divergence in the trends of anosmia showing northern and southern states.

VALIDATION OF COVID-LIKE ILLNESS ESTIMATES

In Figure 4, we present the comparison of publicly available CLI estimates for India with official COVID-19 cases as collated based on diagnostic tests during the time period 23 April 2020 – 31 December 2021. The vertical dotted line denotes the emergence of wave-2 in India also known as Delta wave. Bottom left corner panel represents CTIS-CLI estimates. Other panels represent official data of cases, deaths and testing. During the second wave in India amidst the emergence of delta variant between April-June 2021, both cases and deaths peaked sequentially and then plateaued to a relatively lower level, as per the official data. We see that the 7-day moving average of percentage of respondents with CLI trend mirrors the trend in official cases between April-June 2021. However, while the official number of cases and deaths have collapsed post wave-2 in June 2021, the percentage of respondents with CLI has not. It kept increasing past the official peak, ultimately reaching a peak of nearly 6% in November 2021. We also note that while testing has decreased post wave-2, the decrease is relatively small and perhaps not enough to entirely explain this divergence.

In Figure 5, we present the state level CLI trends by northern and southern states. For the ease of presentation, we include Gujarat and Maharashtra as northern and southern states, respectively, because of their relative geographic position, although they lie in western India strictly speaking. We observe that while the 7-day moving average CLI trends for southern states remain steady at relatively low levels, CLI trends in the northern states keep increasing post wave-2 which contributes majorly to the overall national CLI. There are two questions at play here – 1) Why do we see a persistence of trend in the CLI estimates at national level when daily COVID-19 cases have fallen? 2) Why do the northern and southern states exhibit different trends in CLI post wave-2?

One potential hypothesis is the confounding of symptoms between COVID-19 and seasonal flu. Although the symptoms of COVID-19 are similar to those of the seasonal flu, including fever, cough, sore throat, chest pains, and fatigue, the addition of anosmia and dysgeusia, the loss of smell and taste have been shown to be an important indicator of whether an individual has been affected with COVID-19. Whether the respondents have had anosmia in 24 hours prior to the interview is one of the symptoms questions asked in the CTIS. In Figure 6, we plot the 7-day moving average of the percentage of respondents who have reported anosmia as a symptom in the selected states and by place of residence. Similar to CLI trends, we see a major divergence in the trends of anosmia by northern and southern states. Decomposition of trend in reported anosmia by place of residence also shows that the increase in reporting post wave-2 has occurred mostly in the village or rural areas.

Discussion

Early in the COVID-19 pandemic, there was a scarcity of timely information on regional increases in SARS-CoV-2 infections, people’s knowledge, attitudes and practices of COVID appropriate behaviours and the impact of the pandemic on people’s lives. The routine data collection efforts that existed at that time were too slow to meet the data demands for understanding and managing the pandemic. In India, there were some nimble initiatives at the regional level to understand the social and economic effects of the pandemic, but those were not enough to compare the situation over space and time. In that context, alternative data sources like CTIS over a period that is long enough to cover the seemingly never-ending pandemic and across all states of India is a useful resource for research and policy making around the management of the pandemic.

However, statistical validation of the representativeness of the sample or the lack of it and its impact on survey estimates is crucial and our paper contributes to that much needed evidence base.

Representativeness of the sample

Table 1 presents the unweighted and weighted sample characteristics by interview completion status for a randomly selected daily sample from December 1, 2021. We also present the corresponding estimates from Census 2021 population projection report as an external benchmark.

Table 1 presents the unweighted and weighted sample characteristics by interview completion status for a randomly selected daily sample from December 1, 2021. We also present the corresponding estimates from Indian Census’ 2021 population projection report as an external benchmark. We see that the two-step weighting procedure partially corrects the bias in gender and age groups (to a lesser extent) represented in the CTIS samples. For the age group 25–34 years, CTIS-weighted estimates of percentage of Indian adults belonging to this age group are much higher (35%) than the external benchmark (24.4%). This suggests the abundance of FB users in the age group of 25–34 years. When we compare the distribution of adults across four broad age categories which were considered for the poststratification adjustment (18–24, 25–44, 45–64, 65+), the matching of distribution is much better. For example, for the age group 25–44 years, based on the December 1, 2021 sample, the CTIS-weighted and external benchmark estimates are 51% and 44.4%, respectively.

Although the original objective of CTIS was to provide valuable information to help monitor and forecast how COVID-19 may be spreading at the early stage of the pandemic, the later versions of CTIS, starting 21 December 2020, include questions on vaccination. Availability of gold standard data on COVID-19 vaccination coverage allows us to measure the bias in CTIS vaccine uptake estimates. Results indicate that the vaccination estimates from CTIS are inflated compared to the official numbers. There are many potential reasons behind this systematic overestimation. First, the CTIS-India sample is overrepresented by younger, more educated, internet savvy, urban respondents even after correction for survey nonresponse and frame imperfection. Not all relevant demographic characteristics were used as poststratifying variables in the second stage of weighting adjustment. For example, the area of residence (rural or urban), a key determinant of being a FB user in India, is not used in the poststratification adjustment of weights. Majority of the respondents are from cities whereas the proportion of rural population is much higher in India. No attempt has been made to account for this mismatch between sample and population distribution with respect to area of residence; this is likely to skew the estimates towards urban population. The choice of poststratification variables and their levels play an important role in forming a representative sample and producing near-unbiased estimates for outcome of interest as long as poststratification variables are predictive of the outcomes.

Secondly, inclusion of adult respondents in the CTIS sample, satisfying both the criteria of being a FB user and willing to take a web survey, is highly correlated with vaccination status of individuals. The COVID-19 vaccination roll-out was initially restricted to self-registration on the Co-WIN web portal or mobile application. This suggests that the respondents of CTIS are more likely to be vaccinated as it was much easier for them to book appointments for vaccination in the Co-WIN app. The process of exclusion of target population from the sampling frame and the nonresponse mechanism being correlated with the outcome of interest is known as informative sampling and nonignorable nonresponse, respectively, in the survey methodology literature and they are known to produce biased estimates.

The impact of digital divide on vaccination status started to diminish towards the later part of 2021 through introduction of more walk-in vaccination centers, near-to-home temporary vaccination centers in non-health facility based settings, on-site registration, vaccination at government and private workplaces. The convergence of CTIS and official estimates towards the later part of 2021 is an artefact of the above-mentioned initiatives by the Ministry of Health and Family Welfare, Government of India which led to similar average vaccine uptake in CTIS sample and general adult population. Some of the initiatives might have been prioritized after the Supreme Court ruling on 2 June 2021 which criticized the vaccination policy for relying.  exclusively on a digital portal for vaccinating the adult population and warned that the existing policy might fail to achieve universal immunization owing to a digital divide in the country’s infrastructure.

While assessing the bias in CTIS estimates of vaccine uptake, the official data can be considered as “gold standard” due to the mandate of self or on-site registration on Co-WIN app for COVID-19 vaccination. However, in our second example, where the symptom trends observed in the CTIS diverge from the official estimates, the quality of “gold standard” data is questionable. It is unclear at this moment as to why the percentage of respondents with CLI and anosmia persists at a high level post wave-2. We do not rule out the possibility that it could simply be an artefact of the survey. However, there have been concerns around underestimation of COVID cases and deaths in the official data for various reasons. The Indian Council of Medical Research, the apex body in India for the formulation, coordination and promotion of biomedical research had changed the COVID-19 testing strategy during and post wave-2 to optimize the use of RT-PCR testing and recommended increased use of rapid antigen tests (RATs) particularly in the rural areas where testing facilities are rather meagre. This might have had implications on the test positivity rates based on several news reports and initial research.

Anecdotal evidence suggests people’s unwillingness to get the COVID test done, even in the presence of symptoms, because of fear of institutional quarantine. There is also ongoing research on the persistence of symptoms in long COVID patients, role of absolute humidity and seasonality in the COVID-19 dynamics. In summary, unlike in the case of vaccination, the official COVID-19 diagnostic numbers cannot be substituted for a gold standard. However, they are used widely to track the progress of the pandemic despite the shortcomings. For instance, test positivity rate is a widely used metric which suffers from selection bias in testing and test positivity rates has been documented.

Conclusions

In response to Bradley et al. (2021), Professor Frauke Kreuter who co-led the Global CTIS writes:
“The (survey) quality is very difficult to assess, because there is usually no independently verified ‘ground truth’ or ‘gold standard’ with which to compare survey data”. This rings even truer in the context of India and other developing economies where the gold standard estimates are deficient — collecting the required information in some cases, defunct in others or not at all present in some others. Going forwards, it is clear that any research in public health and social sciences has to make use of data coming in all shapes and forms. For surveys with biased samples, novel techniques such as Multilevel Regression and Poststratification (also known as Mister P or MRP) have shown promise in correcting the survey biases. As Professor Kreuter writes, “For certain inferential tasks, surveys with deficiencies can be useful. The usefulness of a data set can be evaluated only in the context of a specific research question.” We agree with one caveat that the survey methods and data including metadata and survey process data (paradata) be made available transparently. This means embracing more openness in all forms — accessibility, availability and less transaction costs in general to acquire the needed information.

Conflict of Interest:

The authors declare no competing interests.

Funding Statement:

This work was supported by Bill and Melinda Gates Foundation [Grant Number INV-003352, INV-009903, INV-010337] through the NCAER National Data Innovation Centre research grant.

Acknowledgements:

We thank Facebook Global Research team members; Katherine Morris, Kris Barkume, Sarah LaRocca, Kelsey Mulcahy for productive conversations about the Global CTIS survey, weighting method and the bias in vaccine uptake.

estimates. We are grateful to Jaya Koti, Project Analyst, NCAER National Data Innovation Centre for calculating external benchmark estimates based on population projection data.

Data Availability Statement

Data used in the analysis come from various sources. CTIS aggregate estimates at state and national levels are obtained from UMD’s Global CTIS Open Data API. CTIS Microdata is made available through UMD’s Global CTIS Microdata Repository to researchers upon request. Separately, an R interface to access UMD’s Open Data and Microdata APIs is made available as an R package (https://github.com/am0505/ctisglobal).

While CTIS open data estimates are made publicly available for other countries, we restrict our analysis to India. Administrative data for states and national level in India on COVID-19 is obtained from COVID19Bharat (https://covid19bharat.org/).

Data on state and national level population by subgroups is obtained from Population Projections Report by Census of India 2011 (https://nhm.gov.in/New_Updates_2018/Report_Population_Projection_2019.pdf). We restrict the time period of our study to December 2021 as initial reports suggest that the reported symptoms of infections from Omicron variant are perhaps different to that of the other coronavirus variants.

Author contributions

AM, SRC, and SP conceived and formulated the research questions. AM performed CTIS India data analysis and visualization. AM and SP contributed equally to the writing of the paper. SRC reviewed the manuscript, provided critical comments and revised the manuscript. SP supervised the work.

References