Impact Evaluation of a Community Engagement Intervention for Improving Childhood Immunization Coverage in Rural Assam, India: A Cluster Randomized Controlled Trial

Arpita Ghosh 1,2,3, Santanu Pramanik 4,5, Anirban Goswami 5, Samina Parveen 6, Jahnabi Hazarika 7*, Tiken Das 8*, Laurent Billot 2,9, Sandra Albert 10**

1.  The George Institute for Global Health, Delhi, India;
2. University of New South Wales, Sydney, Australia;
3. Prasanna School of Public Health, Manipal Academy of Higher Education, Karnataka, India;
4. Centre for Effective Governance of Indian States, Delhi, India;
5. Assam Project on Forest and Biodiversity Conservation Society, Guwahati, India;  Ipas Development Foundation, Delhi, India;
6. Independent consultant;
7. Nowgong College, Nagaon, Assam, India;
8. The George Institute for Global Health, Sydney, Australia
9. Indian Institute of Public Health, Shillong,India

 Authors were with the Public Health Foundation of India during the study

• Authors were with the Indian Institute of
  Public Health, Shillong during the study

 OPEN ACCESS

PUBLISHED: 31 August 2025

CITATION: Ghosh, A., Pramanik, S., et al., 2025. Impact Evaluation of a Community Engagement Intervention for Improving Childhood Immunization Coverage in Rural Assam, India: A Cluster Randomized Controlled Trial. Medical Research Archives, [online] 13(8). https://doi.org/10.18103/mra.v13i8.6895

COPYRIGHT: © 2025 European Society of Medicine. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

DOI https://doi.org/10.18103/mra.v13i8.6895

ISSN 2375-1924

 

Abstract

Background: For childhood immunization, community engagement interventions can potentially address demand-side barriers to achieving immunization coverage targets, while also mobilizing the community to advocate for better service delivery. However, high-quality evidence that can causally relate such interventions to increased immunization coverage is scarce. We evaluated the impact of a community engagement intervention on childhood immunization coverage in rural India.

Methods and findings: The community engagement intervention, referred to by the acronym SALT (Stimulate, Appreciate, Learn, Transfer), is a complex intervention involving an iterative process of multiple steps and continued engagement with the community. To evaluate the SALT intervention, we conducted a cluster randomized controlled trial in 196 villages spread across 3 districts in rural Assam, a state in the northeast region of India. The villages in the intervention group (n=97) received the SALT intervention for about one year along with routine immunization services and the villages in the control group (n=99) received only routine immunization services. The primary evaluation outcomes were full immunization, defined as one dose of Bacillus Calmette–Guérin vaccine, three doses of oral polio vaccine, three doses of diphtheria-pertussis-tetanus (DPT) vaccine or Pentavalent vaccine and one dose of measles vaccine, in children 12-23 months old, and three doses of DPT or Pentavalent vaccine in 6-23 months old children. To gather information on evaluation outcomes, we conducted cross-sectional household surveys at baseline and after 22 months at endline, in the 196 villages. We interviewed mothers with a 6-23 months old child in a random sample of households in each village. In our analyses, we followed the intention-to-treat principle and used mixed-effects models to account for clustering. At endline, a total of 2,907 mothers were interviewed resulting in a median number of 15 (interquartile range (IQR)= 0) 6-23 months old children and 10 (IQR=2) 12-23 months old children from each village in both the intervention and the control groups. There was no difference between the two groups in the proportion of 12-23 months old children receiving full immunization (OR=0.98, 95% CI: 0.71 – 1.36) or in the proportion of children receiving three or more doses of DPT or Pentavalent among 6-23 months old children (OR=1.01, 95% CI: 0.76 – 1.35).

Conclusions: The intervention was not found to be effective in increasing childhood immunization coverage in our study. After baseline survey, we noted that the vaccination coverage in the three districts was substantially higher than previously reported in national surveys which were used in designing the trial. The higher coverage rates were most likely due to widespread implementation of a supplementary immunization programme led by the Government of India prior to this study. We do not know whether selecting study sites with a lower vaccination coverage rate at baseline or having a more targeted approach in implementing the intervention would have resulted in a positive impact.

Trial registration: The trial was registered on 7th February, 2017 under the Clinical Trials Registry- India (CTRI), hosted at the Indian Council of Medical Research’s National Institute of Medical Statistics, with registration number CTRI/2017/02/007792

Keywords

• Childhood immunization
• Community engagement
• Cluster randomized controlled trial
• Assam, India

Introduction

Vaccination is a globally accepted public health intervention that helps avert vaccine-preventable diseases. Incomplete vaccination and non-vaccination increase the risk of illness and death among children. The Global Vaccine Action Plan (GVAP 2011-2020) sets the goal of 90% coverage at the national level and 80% in every district or equivalent administrative unit with all vaccines in national programs by 2020. India is one of the 194 Member States of the World Health Assembly which endorsed the GVAP framework in 2012. Despite a long standing national program for immunization in India since 1985, only 62% of 12-23 months old children are fully immunized, as estimated in the fourth round of National Family Health Survey (NFHS-4) conducted in 2015-16. Full immunization is defined as children receiving one dose of Bacillus Calmette–Guérin (BCG) vaccine to prevent tuberculosis, three doses of oral polio (OPV) vaccine, three doses of diphtheria-pertussis-tetanus (DPT) vaccine or the more recently introduced Pentavalent (DPT-Hepatitis B- Haemophilus Influenzae type B) vaccine, and one dose of measles vaccine. Completion of schedule of vaccines that require multiple doses (for example, OPV and DPT or Pentavalent) remains a major challenge towards achieving higher full immunization coverage. Both demand- and supply-side bottlenecks contribute to sub-optimal vaccination coverage rates in India.

In the context of universal immunization program (UIP) in India, most of the existing interventions are geared towards addressing supply-side challenges such as ensuring better immunization services and more focused implementation by deploying more health workers across health facilities, introducing alternate vaccine delivery system, including new vaccines in the immunization schedule, organizing sessions in hard-to-reach areas, and initiating supplementary immunization activities for children who are missed out in the routine immunization program. But there is a growing body of literature showing that demand-side interventions lead to significant improvement in childhood vaccination coverage in low- and middle-income countries. Community engagement approaches can address demand-side barriers while also mobilizing the community to advocate for better service delivery.

With the growing realization that community-level factors influence vaccination uptake, more recent strategies to increase vaccination coverage have attempted to focus on community-based interventions. Most of the existing community engagement programs, however, focus on communication activities that do not actively involve communities in the planning, monitoring and surveillance activities. Participatory engagement of communities can help identify barriers to vaccination at the local level and thus might lead to sustainable solutions in a manner that a top-down approach cannot achieve. The SALT (Stimulate, Appreciate, Learn, Transfer) intervention is a community-based intervention which aims to develop and strengthen a sense of community ‘ownership’. This community-based intervention has been shown to be effective in generating behaviour change in combating HIV/AIDS in Papua New Guinea and cost-effective when comparing incremental cost-effectiveness ratio with other HIV prevention programs in Thailand. A retrospective study of SALT versus non-SALT districts in Togo showed significant increase in impregnated bednet use and decrease in malaria prevalence among children under five. A recent study (unpublished) in Democratic Republic of the Congo found that SALT intervention can reduce vaccine hesitancy and increase vaccination rates for oral polio vaccine. However, the research design of the study lacks the methodological rigor needed to measure attributable impact. Studies using experimental designs to evaluate the effectiveness of community engagement approaches to increasing immunization coverage are few.

In order to identify the impact of the SALT approach in increasing immunization coverage, we conducted a cluster randomized trial in 196 villages across 3 districts in Assam, a state in the northeast region of India. Our study assessed the intervention’s impact on two primary outcomes– full immunization and three doses of DPT or Pentavalent in children. We also explored whether the intervention has an impact on a few secondary outcomes – dropout between doses 1 and 3 for DPT or Pentavalent vaccine, availability of vaccination card, mother’s exposure to immunization messages, household’s engagement with village communities and mother’s belief in community’s role in and ability to have impact on her child’s health.

Methods

We adopted a cluster randomized controlled trial design with two groups to evaluate the impact of the intervention, where the villages are the clusters and 6-23 months old children are the study participants. Villages in the intervention group received the SALT intervention for about a year (March 2017-March 2018) along with routine immunization services. Villages in the control group received routine immunization services alone. The evaluation used a repeated cross-sectional design where we tracked the same sampled villages but drew independent random samples of eligible children to measure immunization outcomes at baseline and after 22 months at endline.

The full study protocol has been published. The IRB approval (TRC-IEC- 285/16) for the study was received from the Institutional Ethics Committee of the grant holding institute in April 2016. Written informed consent was obtained from the participants. However, informed consent to randomize was not sought as the unit of randomization (village) and the unit of observation (child) are different. A letter of support was obtained from the Directorate of Health Services, Assam, to facilitate the implementation of the study.

Sampling design: recruitment of clusters and participants

We considered 3 districts from Assam – Bongaigaon, Kamrup Rural, and Udalguri, selected using stratified purposeful sampling so that they represent varied sociodemographic characteristics of Assam. Within each district, we used a two-stage stratified cluster sampling design. In the first stage, we selected 80 villages from each of the three districts using a stratified sampling technique. We stratified eligible villages, defined as having 50-500 households, within a district into four strata and then randomly selected 20 villages from each stratum. The study protocol provides details on how the stratified sampling was carried out. In the second stage, from each of the 240 sampled villages (80 in each district), after identifying the eligible households having a child 6-23 months old, a random sample of 15 households was selected for the baseline survey. In a selected household, all mothers having children in the age group 6-23 months were eligible to participate in the survey. From mothers, we collected information pertaining to her youngest 6-23 months old child.

In the endline survey, the sampled villages remained the same. A fresh round of house-listing exercise was carried out in these villages at endline to identify the eligible households with a 6-23 months old child. Given the restrictive age range of 6-23 months, eligible households at endline were for the large majority, different from the ones at baseline. A random sample of 15 households was drawn from each village to be interviewed during the endline survey and mothers and children were selected for interview following the same procedure as in baseline.

The baseline survey was conducted during June-August 2016. The endline survey was conducted after the conclusion of the intervention, during July-September 2018.

Intervention

The intervention is described in detail in the study protocol paper. The key steps and underlying principles are described here. A pictorial illustration is presented in the appendix (Figure S2.1). The foundation of the intervention is that ‘Communities change themselves. We do not change communities’. The intervention uses the Community Life Competence Process (CLCP), a form of learning cycle where a community identifies a problem and solutions thereof, takes action and learns from the process. The underlying principle of the intervention is that when a community takes ownership of the challenge it faces, it is on the road to sustainable change. CLCP is facilitated in communities by trained facilitators who use an attitudinal approach referred to by the acronym SALT – Simulate, Appreciate, Learn and Transfer. The intervention is a combination of CLCP and SALT, but for simplicity we have used just one acronym, SALT, to refer to the intervention.

The intervention starts with home visits by trained facilitators to build rapport with the community and to identify the strengths of the community. The next step is collective dream building which starts from individuals and small groups and then involves the wider community. The idea is to identify and prioritize a set of achievable ‘dreams’ for the community.

Once the community agrees on a dream, a self-assessment exercise starts under the guidance of the facilitator. Using a scale of 1 to 5 they assess where the community stands with respect to practices linked to their shared dream. The facilitators stimulate conversations so that actionable points emerge. With respect to the dream of healthy children, immunization related practices were discussed during self-assessment. The underlying principle being that members of the community assess themselves, rather than the facilitator (an outsider) assessing them.

The community then lists practices, relevant to their shared dream, where it feels that it can make progress within a stated timeframe (say, 2-3 months). The discussion during the community meetings evolves around what actions need to be taken (action plan) in order to reach a next desired level from the current level agreed to by the community during self-assessment. This is followed by an action phase and a review process to further assess their level of achievement. Subsequently (end of a year) the facilitators bring the communities together to share with and learn from each other in a knowledge fair when transfer of knowledge and experience takes place between communities. Learning, self-assessment, planning and action ideally repeats itself in a continuous but evolving cycle. Each time the same or new dreams and priorities may be identified and actions thereof.

The intervention was implemented in the three districts of Assam by two local non-governmental organizations (NGOs), namely Voluntary Health Association of Assam (VHAA) and the Centre for North East Studies and Policy Research (CNES) under the supervision of the lead NGO, the Constellation. VHAA was responsible for two districts (Kamrup Rural and Udalguri) and CNES was in charge of Bongaigaon district. For each district, the local NGOs hired three SALT facilitators leading to a total of nine facilitators who were the key implementers of the intervention. Each district had a district coordinator who was responsible for overall monitoring and supervision of facilitator’s work and intervention activities. Regular training and support were provided and site visits made by the national and international coaches of the Constellation.

Randomization

The village was our unit of randomization. All 240 villages sampled in the baseline survey were randomized to the intervention and control groups after the baseline survey using a 1:1 ratio. Randomization of villages was stratified – villages within a district were stratified into four strata based on a composite score constructed using village-level data from baseline survey. Details of the composite score are provided in the appendix.

Sample size

The sample size calculation at the design stage was based on estimates from relatively old national surveys available at the time and was subsequently revised using baseline survey data. The initial sample size calculation suggested a requirement of 120 intervention and 120 control villages to detect a difference of at least 10 percentage points in immunization coverage between the two groups with 80% statistical power using a two-sided test at 5% level of significance, after accounting for an intracluster correlation coefficient (ICC) of 0.21 and 0.25 for three doses of DPT or Pentavalent and full immunization, respectively. The expected coverage was 55.3% for full immunization and 65.9% for three doses of DPT or Pentavalent in the control group. We revisited the calculation after completion of baseline survey. The revised sample size calculations, based on updated estimates of coverage and ICC derived from the baseline survey, required 90 intervention and 90 control villages to detect an increase of at least 8 percentage points from an expected coverage of 84% and 79% for three doses of DPT or Pentavalent and full immunization, respectively, in the control group, with 80% statistical power based on a two-sided test having 5% level of significance, after accounting for an ICC of 0.17 for three doses of DPT or Pentavalent and 0.18 for full immunization. Details of the initial sample size calculation and subsequent revisions are provided in the appendix.

We therefore needed 30 intervention and 30 control villages in each district. Instead of randomly excluding villages, we dropped villages that would help ensure a geographical buffer of at least 3 kilometres between the intervention and control villages, to reduce the chance of contamination. The procedure followed for identifying villages to be dropped is described in detail in the study protocol in the subsection Minimizing Intervention-Control Contamination.

Study outcomes

We considered two primary immunization outcomes, three secondary immunization outcomes and two secondary outcomes related to community processes. The two primary outcomes are full immunization in 12-23 months old children and receipt of three doses of DPT or Pentavalent in 6-23 months old children. Full immunization is defined as the 12-23 months old child receiving one dose of BCG vaccine, three doses of OPV, three doses of DPT or Pentavalent vaccine, and one dose of measles vaccine. We defined the 6-23 months old child as having received all three doses of DPT, if the child received three or more doses of either DPT or Pentavalent vaccine. We will hereafter use DPT to indicate vaccination with either DPT or Pentavalent. We combined information from the vaccination card and mother’s recall to define the vaccination status for each vaccine, following the procedure outlined in the appendix.

The three secondary immunization outcomes were – dropout between doses 1 and 3 for DPT vaccine, availability of vaccination card, and mother’s exposure to immunization messages. Dropouts between doses 1 and 3 were defined as children who had failed to receive all three doses but had received at least one dose. The vaccination card was defined as available for the child if it was seen by the interviewer. The card could be in possession of the mother or another member of the household, as in most cases, or in some villages with the community health worker of that village for safekeeping. In the latter case, the interviewer requested to see the card from the community health worker. If the mother had heard, seen or read any immunization messages in the last 6 months then she was considered as being exposed to immunization messages.

We considered two secondary outcome variables indicating community processes – household’s engagement with village communities and mother’s perception about role of community in improving children’s health. If the household reported that their community met occasionally or regularly and that they attended these meetings and they engaged in community actions to collectively tackle issues, then the household was considered as engaging with the village communities. The mother was asked if she believed that community had a role to play in improving the health of her child and if the community was capable of taking actions to prevent her child from getting sick. If the mother replied ‘yes’ to both these questions, then we considered her to believe in community’s role in and ability to have impact on her child’s health. We describe these secondary outcomes in detail in the appendix.

Process evaluation

In order to measure the extent of community’s exposure to the intervention we collected process indicators throughout the intervention phase. Specifically, the aim was to gain understanding about the intervention fidelity around the following aspects: 1) whether the intervention was implemented as intended, 2) whether the intervention incorporated the primary objective of the study (that is, increasing immunization coverage), 3) consistency of intervention delivery across communities in terms of the process of administering the intervention, 4) the reach and coverage of the intervention across villages and districts, and 5) whether contextual factors influenced the implementation of intervention.

The components of the process evaluation were developed based on the framework of Grant et al. Data on the indicators relevant for each step of the intervention were collected through a combination of methods, including monthly reporting format developed by the evaluation team and duly filled in by the SALT facilitators, direct observation of different steps of the intervention by the evaluation team, and in-depth interviews of SALT facilitators and district coordinators by the evaluation team. The team also interacted informally with community members during the dream building, self-assessment, and action planning sessions in the community.

Monthly reporting format

The purpose of developing the monthly reporting format was to collect routine data on the intervention implementation at the community (village) level. We kept the format simple to enable the facilitators to fill it quickly and easily on a monthly basis. The indicators were identified so as to cover the entire intervention process and also to gain understanding of the intervention fidelity. Based on the monthly data, the following village-level indicators were consolidated by the evaluation team for the entire duration of the intervention: total number of visits made to the village by the SALT facilitator, total number of households in the village where home visits were conducted, total number of community meetings, total number of meetings involving specific groups (e.g. community block officers, self-help groups, health workers, religious leaders), timing (number of months after the start of intervention) of the first community dream building in the village, whether immunization emerged as a topic during dream building, timing of the first self-assessment exercise and whether it was repeated, whether practices around immunization were discussed during self-assessment, timing of the action phase, and number of visits made to the village specifically for follow up of action plan. All nine SALT facilitators used to fill in the monthly reporting format and share with the evaluation team after the end of each month.

Direct observation and in-depth interviews

The evaluation team developed checklists for on-site process monitoring of home visits, dream building, self-assessment and action plan. The checklists included indicators to understand whether the facilitators followed the concept and the steps discussed during the training, and more generally, to gain understanding about the consistency of intervention delivery. Furthermore, to understand the implementation of the intervention through the lens of the implementing agencies, in-depth interviews (IDIs) were conducted with SALT facilitators and district coordinators. Key objectives of the IDIs were to cross verify some of the indicators of the monthly reporting format and to know about the challenges faced during the implementation of the intervention.

Process evaluation data collection

The intervention started in March 2017 with training of facilitators. The evaluation team began process evaluation data collection from June. The initial months were used to gain understanding of the intervention and to develop reporting formats, various checklists, and IDI guides. Each of the nine SALT facilitators and three district coordinators were interviewed twice in person during the intervention phase. In addition, one-to-one follow up was done over phone and WhatsApp in order to resolve discrepancies around monthly process indicators. The interviews were conducted in Assamese language and were audio recorded. The audio-recordings were transcribed into English and further analyzed.

Statistical Analysis

All analyses were based on intention-to-treat principle, that is, by analysing all villages according to the group they were randomised to. We conducted individual-level analyses using logistic regression with a random effect for the village to account for correlation between observations from the same village. The effect of the intervention at endline on the primary and secondary outcomes, all binary in nature, was estimated using odds ratios. The model included fixed effects for strata to account for the stratified randomization and adjusted for the baseline level of the outcome variable (as log-odds-cluster-level proportion of the outcome) in order to improve the precision of the intervention effect estimates. Further analyses were conducted after adding the following covariates: gender and birth order of the child, mother’s age, education, spouse’s education, household head’s religion and caste, household wealth quintile (constructed based on housing characteristics, sanitation facility of the household, and asset possession), and village-level variables –proportion of households in the poorest wealth quintile, proportion of mothers who cannot read and/or write, and average travel time to vaccination site.

We used district, gender and birth order of child to explore heterogeneity in intervention effect, as part of predefined subgroup analyses. In addition, as post-hoc analysis, we explored if there is heterogeneity in the effect of the intervention by village-level baseline immunization coverage. For each subgroup, we repeated the main analysis with the addition of the subgroup variable along with its interaction with the intervention group indicator. Heterogeneity was assessed based on the statistical significance of the interaction term. We conducted a sensitivity analysis by combining the dropped villages (no intervention) with the control villages and comparing them to the intervention villages. Data from this repeated cross-sectional cluster randomized trial can also be analysed to assess whether the change from baseline to endline in the outcome differs between the two groups. We conducted a sensitivity analysis to examine the effect of the intervention on the change in outcome from baseline by including an interaction term between the survey round and intervention group.

Study statisticians were blinded to group allocation until all results were finalized. All analyses were performed in R.

Results

Post sample size re-estimation, we dropped 44 villages – 23 and 21 villages from the intervention and control groups respectively. The intervention group had 97 villages and the control group 99 villages with 32-33 intervention villages and 32-34 control villages in each district. The intervention was implemented in 90 villages – 89 villages out of the 97 in the intervention group and 1 village out of the 99 in the control group. The endline survey was conducted in the 97 intervention and 99 control villages, as well as in 43 of the 44 villages that were dropped. The primary, intention-to-treat analysis, uses data from the 97 intervention and 99 control villages, unless specified otherwise. In the 97 intervention and 99 control villages, we administered the survey in 1,424 and 1,465 households, respectively. We had information on 1,429 and 1,478 children aged 6-23 months and 985 and 966 children aged 12-23 months from the intervention and the control groups, respectively.

The 6-23 months old children and their mothers interviewed in the baseline survey in the two groups were very similar with respect to background characteristics. Households also had similar characteristics across the two groups. There were small differences between the two groups, for example, a higher proportion of household heads belonged to the ‘other backward class’ caste category in the control group than in the intervention group (31% vs. 21%). Villages in the control group, on average, had a higher proportion of households in the poorest wealth quintile as compared to the villages in the intervention group (median = 20%, interquartile range (IQR)=27% in control villages vs. median = 9%, IQR = 27% in intervention villages). The control villages also had on average a higher proportion of households where the head of household belonged to a scheduled tribe as compared to the control villages (median = 7%, IQR=47% in control villages vs. median = 0%, IQR=53% in intervention villages).

Characteristics	Intervention (N=1447)	Control (N=1482)	Total (N=2929)
Child	Sex	Boy 731 (51%)	763 (51%)	1494 (51%)
	Girl 716 (49%)	719 (49%)	1435 (49%)
Age in months	6-11 568 (39%)	586 (40%)	1154 (39%)
	12-23 879 (61%)	896 (60%)	1775 (61%)
Birth order	First 766 (53%)	769 (52%)	1535 (52%)
	Second 492 (34%)	475 (32%)	967 (33%)
	Third or more 186 (13%)	235 (16%)	421 (14%)
	Missing 3 (0%)	3 (0%)	6 (0%)
Born in a health facility	No 164 (11%)	220 (15%)	384 (13%)
	Yes 1280 (88%)	1259 (85%)	2539 (87%)
	Missing 3 (0%)	3 (0%)	6 (0%)
Has vaccination card	No 31 (2%)	24 (2%)	55 (2%)
	Yes 1416 (98%)	1458 (98%)	2874 (98%)
Mother	Age in years	Less than 20 80 (6%)	69 (5%)	149 (5%)
	20-24 502 (35%)	511 (34%)	1013 (35%)
	25-34 773 (53%)	792 (53%)	1565 (53%)
	35-plus 92 (6%)	110 (7%)	202 (7%)
Age at marriage	Less than 18 298 (21%)	283 (19%)	581 (20%)
	18-24 962 (66%)	1026 (69%)	1988 (68%)
	25 or more 187 (13%)	173 (12%)	360 (12%)
	Education	No schooling 175 (12%)	201 (14%)	376 (13%)
	Some primary 303 (21%)	369 (25%)	672 (23%)
	Some secondary 773 (53%)	724 (49%)	1497 (51%)
	More than secondary 196 (14%)	188 (13%)	384 (13%)
	Spouse’s education	No schooling 157 (11%)	164 (11%)	321 (11%)
	Some primary 298 (21%)	317 (21%)	615 (21%)
	Some secondary 753 (52%)	734 (50%)	1487 (51%)
	More than secondary 239 (17%)	267 (18%)	506 (17%)
	Received full antenatal care during pregnancy	No 827 (57%)	850 (57%)	1677 (57%)
	Yes 617 (43%)	629 (42%)	1246 (43%)
	Missing 3 (0%)	3 (0%)	6 (0%)
Household	Household head’s caste	Scheduled caste 106 (7%)	96 (6%)	202 (7%)
	Scheduled tribe 398 (28%)	378 (26%)	776 (26%)
	Other backward class 311 (21%)	454 (31%)	765 (26%)
	General/Don’t know 632 (44%)	554 (37%)	1186 (40%)
	Household head’s religion	Hindu 958 (66%)	1019 (69%)	1977 (67%)
	Muslim 431 (30%)	393 (27%)	824 (28%)
	Other 58 (4%)	70 (5%)	128 (4%)
	Household wealth quintile	Poorest 259 (18%)	321 (22%)	580 (20%)
	Poorer 284 (20%)	304 (21%)	588 (20%)
	Middle 290 (20%)	285 (19%)	575 (20%)
	Richer 328 (23%)	271 (18%)	599 (20%)
	Richest 286 (20%)	301 (20%)	587 (20%)
Village (cluster)	Na 97	99	196
	Percentage of households in poorest wealth quintile, Median (IQR)	9% (27%)	20% (27%)	13% (27%)
	Percentage of mothers who cannot read and/or write, Median (IQR)	13% (20%)	13% (20%)	13% (20%)
	Percentage of households with Muslim head of household, Median (IQR)	0% (67%)	0% (50%)	0% (67%)
	Percentage of households with the head of household belonging to a scheduled tribe, Median (IQR)	0% (53%)	7% (47%)	6% (47%)
	Average time to vaccination site (in minutes), Median (IQR)	20 (11)	18 (10)	19 (11)

Data are N (%) unless specified otherwise. N indicates number of children expect for variables measured at the village level (indicated by a).

Results

We present the background characteristics of the participants in the endline survey in the appendix.

Outcome	Intervention N (%)	Control N (%)	Total N (%)	Unadjusted Odds ratio (95% CI)	P value	Adjusted Odds ratio (95% CI)	P value
Full immunization (12-23 month old child)	No 230 (23%)	223 (23%)	453 (23%)	0.99 (0.72, 1.36)	0.94	0.98 (0.71, 1.36)	0.92
	Yes 753 (76%)	742 (77%)	1495 (77%)
Three doses of DPT (6-23 month old child)	No 253 (18%)	258 (17%)	511 (18%)	0.99 (0.74, 1.33)	0.97	1.01 (0.76, 1.35)	0.95
	Yes 1173 (82%)	1218 (82%)	2391 (82%)

Explanatory variables in the unadjusted model include the group assignment indicator, the strata variable used for stratified randomization, and the baseline village-level log-odds of the outcome. Explanatory variables in the adjusted model include, in addition to the variables in the unadjusted analysis, gender and birth order of the child, mother’s age, education, spouse’s education, household head’s religion and caste, household wealth quintile, and village-level variables –proportion of households in the poorest wealth quintile, proportion of mothers who cannot read and/or write, and average travel time to vaccination site.

Discussion

Community engagement intervention like SALT, in theory, has the potential to improve immunization coverage as it can help identify barriers to vaccination at the local level and thus might lead to customized and sustainable solutions. The findings from this evaluation study based on a cluster randomized controlled trial, however, showed no effect of the SALT intervention on children’s immunization coverage in our study population after one year of implementation.

A meta-analysis of the effectiveness of community engagement in public health interventions suggests that there is solid evidence that community engagement approaches have a positive impact on a range of health outcomes including health behaviours (for example, diet, physical activity, smoking habits), health consequences (for example, change in body mass index, reduction in cholesterol) and participant self-efficacy pertaining to the health behaviours. However, studies evaluating the impact of community engagement interventions in improving immunization coverage are sparse. Our study addresses this gap in the literature.

Randomized controlled trials of complex interventions like SALT are often criticised as being a ‘black box’ as it can be difficult to know why and how the intervention worked (or not). The absence of evidence of an impact in our study could be simply because SALT was ineffective in this particular context or it could potentially be because of less than optimal implementation of the intervention. We attempted to collect relevant process indicators to explain the findings from the evaluation study. The potential reasons behind the observed lack of evidence of an impact are elaborated below.

The process evaluation data suggest less than optimal coverage of the intervention – the proportion of households in the village reached out to by SALT facilitators was less than 19% for half the villages. In addition, we do not know whether the households selected for endline evaluation were exposed to the intervention or not. We did not include questions on participation in SALT activities in endline survey to prevent the possibility of data collectors being able to identity the intervention villages from the responses. Unblinding of outcome assessors in a randomized controlled trial can lead to substantial bias in the assessment of the outcome variable. Moreover, for a multi-step complex intervention like SALT, it was not straightforward to define exposure to the intervention.

The intervention could potentially have been more targeted in order to reach the last mile. In our baseline assessment (June-August 2016), we found that all three study districts had significantly higher immunization coverage compared to estimates from previous surveys. This most likely is due to implementation of Mission Indradhanush, a supplementary immunization programme of the Government of India, in the study districts prior to this study. Given the high level of vaccination coverage at baseline in the three districts, to achieve further improvement through SALT intervention, it would have been better if the intervention was adapted to engage with the marginalized and hard to reach population in a targeted manner. While within village transfer and sharing of learning among households was one of the assumptions of SALT intervention, this may not have happened per expectation during the implementation.

Any behavioral change takes time and requires sustained effort until a critical mass is reached, more so if it requires identifying and reaching out to the population who are usually left behind by the health system. The limited time frame for the SALT intervention was perhaps inadequate for sustained interactions with the community, and for development of a sense of ownership at the community level regarding the issue of children’s immunization. Also, the concept of SALT is difficult to grasp for both the facilitators and the communities. Allowing more time for training and implementation of the intervention could have helped improve uptake and achieve the desired level of community ownership. To address vaccine hesitancy in US, a relatively longer term intervention that was implemented for three years showed the promise of using parent advocates as part of a community-based approach to reduce vaccine hesitancy.

The SALT intervention being grounded in a democratic, sensitive and ethical approach, ideally it is the community who decides their priorities. It is therefore natural that some communities may not prioritize immunization, especially if there are other pressing issues. Our experience during process evaluation suggests that many communities initially did not perceive immunization as a major problem. However, child health was accepted by all villages as a shared dream, and it was through this dream that the facilitators stimulated the discussion around immunization. The objective of the study was to evaluate the impact of SALT in increasing immunization coverage. Hence the facilitators had to stimulate the discussion so that immunization related issues ‘emerged’ as a topic during dream building and practices around immunization are taken up for developing action plans. In one sense, this is a conflict between the idea of community ownership and decision-making and some level of external influence or ‘coercion’ by the facilitators during the implementation of the intervention.

In the context of evaluating a community engagement intervention, our study grappled with some dilemmas. Firstly, we debated whether to involve the entire community or include only households of eligible women in the intervention. Had we only recruited households having pregnant women in villages before the start of the intervention and implemented SALT with them, we may have observed better immunization outcomes at endline, because the implementation would have been more targeted and would have utilized the resources in an efficient manner. We, however, believe it would not have led to sustainable improvement in immunization outcomes. Also, it would not have aligned well with the objective of development of community ownership of the issue of immunization, as majority of the community members would have remained unexposed to the intervention.

Secondly, we deliberated on what would be a more appropriate design choice for evaluating a community engagement intervention. The cohort design is commonly used, that is, recruit participants at baseline, implement the intervention with the recruited participants from the community, and use the same set of participants for evaluating outcomes at endline. If the extent of reach of the intervention is an important metric for evaluation, this design choice may not be the ideal. We argue that a repeated cross-sectional design, where the communities remain the same at baseline and endline but a new random sample of eligible participants are recruited at endline for evaluation of outcomes, is perhaps more appropriate to ensure reach of intervention is incorporated in evaluation.

Strengths of our study include the use of a theoretically grounded community engagement intervention and the use of robust methods to assess the intervention. The SALT community engagement intervention is participatory, interactive, involves multiple inter-related steps and requires continued engagement. Based on the level of participation and involvement of the community, community engagement interventions can be classified into five graded categories ranging from inform, consult, involve, to collaborate and empower; the last being the highest form of engagement. Most common intervention strategies in public health include providing education and advice which can be considered as low-level of community engagement. The next common intervention strategies provide social support and skill development training through involvement with the community and can be classified as mid-level of community engagement. The SALT intervention belongs in the fifth category of empowering the community. In the context of evaluation design, the internal validity of our study is justified through identification of a comparable control group, selection of clusters and participants using probability sampling design, consideration of a sample size adequate to detect programmatically significant effect size with recommended statistical power, random allocation of clusters to the intervention and control groups ensuring treatment allocation being independent of outcome, measurement of relevant covariates at the participant and cluster level.

Our impact evaluation study includes two primary outcomes – full immunization in 12-23 months old children and three doses of DPT in 6-23 months old children. Having two primary outcomes may require adjustment for multiplicity of testing. However, the two age-groups overlap, and the two outcomes are strongly correlated. It would be difficult to control the error rate without being overly conservative. We therefore made no adjustments for multiplicity of testing. Moreover, the vaccination outcomes are constructed using information recorded from vaccination cards and that elicited from mothers. Depending on the age of the child and complexity of vaccination schedule, accuracy of mother’s recall data is questionable. On the other hand, vaccination card may be unavailable, or all vaccinations may not be recorded accurately in the card. Sensitivity analysis suggest that coverage rates are different depending on the data source used for defining child’s vaccination status – vaccination card only, mother’s recall only and combination of the two.

Conclusion

The null results from our study suggest that a more targeted implementation strategy may be the way forward if SALT community engagement intervention has to be effective across various contexts. Behavior change in villages could take time and require sustained effort. We believe that the SALT approach may be better suited in situations that target a smaller group of individuals or where the issue is perceived as a priority by the community and emerges organically through discussions, for instance, in villages with high prevalence of vaccine hesitancy.

Acknowledgements

The study team acknowledges the support from the grant holding institute Public Health Foundation of India (PHFI), particularly during the period when PHFI could not receive foreign funds for the project due to withdrawal of its FCRA approval. The detailed feedback from the 3ie review panel through the course of the Study helped the study team think through and improve different components of the study. We are grateful to our implementation partner the Constellation for allowing an evaluation team to scrutinize and critique their work. Several individuals on their team led by Philip Forth, Rituu Nanda and Marlou De Row made invaluable contributions throughout the study period. We thank Bobby Zachariah, Ian Campbell, Jean-Louis Lamboray, Luc Barriere Constantin and other from the Constellation for their support. The Constellation was supported by their local partners in Assam; CNES and VHAA. We thank Ashok Rao and Ruchira Neog and their team, in particular, each SALT facilitator for their heartfelt and sincere efforts in the field. We thank the Department of Health & Family Welfare, Government of Assam for their support and openness to this study being done in Assam.

Author Contributions

Conceptualization: Santanu Pramanik, Arpita Ghosh, Sandra Albert

Data collection and monitoring: Samina Parveen, Jahnabi Hazarika, Anirban Goswami

Data validation and curation: Arpita Ghosh, Tiken Das, Santanu Pramanik

Methodology: Santanu Pramanik, Arpita Ghosh

Process evaluation: Anirban Goswami, Samina Parveen, Jahnabi Hazarika

Statistical analyses: Arpita Ghosh, Tiken Das, Laurent Billot

Supervision and management: Sandra Albert, Santanu Pramanik

Writing- original draft: Arpita Ghosh, Santanu Pramanik

Writing- review & editing: Arpita Ghosh, Santanu Pramanik, Laurent Billot, Sandra Albert

References

1. WHO. Global Vaccine Action Plan 2011-2020. 2013. 9241504986.
2. IIPS. National Family Health Survey (NFHS-4) India 2015-16. India Fact Sheets. 2017.
3. Vashishtha VM. Status of immunization and need for intensification of routine immunization in India. Indian pediatrics. 2012;49(5):357-361.
4. UNICEF India. Coverage Evaluation Survey (CES), 2009: All India report. 2010:Accessed on Oct 10, 2013 at http://www.unicef.org/india/1_-_CES_2009_All_India_Report.pdf.
5. Pramanik S, Muthusamy N, Gera R, Laxminarayan R. Vaccination coverage in India: A small area estimation approach. Vaccine. 2015;33(14):1731-1738.
6. Favin M, Steinglass R, Fields R, Banerjee K, Sawhney M. Why children are not vaccinated: a review of the grey literature. International health. 2012;4(4):229-238.
7. Rainey JJ, Watkins M, Ryman TK, Sandhu P, Bo A, Banerjee K. Reasons related to non-vaccination and under-vaccination of children in low and middle income countries: findings from a systematic review of the published literature, 1999–2009. Vaccine. 2011;29(46):8215-8221.
8. Ghosh A, Laxminarayan R. Demand-and supply-side determinants of diphtheria-pertussis-tetanus nonvaccination and dropout in rural India. Vaccine. 2017;
9. MoHFW. Four years of NRHM (2005-2009)- Making a Difference Everywhere. 2009.
10. MoHFW. Mission Indradhanush Operational_Guidelines. 2015.
11. Oyo‐Ita A, Nwachukwu CE, Oringanje C, Meremikwu MM. Cochrane Review: Interventions for improving coverage of child immunization in low‐and middle‐income countries. Evidence‐Based Child Health: A Cochrane Review Journal. 2012;7(3):959-1012.
12. Johri M, Pérez MC, Arsenault C, et al. Strategies to increase the demand for childhood vaccination in low‐and middle‐income countries: a systematic review and meta-analysis. Bulletin of the World Health Organization. 2015;93(5):339-346.
13. Hahn S, Puffer S, Torgerson DJ, Watson J. Methodological bias in cluster randomised trials. BMC Medical Research Methodology. 2005;5(1):10.
14. Jain M, Taneja G, Amin R, Steinglass R, Favin M. Engaging communities with a simple tool to help increase immunization coverage. Global Health: Science and Practice. 2015;3(1):117-125.
15. Sabarwal S, Bhatia R, Dhody B, Perumal S, White H, Puri J. Breaking through stagnation: testing innovative approaches to engaging communities in increasing immunisation coverage A scoping study. 2015;
16. Saeterdal I, Lewin S, Austvoll‐Dahlgren A, Glenton C, Munabi‐Babigumira S. Interventions aimed at communities to inform and/or educate about early childhood vaccination. The Cochrane Library. 2014;
17. Oyo‐Ita A, Wiysonge CS, Oringanje C, Nwachukwu CE, Oduwole O, Meremikwu MM. Interventions for improving coverage of childhood immunisation in low‐and middle‐income countries. The Cochrane Library. 2016;
18. Owais A, Hanif B, Siddiqui AR, Agha A, Zaidi AK. Does improving maternal knowledge of vaccines impact infant immunization rates? A community-based randomized-controlled trial in Karachi, Pakistan. BMC public health. 2011;11(1):239.
19. WHO, UNICEF. Report on the Review of ACP and COMATAA re HIV/AIDS in Papua New Guinea. Available at http://www.communitylifecompetence.org/uploads/File/evaluations/Evaluation%20PNG.pdf. 2009.
20. Teerawattananon Y, Yamabhai I, Leelahavarong P. Using capability index to determine a value for money of the AIDS Competence Process in Thailand. Research Report, Health Intervention and Technology Assessment Program (HITAP), Thailand. 2011. http://aidscompetence. ning.com/forum/attachment/download; 2011:
21. Legastelois J, Forth P. The ‘Competence approach’ in the battle against malaria in Togo and The Gambia. 2009.
22. Bricq E, Vololomanitra B, Jean-Baby F, Raymond H, Paul K, Jean-Louis L. Community-based intervention increases vaccine rates: A study of polio vaccination in the Democratic Republic of Congo. Unpublished Manuscript. 2015;
23. Pandey P, Sehgal AR, Riboud M, Levine D, Goyal M. Informing resource-poor populations and the delivery of entitled health and social services in rural India: a cluster randomized controlled trial. Jama. 2007;298(16):1867-1875.
24. Andersson N, Cockcroft A, Ansari NM, et al. Evidence-based discussion increases childhood vaccination uptake: a randomised cluster controlled trial of knowledge translation in Pakistan. BMC International Health Human Rights. 2009;9(S1):S8.
25. Habib MA, Soofi S, Cousens S, et al. Community engagement and integrated health and polio immunisation campaigns in conflict-affected areas of Pakistan: a cluster randomised controlled trial. The Lancet Global Health. 2017;5(6):e593-e603.
26. Pramanik S, Ghosh A, Nanda RB, de Rouw M, Forth P, Albert S. Impact evaluation of a community engagement intervention in improving childhood immunization coverage: a cluster randomized controlled trial in Assam, India. BMC public health. 2018;18(1):534.
27. McRae AD, Weijer C, Binik A, et al. When is informed consent required in cluster randomized trials in health research? Trials. 2011;12(1):202.
28. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials. 2013;14(1):15.
29. Ukoumunne OC, Thompson SG. Analysis of cluster randomized trials with repeated cross‐sectional binary measurements. 2001;20(3):417-433.
30. Austin PC. A comparison of the statistical power of different methods for the analysis of repeated cross-sectional cluster randomization trials with binary outcomes. 2010;6(1).
31. O’Mara-Eves A, Brunton G, Oliver S, Kavanagh J, Jamal F, Thomas J. The effectiveness of community engagement in public health interventions for disadvantaged groups: a meta-analysis. BMC public health. 2015;15(1):129.
32. Schoeppe J, Cheadle A, Melton M, et al. The Immunity Community: a community engagement strategy for reducing vaccine hesitancy. Health promotion practice. 2017;18(5):654-661.
33. Guyatt GH, Oxman AD, Vist G, et al. GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias). Journal of clinical epidemiology. 2011;64(4):407-415.
34. Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. International journal of surgery. 2012;10(1):28-55.
35. Higgins JP, Savović J, Page MJ, Elbers RG, Sterne JA. Assessing risk of bias in a randomized trial. Cochrane Handbook for Systematic Reviews of Interventions. 2019:205-228.
36. di Ruffano LF, Dinnes J, Sitch AJ, Hyde C, Deeks JJ. Test-treatment RCTs are susceptible to bias: a review of the methodological quality of randomized trials that evaluate diagnostic tests. BMC medical research methodology. 2017;17(1):35.
37. Victorian Auditor-General. Public Participation in Government Decision-Making. 2017.
38. Pramanik S, Agrahari, K, Srivastava, A., Varanasi, V., Setia, M., Laxminarayan, R. Integrated Child Health and Immunization Survey- Rounds 1 & 2. 2016. 1530-1567.