RR:C19 Evidence Scale rating by reviewer:
Reliable. The main study claims are generally justified by its methods and data. The results and conclusions are likely to be similar to the hypothetical ideal study. There are some minor caveats or limitations, but they would/do not change the major claims of the study. The study provides sufficient strength of evidence on its own that its main claims should be considered actionable, with some room for future revision.
***************************************
Review: Clarke-Deelder E, et al. (2023, original version): Health impact and cost-effectiveness of expanding routine immunization coverage in India through Intensified Mission Indradhanush: a quasi-experimental study and economic evaluation.
Overall, we commend the authors. Their analysis has been carefully and systematically performed. Their conclusions are generally supported by the data and analyses that they present. We do, however, wish to offer the following suggestions for refinements in the analysis and clarifications in the presentation.
Page 3:
Regarding “In 2019, an estimated 10% of children did not receive a single dose.” Here we suggest that the authors add the word “globally” for clarification.
Regarding “…leading to sharp reductions in coverage.” We suggest that the authors expand and quantify their statement based on the study cited, so the sentence reads “…leading to sharp reductions in coverage with 30 million children missing DTP3 doses globally in one year.”
Page 6:
Regarding “Sub-center-level cost data were imputed for two districts.” We suggest that the authors describe briefly how this was done and what it was based on.
It was helpful that the authors could cite their previous (2021) publication for methods on the cost analysis. It appears that time estimation was based entirely on interviews. The authors may wish to state whether they were able to validate the responses against work schedules, time logs, or other objective measures.
In addition, one recurrent challenge in cost analysis, particularly in government facilities, concerns “unallocated” time. That is, there may be a gap between the time allocated to specified activities, such as IMI in this study and other responsibilities, and the total paid time. For example, in many countries, patients come in the morning and staff are less busy in the afternoons, perhaps spending some of the time completing documentation. Often it is useful to allocate the unallocated time in a pro-rata portion to the allocated time. For example, half of the time were not explicitly allocated, then this allocation would double the total time for a specified activity. Implicitly, the cost analysis assumes that staff had time available that could easily be rechanneled into the IMI activities. This reviewer confronted the challenge of unallocated staff time in costing studies of intensified efforts in the United States in the citations below:
R165. Shepard DS, Daley MC, Neuman MJ, Blaakman AP, McKay JR. Telephone-based counseling in substance abuse treatment: Economic analysis of a randomized trial. Drug and Alcohol Dependence. 2016, 159:109–116. ISSN 0376-8716. Web: dx.doi.org/10.1016/j.drugalcdep.2015.11.034. (PMID: 26718395; NIHMSID 745000)
R166. Shepard DS, Lwin AK, Barnett NP, Mastroleo N, Colby SM, Gwaltney C, Monti PM. Cost-effectiveness of motivational intervention with significant others for patients with alcohol misuse. Addiction. 2016, 111(5):832-839. doi: 10.1111/add.13233 (PMID: 26574195).
We suggest that the authors comment on whether they were able to compare the total time spent on specified activities against paid time and validate their interview results, and the implications of what they were able, and not able, to measure in their study.
Page 8:
With zero-dose children, we suggest that the authors consider how limitations in data collection could cause misclassification. No data system is perfect. If a second dose were mistakenly recorded as a different child receiving a first dose, then the number of children vaccinated would be artificially increased. Conversely, the number of children completing the vaccination series would have been artificially reduced. The authors may wish to discuss this possibility. They should discuss what, if any, differences in the degree of data imperfections might have occurred between the IMI and control districts over time.
Regarding the measurement of the impact of IMI on under-five deaths averted. The use of the LiST tool seems reasonable but would appear to exclude the main benefits of hepatitis B vaccination. As the authors noted on p20, hepatitis B vaccine helps prevent deaths from liver cancer, but most of these would occur in adulthood, rather than in children under 5. The authors should seek some other way to incorporate this benefit beyond their focus on under-5 deaths if possible. If they cannot do this, they should discuss how this limitation may bias their results.
Page 9:
We would suggest that the authors consider several refinements for Table 1. First, we suggest that the rows be organized into panels based on the category of data, with a heading for each category, such as: economic data, latest population data, vaccination coverage, etc. Second, for clarity, we suggest that the table specify the units for each parameter (e.g. per 1000 births, years, etc). Third, for each source, we suggest that the table provide a numbered citation to the specific document rather than just the name of the organization. Fourth, we suggest a formatting change. We suggest that the column “Study parameter” be left justified to increase readability.
We were pleased that Table S1 was systematically organized according to the time points in the Indian immunization schedule. However, we would suggest some formatting refinements. Instead of making the time point a column, where the information is repeated at the start of each row, it would be clearer to show it in the title for a subsection or panel within the table. Also, as the table primarily contains text, it would be easier to read if each column were left formatted.
Page 10:
In many economic analyses, it is preferable to discount the life years gained to make them parallel to the treatment of economic data. The methods regarding the life expectancy calculation did not describe any discounting on p 13. However, we were pleased to read results for both discounted life years and discounted DALYs subsequently. We suggest that the methods note that these discounted calculations will be done and describe how. It appears that the authors assumed that every infant lived for exactly the average life expectancy at age 2.5 years. This as a reasonable assumption but should be stated explicitly.
The inclusion of treatment costs averted by using the work of Ozawa et al is clever. To clarify the process, however, it would be helpful to put in the main text the fact that the costs have been discounted. In the appendix, the authors could also note the citation number from the main text.
Page 11:
Page 12:
Page 13:
Page 14:
Page 16:
Page 17:
Page 18:
We suggest that several items be added to the discussion, enumerated below. As the discussion begins on p18, we have labeled them as 18a, 18b, etc.
18a. It is noteworthy that IMI proved to be effective and highly cost effective, but apparently was discontinued after only a few months. If this ending was due to the expiration of some external funding, the authors should state that. It would be helpful to indicate what, if any, attempts were made to continue this intensified effort in the intervening 5 years, and the results.
18b. When a public health service changes, it often has spillover effects towards other services. To illustrate a harmful change, vaccine hesitancy related to the COVID 19 vaccine globally or the Sanofi dengue vaccine (Dengvaxia) in the Philippines dampened update of other vaccines. On a positive side, IMI might have increased use of primary care services as the IMI contacts might have increased rapport between health workers and the population. It would be helpful to share any data or speculation available.
18c. While we agree that there is little literature on the cost-effectiveness of interventions to increase immunization coverage, we feel that conditional cash transfers deserve more attention as a point of comparison. As a start, the World Health Organization has published the following systematic review:
https://www.who.int/tools/elena/review-summaries/cash-transfer--the-impact-of-conditional-cash-transfers-on-health-outcomes-and-use-of-health-services-in-low-and-middle-income-countries
18d. The authors indicated that IMI failed in some districts (indicated by “dominated” on the cost-effectiveness confidence intervals). Their previous paper also noted substantial variation in incremental cost per zero-dose child and called for more research find out why. With this paper being written several years later, it would be useful to understand what steps were taken to find out the reasons and any findings that were found.
18e. The time trends in Figure 1 are very informative. Even though the period of IMI was less than a year, the impacts appear to highly concentrated in the first part of the IMI period for several vaccines: OPV1, DTP1, M1, M2, DTPb, and OPVb. One worrying interpretation of this finding is that the impact the result of the novelty of the IMI approach and was short lived. If that were the case, the wider replication of the approach would be limited. We would encourage the authors to describe and interpret the time pattern.
In summary, this paper provides a detailed estimate of the effectiveness and cost-effectiveness of supply-side interventions to increase vaccination doses in India. It will be instructive to compare the results against demand side measures, such as the conditional cash transfers offered to eligible low income families in some countries.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
