
Description
Founded in 1920, the NBER is a private, non-profit, non-partisan organization dedicated to conducting economic research and to disseminating research findings among academics, public policy makers, and business professionals.
Reviewers recommend incorporating demographic data on staff to better understand differences in mandate compliance and its effects across various groups.
RR\ID Evidence Scale rating by reviewer:
Reliable. The main study claims are generally justified by its methods and data. The results and conclusions are likely to be similar to the hypothetical ideal study. There are some minor caveats or limitations, but they would/do not change the major claims of the study. The study provides sufficient strength of evidence on its own that its main claims should be considered actionable, with some room for future revision.
***************************************
Review: Gandhi et al.’s study of the effects of private vaccination mandates on nursing home staff vaccination rates, staffing levels, and patient health outcomes is timely and important. By combining data on COVID-19 vaccine mandates from the publication McKnights, on nursing home staffing data from the Centers for Medicare and Medicaid Services, and on resident health outcomes from the Centers for Disease Control and Prevention, the authors compile a valuable data set and analyze it using difference-in-differences (DID) methods that are largely appropriate for panel data. This serves to answer pressing questions about the effect of these mandates on the various outcomes.
The authors find that mandates reduce staff and resident COVID-19 cases and resident COVID-19 deaths substantially. These mandates also increase staff vaccination rates substantially but have small (yet statistically significant) negative effects on nursing home staffing, especially among workers who work less than 20 hours per week. The authors find limited to no evidence of negative effects on various staffing-associated resident care outcomes.
The main claims are well supported by the evidence generated in this study. The data source is a particular strength, with the authors carefully linking data sets targeted to answer the question at hand and clearly describing criteria for exclusion based on incomplete data. The methods are generally sound—the authors use DID approaches that can account for dynamic effects to avoid biases that are common in staggered adoption setting [2] and use various sensitivity analyses and appropriate checks for pre-trends. For the analyses with COVID-19 health outcomes, the authors identify a matched set of control units with similar geographies and recent case histories, increasing the likelihood that differences in recent cases are not drivers of non-parallel trends between the groups [3,4]. Definition of treatment timing is also handled with care to try to reduce anticipation effects [3].
Some limitations are inherent to any observational study, and to the use of DID-based methods for causal inference. The most substantial here are the challenging interpretation of the estimands in this paper due to: (1) potential heterogeneity in effects by calendar time (specifically, by timing within epidemic waves); and (2) the challenge of generalizability to a broader set of nursing home facilities. Regarding the former, while the methods used by the authors account for dynamic effects, they do not provide a clear weighting scheme across cohorts or, in this case, across the timing of intervention [2,5]. As indirect effects of vaccination can vary greatly by pathogen prevalence and mutation, this is a notable limitation. Other methods that can clarify these estimands and assess the extent of the heterogeneity would be helpful [2,5]. The latter is a more inherent limitation to the quasi-experimental approach to causal inference. The authors are generally cautious not to extrapolate beyond the average treatment effect on the treated that they identify, but they should clarify some of the specific features of the treated nursing homes more clearly to be specific about for which settings their results are most reliable. Quantitative extrapolations (as at the bottom of p. 23), should be taken with a great deal of uncertainty.
Overall, this is a reliable and valuable study. Its results add to the growing body of evidence on indirect effects of the COVID-19 vaccines. More importantly, they provide among the best evidence so far on the effects of vaccine mandates. By using specific settings, reliable and targeted data, and generally appropriate methods, this study is much more valuable than studies that have made sweeping claims based on much shakier evidence. The results here will be particularly meaningful for administrators of nursing homes and other health care settings. Despite the challenges of transportability, the results should also be taken into account by policymakers and regulators, as well as other researchers investigating population-level effects of vaccination.
References:
Gandhi A, Larkin I, McGarry B, et al. The health and employment effects of employer vaccination mandates. 2024. NBER Working Paper 33072; https://doi.org/10.3386/w33072.
Roth J, Sant’Anna PHC, Bilinski A, Poe J. What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. J Econom. 2023;235(2):2218–2244. https://doi.org/10.1016/j.jeconom.2023.03.008.
Goodman-Bacon A, Marcus J. Using difference-in-differences to identify causal effects of COVID-19 policies. Surv Res Meth. 2020;14(2):153–158. http://doi.org/10.18148/srm/2020.v14i2.7723.
Feng S, Bilinski A. Parallel trends in an unparalleled pandemic: Difference-in-differences for infectious disease policy evaluation. 2024. medRxiv; https://doi.org/10.1101/2024.04.08.24305335.
Feng S, Ganguli I, Lee Y, Poe J, Ryan A, Bilinski A. Difference-in-differences for health policy and practice: A review of modern methods. 2024. arXiv; https://doi.org/10.48550/arXiv.2408.04617.
Reviewers recommend incorporating demographic data on staff to better understand differences in mandate compliance and its effects across various groups.