Skip to main content
SearchLoginLogin or Signup

Review 1: "Promoting Public Health with Blunt Instruments: Evidence from Vaccine Mandates"

The reviewers agree that the study is generally well-executed, using a difference-in-difference model to demonstrate that mandates likely decreased the probability of working in healthcare by 6%.

Published onMay 16, 2024
Review 1: "Promoting Public Health with Blunt Instruments: Evidence from Vaccine Mandates"

RR:C19 Evidence Scale rating by reviewer:

  • Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.


Review: The paper is generally well-executed, with reasonable methods and a solid set of robustness checks. My major critiques are outlined below:

  1. The authors need to do more to engage with the context under which state mandates were being implemented. In particular, a federal vaccine mandate for health care workers was announced in November 2021, with an initial enactment date of December 2021. Of course, the mandate went through a number of legal challenges, but was eventually upheld in January of 2022 (with enactment in March of 2022). As such, the control states undergo various degrees of “treatment” for much of the study window underlying the main event study analyses. This nuance should be discussed upfront in the manuscript, and the authors should consider using a shorter follow-up period as their primary specification. For example, previous work on the effect of state-level vaccine mandates for health workers ended follow-up in mid-November 2021 to avoid this type of confounding (see McGarry et al, 2022).

  2. A fair rebuttal to the above point is that the federal vaccine mandate might be expected to bias results towards the null of no difference in employment outcomes between states with and without a mandate if the threat of a federal mandate led to some workers proactively look for jobs outside the health care sector. To that end, I am puzzled as to why there is no evidence of any response to the federal mandate in the unadjusted data presented in Figure 4. Even when fully enacted in March of 2022, there is not a discernable drop in health worker probability in non-mandate states. I realize this is outside the stated scope of the paper, but it is hard for me to think of a story whereby a state mandate would compel individuals to exit the health care sector but a federal mandate would not. In the absence of a convincing explanation, I am left wondering if the employment patterns are somehow related to sampling or response rate differences between mandate and non-mandate states or some other, non-causal, explanation.

  3. Another important contextual factor not considered in the paper is that decisions about whether and when to implement a state vaccine mandate are endogenous. The authors do some balance checks, but these largely focused on static pre-pandemic state characteristics. One important factor not considered is each state’s monthly Covid-19 burden. For example, imagine that some states elected to implement a vaccine mandate because they had prior experience or other private information that suggested their health facilities would be especially hard hit by Covid. Because Covid outbreaks within health care facilities can be a driver of staff departures (see for example Shen, K., et al (2022, July). Staffing patterns in US nursing homes during COVID-19 outbreaks. In JAMA Health Forum) this type of selection bias could inflate the estimated staffing effects of the mandate. This specific example could be addressed by including in the regression estimates of the monthly Covid-19 burden (community covid case rates and covid hospitalization rates are both available at the county-week level). Additional efforts to account for pre-trends could strengthen the inferences drawn in this paper as there does appear to be a negative, albeit non-significant, pretend in Figure 5 that might explain about 1/3 of the current effect estimate based on my eyeball test.

  4. I think this paper would be significantly improved if it dealt with not only the potential employment costs of a mandate, but with the potential benefits as well. Expanding the analysis to include vaccination rates, workplace absences, Covid-19 case and mortality rates for the targeted population, or some other marker of the potential beneficial effects of a mandate would help policy makers more directly weigh the relevant tradeoff when considering future mandate policies.

Minor Comments:

  1. I did not find the mortality analyses particularly compelling, and as such, the discussion of these results in the “Robustness and Extensions” felt speculative. Admittedly, this might be related to the fact that the supporting graph in the appendix (Figure A5) appears to be an accidental duplication of Figure A1. Regardless, I would suggest removing this secondary analysis from the paper. It seems quite complex to disentangle the potential quality effects from staff exits from the potential protective effects of higher health care worker vaccination rates, and an appendix-only analysis does not due this important question justice.

  2. The findings that the negative mandate effects on employment were stronger in ACA Medicaid expansion states does not feel well-motivated or reasonably interpreted. Insurance as a mechanism to access the vaccine doesn’t feel relevant because virtually all health care facilities were able to offer the vaccine to their staff for free. Its not clear what the analyses in Table 4 are adding.

No comments here
Why not start the discussion?