Review 2: "Exploratory analysis of immunization records highlights decreased SARS-CoV-2 rates in individuals with recent non-COVID-19 vaccinations"

RR:C19 Evidence Scale rating by reviewer:

• Misleading. Serious flaws and errors in the methods and data render the study conclusions misinformative. The results and conclusions of the ideal study are at least as likely to conclude the opposite of its results and conclusions than agree. Decision-makers should not consider this evidence in any decision.

***************************************

Review:

The authors present an interesting attempt to explore the potential protective effect against SARS-CoV-2 infection for recent vaccination with 18 common life-course vaccines. This analysis uses a rich electronic health record (EHR) dataset starting early in the pandemic that includes a large number of infected and negative individual. The authors find significant protective effect from multiple vaccinations at different time horizons. However, the methods and potential for bias create substantial doubt in these effects, and causes concern for publishing and communicating such strong results.

There are several details that are not included in this manuscript that are critical to assess the validity of this analysis. This includes what specifically is meant by the inclusion criteria of having had at least 1 ICD code in the past 5 years. Does this include codes for the current visit, during which the PCR test for SARS-CoV-2 is being performed? Additionally, what population coverage does the Mayo health system have? Does it include all clinics, including public health clinics, where populations that we know have been at higher risk (lower SES in particular) would have been more likely to be vaccinated? 

The associations between recent vaccination and lower rates of infection found in this manuscript are very possibly the result of selection bias in the population included. Specifically, individuals with higher SES or better insurance may be more likely to be regular recipients of healthcare through the Mayo health system and may be more likely to have documented vaccination or have been received vaccination at all. Those becoming infected may be more likely to be not receive vaccination or not have it documented within the Mayo EHR, having received vaccination elsewhere. They may be more at risk of infection due to working jobs that put them more at risk, having higher household crowding, or other reasons. An immediate indicator of these spurious associations is that specific vaccines have a substantial and significantly negative effect, which seem to be more indicators of who the people receiving them are, than the immunological effects of the vaccines.

Furthermore, it strikes me as strange in the race/ethnicity-stratified tables that all vaccines are not listed for all races? For example, “RZV Zoster (ZOSTAVAX, SHINGRIX)” is listed as having a protective effect in the 1-year time horizon for Asians (Table 7) and the 5-year time horizon for Blacks (Table 9), but not listed at all for Whites or Hispanics. There seems to be selective reporting of results.

Overall, I am concerned that the potential biases, which are detailed in the discussion, are not given more weight by the authors, and by ignoring them, and presenting such apparently strong impact of recent vaccination on prevention of infection, this paper pushes a narrative that might be largely invalid, but will be consumed widely without considering these problems.