RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
The authors used propensity-score matching among individuals in the Mayo Clinic network to examine the real-world efficacy of COVID-19 vaccines on testing positive for COVID-19 and being hospitalized with COVID-19. They found that COVID-19 vaccination was 88.7% effective in preventing SARS-CoV-2 infection and reduced hospital admission rates (3.7% vs. 9.2%)
Overall, this is a reasonably well-done study that provides useful real-world evidence on COVID-19 vaccines. The main point of caution when interpreting their findings and point estimates is that the propensity-score matching was likely not fully successful at generating exchangeable populations. The amount of bias this conferred in the point estimates is difficult to know, but the qualitative findings are likely to be similar. This paper could be improved with attempts to develop a more robust propensity score model to strengthen the causal inference.
This is a reasonable study using propensity-score matching to compare the incidence of symptomatic COVID-19 and COVID-19 related hospitalizations among those vaccinated and those not vaccinated. There are several benefits to examining the efficacy of COVID-19 vaccines in real-world settings. First, the sample size can be much larger and allow for more precise estimates for more distal outcomes. It also may account for potential behavior change that occurs when knowing one is protected after getting vaccinated (which would not happen in the original RCTs). Their findings do align with RCT evidence as well as the evidence from another large observational study from Israel. Namely, vaccines confer 90-95% protection against symptomatic COVID-19 and also prevent more severe outcomes such as hospitalizations. This body of literature helps build the growing evidence on vaccines and study designs such as this will allow us to rigorously examine outcomes that could not be assessed in the original trials.
The main issue at hand when assessing this study is whether their propensity-score matching was successful at generating populations who did and did not receive the vaccine that were similar on both measured and unmeasured characteristics. If they were, the populations could be compared and the propensity-matched unvaccinated population would represent the counterfactual outcomes for the vaccinated population—just as you would expect from a randomized controlled trial.
At first glance, their methods and approach appear reasonable and sound. They generated propensity scores based on geography, sociodemographic characteristics, and the number of SARS-CoV-2 PCR tests an individual had previously had; time-zero for analyses were the same calendar day for matched individuals so that they were under observation during the same time periods. The number of prior COVID-19 tests is meant to be an indication for the baseline risk for COVID-19—presumably, the more potential exposures an individual has had, the more tests they are likely to have gotten. Once matched, participants appear well-matched on the measured characteristics.
However, Figure 2A gives us an indication that participants may not have ended up well-matched on unobserved characteristics that end up conferring different baseline risks of COVID-19. In both RCTs of mRNA COVID-19 vaccinations, the risk for COVID-19 began to diverge only after 10-14 days after vaccination due to the fact that protection is not immediate and immune response takes some time to build up after vaccination. In Figure 2A, Kaplan-Meier curves begin to diverge almost immediately after dose 1 of the first vaccine, which likely indicates that these populations have differing baseline risks. A recent observational study using similar methods to examine the real-world efficacy of vaccines in Israel identified similar early divergence when using minimal matching criteria, but not when they used a more robust propensity-score model that included many more metrics of healthcare utilization (e.g., number of prior flu shots) (DOI: 10.1056/NEJMoa2101765).
This finding raises concerns that propensity-score matching may have only been partially successful in matching the baseline risk of COVID-19; specifically, those who received the vaccine had less baseline risk. Although the initial premise was sound, one could speculate about several reasons that their matching methods were not entirely successful. For example, the reasons for the same number of tests may have been different between vaccinated and unvaccinated individuals (e.g., an abundance of caution vs. lots of COVID-exposure, working in healthcare vs. testing for travel). The vaccinated group does also seem to have had fewer PCR tests. One thing to consider is that the authors appeared to use more robust matching methods for the analyses only among those with a positive COVID-19 test, and those methods may have also been beneficial here.
A secondary point of note is the analysis examining individuals after they developed a positive COVID-19 test. First, they report a very significant p-value but don’t provide confidence intervals and the curves in Figure 4A seem to overlap completely. Second, the vaccine influences the likelihood of hospitalization after infection, but also the probability of developing COVID-19 to begin with. In this analysis, patients are thus selected based on a post-time-zero characteristic that is influenced by whether they have been vaccinated or not. Essentially, the mildest cases may be excluded in the vaccinated group because they remained asymptomatic or it was prevented completely but would be included in the unvaccinated group. The authors state that they included anyone with a positive COVID-19 test (and thus not exclusively symptomatic COVID), but it is unclear whether that is meaningful as I would expect a vast majority of individuals to get tested because of symptoms. It seems that the best way to do it would be to do the same analysis as the main analysis, but with a different outcome variable. The data on hospitalizations after vaccination in and of itself is interesting and useful to know though.
Overall, I think this is a well-done study that provides useful real-world evidence on COVID-19 vaccines. The main point of caution when interpreting their findings and point estimates is that the propensity-score matching was likely not fully successful at generating exchangeable populations and the amount of bias this conferred in the point estimates is difficult to know (I would expect results to be qualitatively similar). This paper could be improved with attempts to develop a more robust propensity score model to strengthen the causal inference.