Skip to main content
SearchLoginLogin or Signup

Review 2: "FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system"

This preprint found that two vaccine doses provide 88.7% effectiveness at preventing infection and lowered hospitalizations. The reviewers found the preprint potentially informative, due to the propensity score matching method, and could be improved by additional studies.

Published onApr 26, 2021
Review 2: "FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system"
1 of 2
key-enterThis Pub is a Review of
FDA-authorized COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system

AbstractLarge Phase 3 clinical trials of the two FDA-authorized COVID-19 vaccines, mRNA-1273 (Moderna) and BNT162b2 (Pfizer/BioNTech), have demonstrated efficacies of 94.1% (n = 30,420, 95% CI: 89.3-96.8) and 95% (n = 43,448, 95% CI: 90.3-97.6) in preventing symptomatic COVID-19, respectively. Given the ongoing vaccine rollout to healthcare personnel and residents of long-term care facilities, here we provide a preliminary assessment of real-world vaccination efficacy in 62,138 individuals from the Mayo Clinic and associated health system (Arizona, Florida, Minnesota, Wisconsin) between December 1st 2020 and February 8th 2021. Our retrospective analysis contrasts 31,069 individuals receiving at least one dose of either vaccine with 31,069 unvaccinated individuals who are propensity-matched based on demographics, location (zip code), and number of prior SARS-CoV-2 PCR tests. 8,041 individuals received two doses of a COVID-19 vaccine and were at risk for infection at least 36 days after their first dose. Administration of two COVID-19 vaccine doses was 88.7% effective in preventing SARS-CoV-2 infection (95% CI: 68.4-97.1%) with onset at least 36 days after the first dose. Furthermore, vaccinated patients who were subsequently diagnosed with COVID-19 had significantly lower 14-day hospital admission rates than propensity-matched unvaccinated COVID-19 patients (3.7% vs. 9.2%; Relative Risk: 0.4; p-value: 0.007). Building upon the previous randomized trials of these vaccines, this study demonstrates their real-world effectiveness in reducing the rates of SARS-CoV-2 infection and COVID-19 severity among individuals at highest risk for infection.

RR:C19 Evidence Scale rating by reviewer:

  • Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.



First, let me congratulate the authors on identifying an important and relevant research topic, which is measuring the efficacy of COVID vaccines in practice, both on preventing infection and reducing the severity of infection. The author uses vaccination data from multiple states of the US. They used propensity score matching to construct an unvaccinated group that is similar to the treatment group, and they determined the efficacy of the vaccines by comparing the outcome of the two groups and determining that the results are significant and statistically similar to that of the trials of the vaccine providers. In addition, the authors showed that vaccination is helpful in preventing serious outcomes and the efficacy is different at different times after the first dose.

Main comments:

My major comment is regarding the propensity score matching procedure in the paper. From the description (for example, first paragraph of page 4) in the manuscript, it seems that the authors are identifying a propensity of P(infection=1|other observed covariates) instead of P(vaccination=1|other observed covariates). The description in the paper is kind of misleading to me, so the authors may need to further clarify it. But if the authors are really estimating a propensity score of P(infection=1|observed covariates), then this step is not correct, and they may have to redo the estimation from the very beginning.

My second comment is regarding the data efficiency of the study. The dataset the authors use is a very large and rich dataset, the problem is that ratio of vaccinated people is low in the dataset, not to mention the ratio of vaccinated people who is infected. And the main model that the authors estimated uses only a small proportion of the dataset. I have two suggestions for this. First, the dataset is up to February 2021, and many vaccine doses have been issued during the past month, so is it possible that the authors obtain the most recent data in which more people are getting vaccinated. Second, the authors may consider methods like inverse probability weighting (IPW) or even the X-Learner (Sören R. Künzel, 2019) to deal with the data inefficiency issue. Personally, I would expect the confidence interval of the efficacy estimate to shrink, thus it is more informative.

There are several other comments from my side. First, it is interesting to compare the effect of the two vaccine variants from Moderna and Pfizer. Second, I like the fact that the authors studied the efficacy over multiple time intervals after injection of the first vaccine dose. However, I did not find a place where the author removed the observation which does not have enough follow-up time after vaccination. Maybe the author should make this clear. Third, the efficacy of vaccines over time can possibly be explained by two causes, one is more and more people get vaccinated so the risk of exposure to the virus is declining, the other one is the vaccine is indeed becoming more effective to the injected individual over time. Is there any possible way to distinguish these two causes?

Sören R. Künzel, J. S. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences of the United States of America, 4156-4165.

Renee Chapman:

A few of additional points from my end are as follows: first, it is intriguing to compare the efficacy of the two vaccine versions developed by Pfizer and Moderna. geometry dash wave

Drew Binsky:

he dataset used is large and rich, but the ratio heic to jpg of vaccinated individuals, particularly those who are infected, is low