RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
First, let me congratulate the authors on identifying an important and relevant research topic, which is measuring the efficacy of COVID vaccines in practice, both on preventing infection and reducing the severity of infection. The author uses vaccination data from multiple states of the US. They used propensity score matching to construct an unvaccinated group that is similar to the treatment group, and they determined the efficacy of the vaccines by comparing the outcome of the two groups and determining that the results are significant and statistically similar to that of the trials of the vaccine providers. In addition, the authors showed that vaccination is helpful in preventing serious outcomes and the efficacy is different at different times after the first dose.
My major comment is regarding the propensity score matching procedure in the paper. From the description (for example, first paragraph of page 4) in the manuscript, it seems that the authors are identifying a propensity of P(infection=1|other observed covariates) instead of P(vaccination=1|other observed covariates). The description in the paper is kind of misleading to me, so the authors may need to further clarify it. But if the authors are really estimating a propensity score of P(infection=1|observed covariates), then this step is not correct, and they may have to redo the estimation from the very beginning.
My second comment is regarding the data efficiency of the study. The dataset the authors use is a very large and rich dataset, the problem is that ratio of vaccinated people is low in the dataset, not to mention the ratio of vaccinated people who is infected. And the main model that the authors estimated uses only a small proportion of the dataset. I have two suggestions for this. First, the dataset is up to February 2021, and many vaccine doses have been issued during the past month, so is it possible that the authors obtain the most recent data in which more people are getting vaccinated. Second, the authors may consider methods like inverse probability weighting (IPW) or even the X-Learner (Sören R. Künzel, 2019) to deal with the data inefficiency issue. Personally, I would expect the confidence interval of the efficacy estimate to shrink, thus it is more informative.
There are several other comments from my side. First, it is interesting to compare the effect of the two vaccine variants from Moderna and Pfizer. Second, I like the fact that the authors studied the efficacy over multiple time intervals after injection of the first vaccine dose. However, I did not find a place where the author removed the observation which does not have enough follow-up time after vaccination. Maybe the author should make this clear. Third, the efficacy of vaccines over time can possibly be explained by two causes, one is more and more people get vaccinated so the risk of exposure to the virus is declining, the other one is the vaccine is indeed becoming more effective to the injected individual over time. Is there any possible way to distinguish these two causes?
Sören R. Künzel, J. S. (2019). Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the National Academy of Sciences of the United States of America, 4156-4165.