RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
***************************************
Review:
This preprint is an important and timely study. The authors collected a dataset on infections with associated data on vaccination. They used a case-case design to assess the odds ratio of being vaccinated given infection with the Delta (versus the Alpha) variant. The authors report a higher probability of vaccination given infection with Delta than with the Alpha variant. The authors interpret this result to indicate that the Delta variant is less susceptible to vaccine-derived immunity, i.e., that the Delta variant may be an immune escape variant. This study is likely to contribute to public health decision-making. It is, therefore, a critical publication priority, and the primary analysis (based on the OR of vaccination among Delta/Alpha infections) is reasonably well-supported. However, I find some critical issues that need addressing before publication can occur.
Major comments:
The first primary concern is around the author’s reported confidence intervals and statistical significance. In the following instances, the authors report overlapping confidence intervals and state the difference is significant, e.g.,
“Additionally, the Delta variant cases revealed a Ct-value mean increase of 2.24 (CI95% 0.85 to 3.64) between unvaccinated and fully vaccinated breakthrough cases contrasting with 4.49 (CI95% 2.07 to 6.91) in the Alpha VOC, suggesting a lower impact of vaccine on viral load of Delta cases.”
However, since, in this case, the 95% confidence intervals overlap widely, a statistical test is required to determine significance. To my reading, the authors did not report any additional tests to determine significance. Thus, the authors may need a more careful statistical analysis to support the authors’ claims. Since this study is such an important and timely issue, I urge the authors to revisit the study, perhaps recruiting more substantial or more detailed statistical analytics or better reporting of statistical methodology.
Relatedly, in the first paragraph of the discussion, the authors write:
However, since the authors do not report a test for significance and judging by the mean and SD in Table 3, this difference is not statistically significant. Without this significance, it appears inappropriate to claim such a difference without indicating that statistical confidence is lacking. I recommend that the authors make this clear.
In the discussion section, the authors write,
That is not the case, as the authors write in the sensitivity analysis section of the results the following: “Restricting the analysis to the cases identified through WGS, we observed a drop in the adjusted OR point estimate of complete vaccination (OR=1.48; CI95% 0.75 to 2.93) with a loss of statistical significance…” The discussion must clarify this point.
Later, the authors write:
The authors suffer from the same problem. Since the 95% CI crosses 0, we cannot have confidence that the difference in Ct values the authors describe is reliable. Therefore, I recommend that the authors make this distinction very clear. Again, a more careful statistical treatment of these data may yield more substantial support for the authors’ conclusion.
The discussion (for example, lines 345-355) and abstract (for example, lines 100-101) need to be edited to reflect changes made in response to the above comments.
The authors conduct a sensitivity analysis to understand the impact of reporting; however, they fail to account for the most apparent potential confounder: waning immunity. With increasing numbers of studies reporting that vaccine-derived immunity wanes over time, it is ideal if the authors could look at the time since vaccination. Nevertheless, I understand that data may not be available, so a prominent discussion of the impacts of waning immunity on their primary result is warranted.
Additionally, the authors calculate the odds of breakthrough infection and discuss how this is impacted by vaccine effectiveness; I wonder why the authors did not calculate VE directly for Alpha and Delta? (using these methods: https://pubmed.ncbi.nlm.nih.gov/3879673/)
The study's second objective is to compare a proxy for infectiousness (Ct values) between variants. The authors interpret the Ct value data in the results and discussion concerning the first objective, evaluating Delta-specific vaccine effectiveness through the mean difference between unvaccinated and vaccinated Ct values. I recommend that the Ct value results be analyzed, described, and interpreted following this stated objective to compare infectiousness between Delta and Alpha variants, rather than providing supplemental support for the first objective. Alternatively, this contribution could be described along with the other objectives early in the manuscript.
Lastly, I am calculating the crude odds ratio of being fully vaccinated given infection with delta, following methods in the supplement, as (162*517)/(28*777) = 2.84, not 1.96. I’m unsure why this discrepancy exists. If the authors used a different method to arrive at the crude OR, I recommend that they specify that in the methods.
Minor comments:
Throughout the manuscript, Ct values are interpreted as a measure of viral load, which is interpreted as an indicator of infectiousness. The authors should thoroughly support this interpretation with references since all researchers may not accept such an interpretation.
The supplemental methods seem better suited for the main text methods, contingent on the editor's approval.
Please add “B.1.1.7” after the first instance of Alpha in the abstract/background.
The authors have many instances of “cases” where they may intend “instance” or “situation.” This may be confusing to some readers.
Line 93/94 – add “of” so that it reads “regardless of the analysis”
Line 349 – “discrete” does not seem like the appropriate word choice here. “Lower” may be more appropriate.
On line 117 of the introduction, the authors indicate a 28-day dose interval in Portugal, whereas, in the discussion, they indicate a 14-day interval in Portugal. Please remedy this discrepancy.
I recommend that lines 248-249 be rephrased for clarity.
Figure 1 caption should describe the features of the plots (median, quartiles, outliers, etc.).
Line 300 – typo: should be a singular vaccine, not plural vaccines
Lines 326-331 are a bit awkwardly written – I recommend revising for clarity and consistency.
Line 342 – I suggest editing for clarity.
Line 376 – typo: “maybe” should be “may be”
Another valuable component would be displaying the data in Figure 1 as mean +/- 95% CIs rather than boxplots. That method would allow the reader a rapid statistical assessment of the difference between the means.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
