Skip to main content
SearchLoginLogin or Signup

Review 2: "Delta variant and mRNA Covid-19 vaccines effectiveness: higher odds of vaccine infection breakthroughs"

This preprint claims that, compared to Alpha, Delta cases have higher odds of vaccine breakthrough infections and lower mean Ct values. Although reviewers agree on the timeliness of this paper, they had concerns with the statistical methods used to get to the conclusions.

Published onSep 27, 2021
Review 2: "Delta variant and mRNA Covid-19 vaccines effectiveness: higher odds of vaccine infection breakthroughs"
1 of 2
key-enterThis Pub is a Review of
Delta variant and mRNA Covid-19 vaccines effectiveness: higher odds of vaccine infection breakthroughs

AbstractBackgroundThe SARS-CoV-2 Delta variant (B.1.617.2), initially identified in India, has become predominant in several countries, including Portugal. Few studies have compared the effectiveness of mRNA vaccines against Delta versus Alpha variant of concern (VOC) and estimated variant-specific viral loads in vaccine infection breakthroughs cases. In the context of Delta dominance, this information is critical to inform decision-makers regarding the planning of restrictions and vaccination roll-out.MethodsWe developed a case-case study to compare mRNA vaccines’ effectiveness against Delta (B.1.617.2) versus Alpha (B.1.1.7) variants. We used RT-PCR positive cases notified to the National Surveillance System between 17th of May and 4th of July 2021 (week 20 to 26) and information about demographics and vaccination status through the electronic vaccination register. Whole-genome sequencing (WGS) or spike (S) gene target failure (SGTF) data were used to classify SARS-CoV-2 variants. The odds of vaccinated individuals to become infected (odds of vaccine infection breakthrough) in Delta cases compared to Alpha SARS-CoV-2 cases was estimated by conditional logistic regression adjusted for age group, sex, and matched by the week of diagnosis. As a surrogate of viral load, mean RT-PCR Ct values were stratified and compared between vaccine status and VOC.ResultsOf the 2 097 SARS-CoV-2 RT-PCR positive cases included in the analysis, 966 (46.1%) were classified with WGS and 1131 (53.9%) with SGTF. Individuals infected with the Delta variant were more frequently vaccinated 162 (12%) than individuals infected with the Alpha variant 38 (5%). We report a statistically significant higher odds of vaccine infection breakthrough for partial (OR=1.70; CI95% 1.18 to 2.47) and complete vaccination (OR=1.96; CI95% 1.22 to 3.14) in the Delta cases when compared to the Alpha cases, suggesting lower mRNA vaccine effectiveness against Delta cases. On our secondary analysis, we observed lower mean Ct values for the Delta VOC cases versus Alpha, regardless the vaccination status. Additionally, the Delta variant cases revealed a Ct-value mean increase of 2.24 (CI95% 0.85 to 3.64) between unvaccinated and fully vaccinated breakthrough cases contrasting with 4.49 (CI95% 2.07 to 6.91) in the Alpha VOC, suggesting a lower impact of vaccine on viral load of Delta cases.ConclusionsWe found significantly higher odds of vaccine infection breakthrough in Delta cases when compared to Alpha cases, suggesting lower effectiveness of the mRNA vaccines in preventing infection with the Delta variant. Additionally, the vaccine breakthrough cases are estimated to be of higher mean Ct values, suggesting higher infectiousness with the Delta variant infection. These findings can help decision-makers weigh on the application or lifting of control measures and adjusting vaccine roll-out depending on the predominance of the Delta variant and the coverage of partial and complete mRNA vaccination.

RR:C19 Evidence Scale rating by reviewer:

  • Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.



This preprint is an important and timely study. The authors collected a dataset on infections with associated data on vaccination. They used a case-case design to assess the odds ratio of being vaccinated given infection with the Delta (versus the Alpha) variant. The authors report a higher probability of vaccination given infection with Delta than with the Alpha variant. The authors interpret this result to indicate that the Delta variant is less susceptible to vaccine-derived immunity, i.e., that the Delta variant may be an immune escape variant. This study is likely to contribute to public health decision-making. It is, therefore, a critical publication priority, and the primary analysis (based on the OR of vaccination among Delta/Alpha infections) is reasonably well-supported. However, I find some critical issues that need addressing before publication can occur. 

Major comments:

The first primary concern is around the author’s reported confidence intervals and statistical significance. In the following instances, the authors report overlapping confidence intervals and state the difference is significant, e.g.,

  • “Additionally, the Delta variant cases revealed a Ct-value mean  increase  of 2.24 (CI95% 0.85 to 3.64) between unvaccinated and fully vaccinated breakthrough cases  contrasting  with 4.49 (CI95% 2.07 to 6.91) in the Alpha VOC, suggesting a lower impact of vaccine on viral load of Delta cases.”

However, since, in this case, the 95% confidence intervals overlap widely, a statistical test is required to determine significance. To my reading, the authors did not report any additional tests to determine significance. Thus, the authors may need a more careful statistical analysis to support the authors’ claims. Since this study is such an important and timely issue, I urge the authors to revisit the study, perhaps recruiting more substantial or more detailed statistical analytics or better reporting of statistical methodology.

Relatedly, in the first paragraph of the discussion, the authors write:

  • “Delta breakthrough cases have a higher viral load (lower Ct values)
    compared Alpha breakthrough cases.” 

However, since the authors do not report a test for significance and judging by the mean and SD in Table 3, this difference is not statistically significant. Without this significance, it appears inappropriate to claim such a difference without indicating that statistical confidence is lacking. I recommend that the authors make this clear.

In the discussion section, the authors write,

  • “Our results are robust to variations in the sampling strategy, including changing the weeks of diagnosis included, and the potential misclassification of using SGTF to identify Delta Variant.”

That is not the case, as the authors write in the sensitivity analysis section of the results the following: “Restricting the analysis to the cases identified through WGS, we observed a drop in the adjusted OR point estimate of complete vaccination (OR=1.48; CI95% 0.75 to 2.93) with a loss of statistical significance…” The discussion must clarify this point.

Later, the authors write:

  • “On our secondary analysis, we observed lower Ct values — indicative of higher viral loads — among Delta cases compared to Alpha after both complete (MD=-2.24; CI95% -4.8 to 0.32) and partial vaccination (MD=–1.88; CI95% -3.77 to 0.003).”

The authors suffer from the same problem. Since the 95% CI crosses 0, we cannot have confidence that the difference in Ct values the authors describe is reliable. Therefore, I recommend that the authors make this distinction very clear. Again, a more careful statistical treatment of these data may yield more substantial support for the authors’ conclusion.

The discussion (for example, lines 345-355) and abstract (for example, lines 100-101) need to be edited to reflect changes made in response to the above comments.

The authors conduct a sensitivity analysis to understand the impact of reporting; however, they fail to account for the most apparent potential confounder: waning immunity. With increasing numbers of studies reporting that vaccine-derived immunity wanes over time, it is ideal if the authors could look at the time since vaccination. Nevertheless, I understand that data may not be available, so a prominent discussion of the impacts of waning immunity on their primary result is warranted. 

Additionally, the authors calculate the odds of breakthrough infection and discuss how this is impacted by vaccine effectiveness; I wonder why the authors did not calculate VE directly for Alpha and Delta? (using these methods:

The study's second objective is to compare a proxy for infectiousness (Ct values) between variants. The authors interpret the Ct value data in the results and discussion concerning the first objective, evaluating Delta-specific vaccine effectiveness through the mean difference between unvaccinated and vaccinated Ct values. I recommend that the Ct value results be analyzed, described, and interpreted following this stated objective to compare infectiousness between Delta and Alpha variants, rather than providing supplemental support for the first objective. Alternatively, this contribution could be described along with the other objectives early in the manuscript. 

Lastly, I am calculating the crude odds ratio of being fully vaccinated given infection with delta, following methods in the supplement, as (162*517)/(28*777) = 2.84, not 1.96. I’m unsure why this discrepancy exists. If the authors used a different method to arrive at the crude OR, I recommend that they specify that in the methods.

Minor comments:

  1. Throughout the manuscript, Ct values are interpreted as a measure of viral load, which is interpreted as an indicator of infectiousness. The authors should thoroughly support this interpretation with references since all researchers may not accept such an interpretation. 

  2. The supplemental methods seem better suited for the main text methods, contingent on the editor's approval.

  3. Please add “B.1.1.7” after the first instance of Alpha in the abstract/background.

  4. The authors have many instances of “cases” where they may intend “instance” or “situation.” This may be confusing to some readers.

  5. Line 93/94 – add “of” so that it reads “regardless of the analysis”

  6. Line 349 – “discrete” does not seem like the appropriate word choice here. “Lower” may be more appropriate.

  7. On line 117 of the introduction, the authors indicate a 28-day dose interval in Portugal, whereas, in the discussion, they indicate a 14-day interval in Portugal. Please remedy this discrepancy.

  8. I recommend that lines 248-249 be rephrased for clarity.

  9. Figure 1 caption should describe the features of the plots (median, quartiles, outliers, etc.).

  10. Line 300 – typo: should be a singular vaccine, not plural vaccines

  11. Lines 326-331 are a bit awkwardly written – I recommend revising for clarity and consistency.

  12. Line 342 – I suggest editing for clarity.

  13. Line 376 – typo: “maybe” should be “may be”

  14. Another valuable component would be displaying the data in Figure 1 as mean +/- 95% CIs rather than boxplots. That method would allow the reader a rapid statistical assessment of the difference between the means. 

No comments here
Why not start the discussion?