RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
***************************************
Review:
The main question the author poses is what to do if vaccines are no longer as effective against emerging viral variants. They use a simple SIR-type model, implemented stochastically (SSA) to address this question. The main result suggests that it is better to target a faster growing variant, rather than prioritizing based on current prevalence. This, to me, amounts to a fairly intuitive result about exponential growth. In vague analogy, if one is in debt, typical advice is to pay off accounts with higher interest rates first, rather than accounts with higher absolute amounts.
Several of the model variables are changed and the overall message remains constant. However, this exploration is very cursory. While currently very clean, I would want to see some additional analysis before accepting that main message. I have to say too, the utility here ("rules of thumb") may not really be what's needed in reality, but this work has merit in bringing up the point and highlighting the need for work on future variant vaccines. The main criticism is likely to be expected from the author, if one explores a set of scenarios in which the result is true, that does not guarantee that other unexplored scenarios admit the same result, a broader and more quantitative sensitivity analysis would do a lot to help with this criticism, though it is fundamentally impossible to explore all scenarios. It would help to say not "almost always better to target the faster spreading strain", but something like "in 50/60 scenarios with such and such assumptions it was better to target the faster..."
The work has other merits, it appears the analysis was performed well, it is especially clearly written and honest, the code shared is clean and readable, all of which I think does a great service to this paper and modelling as a field.
You address some limitations in the discussion, but I think others are actually more important. Our own group has published models of vaccination with many of these features, see (https://www.medrxiv.org/content/10.1101/2020.12.13.20248120v1.full.pdf+html for example), and somewhat unfortunately, the nuances continue to matter. The message would be substantially enhanced if shown robust to any (or all!) of the following considerations:
- What if the vaccine does not block infection, but simply reduces disease? This is unfortunately not ruled out based on how current trials were performed, secondarily, the whole story could be more complicated if asymptomatic individuals are included at all.
- You mention age structure briefly, but given the obvious functional relationship between age and fatality, I'm not immediately convinced the result would sufficiently hold in a more complex system.
- What if vaccination rate is much slower (likely), or faster (unlikely, but worth knowing)?
- What if immunity wanes?
Additionally, because there is no illustration of what the model predicts the epidemic will look like, it cannot be compared to any real data, it remains also unclear whether the results would be robust to a shifting pandemic situation outside of vaccination. For example, social distancing and mask wearing have a huge impact in our modelling, and most governments are unlikely to let an epidemic continue exponential growth without implementing social control measures -- this remains a weakness of the present work that must be discussed.
In the abstract you write: "meaning that the choice of vaccination strategy has a small effect on public health outcomes". This must be clarified to suggest that the choice of vaccination strategy in terms of multiple monovalent vaccines is unimportant. Some features of vaccination strategy have a huge impact on public health.
A final criticism involves the choice of SSA. First, it seems somewhat misplaced to make the model more complicated (and take longer to run) by making it stochastic, rather than more complicated in structure. Since the model isn't really concerned with burn-out of variants, a deterministic model (perhaps with a cutoff frequency to represent burnout) might be better suited to rapidly explore complicated scenarios that are possible. Second, since you show medians of 30 simulations (which seems too few), you should at least show the ranges, or individual simulations, such that the stochastic variability is visualized (and can be judged if useful).