RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
***************************************
Review:
Motivated by the case of COVID-19 in California, the authors studied vaccination and non-pharmaceutical interventions (NPIs) using a modified version of a previously published age-stratified compartmental model. Individual NPIs such as quarantine, wearing masks, or social distancing, are not represented in explicit categories. Instead, they are handled all together by changing the contact matrix between specific groups. This also includes the adherence to such interventions. The model was initialised to January 1, 2021 and statewide seroprevalence data was used to make the model relevant to California. The authors considered 3 different contact rates (accounting for NPIs), 3 levels of vaccine coverage and 2 ways to prioritize vaccines (either everyone 18+ or only 65+), thus resulting in 3x3x2 = 18 scenarios. Vaccines were assumed to be 50% effective at reducing transmission, 90% effective in averting severe disease ('all-or-nothing'). Results included examining synergies between interventions. Some of the take-away messages for public health in California would be that prioritizing 65+ is the best of two strategies studied. Such conclusions match the work of Bubar et al., of which this piece is an extension: "In almost all circumstances, reducing fatalities required distributing the vaccine to those who are most at risk of death, usually persons over 60 years of age and those with comorbidities. If a vaccine is leaky or poorly efficacious in older adults, then priority could be given to younger age groups." The main limitation of this extended abstract is the lack of heterogeneity obtained from a compartmental model. In some situations, heterogeneity does not particularly matter, as an 'average behavior' (or mean field equation) would result in the same conclusions. But in cases like this, particularly with the conclusions made, heterogeneity is particularly important. When simulations with heterogeneity are used (e.g., agent-based models in which agents of various ages are embedded in different contact networks), then the conclusion can be reversed (18+ outperforms 65+) due to different, not only in the amount of social interactions across ages, but in the structure of these interactions (e.g., school/work networks involve fewer 65+). If the conclusions that we make are so sensitive to the modelling assumptions, then at the very least we should acknowledge the possibility that the conclusion is just an artifact of the approach employed.
Other important limitations include how vaccines work. There are already potential (and significant) benefits after the first dose. Most of the models of COVID-19 that I see currently do include a period of a few weeks from the first to the second dose, and a benefit after the first dose. The 'all-or-nothing' protection and apparent absence of the delay are odd in a current compartmental model. It could easily be remedied by creating more compartments, without having to switch to something as complex as an agent-based model.
Finally, the implementation of non-pharmaceutical interventions is heavily simplified. By lumping them together (plus the level of adherence), it should then be impossible to make any suggestion about specific interventions. For example, it would be difficult to look at the role of adherence separately from physical distancing or mask wearing. I thus find the discussion problematic, because it refers to "moderate level of physical distancing" and "adherence" by themselves. Even the conclusion that we should continue efforts "including masking, social/physical distancing, ventilation, and hang hygiene" can be misinterpreted by the readers: they may think that the model tells us that we need to continue doing all of the above, whereas results tell us that interventions lowering the contact rate (in general) are beneficial (which is obvious from an epidemiological standpoint). I do not disagree with the viewpoint that interventions are necessary during the vaccine rollout -- I merely emphasize that lumping all interventions together is a heavy simplification that should then severely limit how public health messages are constructed.