Review 3: "Estimating The Uncertain Effect of the COVID Pandemic on Drug Overdoses"

In this study, the authors use pre-pandemic data to develop a predictive model for drug-related overdose deaths and emergency department visits, and then leverage this model to estimate whether the first year of the COVID-19 pandemic was associated with changes in these outcomes. The authors’ stated intentions were to address the methodological deficiencies of previous studies, specifically the lack of controlling for pandemic pre-trends, and either refute or confirm media anecdotes about a surge in opioid overdoses starting in March 2020. While this goal is laudable, I have serious concerns regarding novelty of this work, and I believe the methods used are inappropriate. Thus, the study should not be used as evidence by policymakers and fellow researchers in its current state.

Conceptually, the authors assume that a rise in drug overdose deaths during the pandemic should mechanistically coincide with a rise in drug-related emergency department visits. There is no discussion of how millions of Americans reported they avoided care for various reasons during the pandemic’s early phases, even for serious indications, and thus trends in rates of mortality and health services utilization may plausibly trend in opposite directions.

The authors cite several published studies that examine changes in overdose deaths associated with the onset of the COVID-19 pandemic, and discuss methodological weaknesses of each. Many of these criticisms are valid. In several places throughout the manuscript, they state there is only one existing study that attempts to model counterfactual mortality (Faust et al., 2021), and that study produces “unreasonable predictions.” It is unclear why their predictions are unreasonable; Faust et al. used seasonal autoregressive integrated moving average models to estimate an observed-versus-expected ratio of 1.28, equating to 10,443 excess deaths during March-August 2020. The literature review also omits a key citation by Griffith & Auty (2022) in Drug & Alcohol Dependence, who used pre-pandemic data from CDC WONDER to identify changes in drug and opioid overdose mortality during 2022. They estimated that drug overdose deaths were 29.9% higher in 2020, and overdose deaths specifically attributed to opioids were 37.6% higher. Thus, the authors may need to rethink what methodological or knowledge gaps the present study is designed to address and clarify its novel contributions.

The authors do not provide many details on their modeling approach, but what detail is provided is concerning. The use of a rolling 12-month overdose mortality measure will bias any estimate of effect downwards during the early phase of the pandemic, since it is contaminated with pre-pandemic data. The data from a “national emergency physician staffing group” needs to more detail, and a discussion of how this may be non-representative of the United States as a whole. My greatest concern is the choice of a fourth order polynomial model. The use of higher-order polynomials in predictive algorithms is generally frowned upon unless there is a clear theoretical basis, as their use may lead to estimates that are highly sensitive to model specification and thus raise suspicions of data mining. The reader may directly observe why this is the case in Figure 1, where the models’ predictions swing wildly upwards without a discernible reason during the pandemic period, despite never observing changes of similar magnitude during the pre-pandemic period on which the model was trained. Readers may plainly view the acceleration of drug overdose mortality during the pandemic in the raw data, which we are then asked to disbelieve. The exact model specification, including all covariates and how they are measured, should be provided at least in an appendix not provided, raising the potential for data mining and bias.

Taken together, I think these issues are serious enough to consider the manuscript as potentially misleading and non-informative in its current state, and I hope they are addressed satisfactorily in future versions.