Skip to main content
SearchLoginLogin or Signup

Review 1: "Parallel Trends in an Unparalleled Pandemic: Difference-in-Differences for Infectious Disease Policy Evaluation"

Reviewers commend the study for addressing the limitations of standard DiD methods and proposing robust alternatives. They suggest further elaboration on the differences between the new methods and traditional estimates, as well as comparisons to other modeling approaches.

Published onMay 20, 2024
Review 1: "Parallel Trends in an Unparalleled Pandemic: Difference-in-Differences for Infectious Disease Policy Evaluation"
1 of 2
key-enterThis Pub is a Review of
Parallel Trends in an Unparalleled Pandemic Difference-in-differences for infectious disease policy evaluation
Parallel Trends in an Unparalleled Pandemic Difference-in-differences for infectious disease policy evaluation

Researchers frequently employ difference-in-differences (DiD) to study the impact of public health interventions on infectious disease outcomes. DiD assumes that treatment and non-experimental comparison groups would have moved in parallel in expectation, absent the intervention (“parallel trends assumption”). However, the plausibility of parallel trends assumption in the context of infectious disease transmission is not well-understood. Our work bridges this gap by formalizing epidemiological assumptions required for common DiD specifications, positing an underlying Susceptible-Infectious-Recovered (SIR) data-generating process. We demonstrate that popular specifications can encode strict epidemiological assumptions. For example, DiD modeling incident case numbers or rates as outcomes will produce biased treatment effect estimates unless untreated potential outcomes for treatment and comparison groups come from a data-generating process with the same initial infection and equal transmission rates at each time step. Applying a log transformation or modeling log growth allows for different initial infection rates under an “infinite susceptible population” assumption, but invokes conditions on transmission parameters. We then propose alternative DiD specifications based on epidemiological parameters – the effective reproduction number and the effective contact rate – that are both more robust to differences between treatment and comparison groups and can be extended to complex transmission dynamics. With minimal power difference incidence and log incidence models, we recommend a default of the more robust log specification. Our alternative specifications have lower power than incidence or log incidence models, but have higher power than log growth models. We illustrate implications of our work by re-analyzing published studies of COVID-19 mask policies.Significance Statement Difference-in-differences is a popular observational study design for policy evaluation. However, it may not perform well when modeling infectious disease outcomes. Although many COVID-19 DiD studies in the medical literature have used incident case numbers or rates as the outcome variable, we demonstrate that this and other common model specifications may encode strict epidemiological assumptions as a result of non-linear infectious disease transmission. We unpack the assumptions embedded in popular DiD specifications assuming a Susceptible-Infected-Recovered data-generating process and propose more robust alternatives, modeling the effective reproduction number and effective contact rate.

RR:C19 Evidence Scale rating by reviewer:

  • Strong. The main study claims are very well-justified by the data and analytic methods used. There is little room for doubt that the study produced has very similar results and conclusions as compared with the hypothetical ideal study. The study’s main claims should be considered conclusive and actionable without reservation.


Review: In this manuscript, the authors review and evaluate the DID design in the context of infectious disease applications.  Overall, I thought this was an excellent manuscript.  I have a few minor suggestions for improvement.

In this manuscript the authors seek to reconcile the classic DID design, which assumes linear and additive effects with extant infectious disease models which posit functional forms that are incompatible with standard DID methods.  They propose some key changes which helps reconcile the two approaches.  Overall, I found this paper of the paper to be quite compelling.  There is little doubt that standard DID methods are not well suited to the models of disease spread. The proposed methods appear to be a notable improvement over standard DID methods.

I do have one suggestion related to the re-analyses.  First for the replication of the MA masking study.  It appears from my reading that the alternative methods produce substantively different results. While in some cases the new estimates have the same sign and are statistically significant the magnitude of this estimate is much smaller. I think it would be useful to elaborate more on why there are such large differences between these different sets of estimates.  Be clearer about what specifically is being assumed by the incidence specification. Also in Table 2, I think it would be helpful to better label which estimates are follow the original specification. The same is true for the Kansas application as well, though the differences there are not as large between methods.

Overall, however, I thought there as much to like in this manuscript and I think with these minor revisions it is easily publishable.

No comments here
Why not start the discussion?