RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
***************************************
Review:
The issue of whether (and how much) re-opening of higher education institutions for in-person teaching contributes to the spread of COVID-19 is an open question, and subject of ongoing debates worldwide[1] . The main concern is that young people may contribute to the spread of COVID-19, both by importing cases to the community where their college is located, and spreading the virus among themselves and the community, through social interactions. This study attempts to evaluate the impact of recent college reopening across the USA on movement on campus, COVID-19 incidence and the daily reproduction number R(t). The authors use an innovative dataset, relying on mobile phone signals to capture movements.
The study exploits the standard toolkit economists turn to when aiming to ascertain the causal effects of polices: the differences in differences approach, and its extension, the approach of “event studies”. Both approaches exploit variation in an exposure or policy (here: reopening of colleges), both across units (here: colleges and counties) and over time, each unit providing data points for before and after the event of reopening. Hence, counties that don’t have colleges, or have colleges that have not re-opened provide a control group in a hypothetical policy experiment. The central idea is that this strategy can control for two main sources of confounding: first, of common trends that would bias a simple before-after comparison (e.g. the overall dynamics of the disease in this time period, nationwide COVID-19 policies), and second, the potentially unobserved differences between counties that have colleges and those that don’t have colleges (e.g. the composition of the local population in terms of demographics and socioeconomics).
The central assumption of the approaches is that the only differential change in the study period was the policy under examination, implying that the outcomes in colleges which opened and which did not open would have evolved parallel, in the absence of opening. If this assumption holds, the associations reported by the authors can be interpreted as the causal effects of reopening colleges. While the authors are careful in their choice of language to not talk about causality (and use the term association instead), an implicit causal interpretation of their estimates is present in the conclusions (e.g. ”Our second main finding is that re-opening a college with in-person instruction significantly increases the rate of new daily cases in the county”). It is expected that policy makers wishing to use the presented evidence, may interpret the results causally, hence the aforementioned assumptions must be carefully scrutinised.
In the current version of the article, the authors provide no substantial discussion of these assumptions, which may make it hard for readers without the economics background to appraise the methods, and interpret the findings. Figure 2 does contain information that can provide some reassurance: the outcomes (transformed by subtracting the baseline) have followed a parallel path, and were close to zero, in the relevant time period, 10 days or more before the opening. Another reassuring factor can be the relatively short length of the study period, making it unlikely that anything else relevant (e.g. local policy changes) happened. What cannot be so easily read off the figures, is the possibility of college openings being influenced by observed baseline outcomes (such as high R or high incidence rate. Would this bias the results, and if yes, in which direction?
On the more technical side, there are a few more points which would benefit from further clarification. How are the control groups exactly utilised in the regressions? In the event study regressions, for each county-day observation, a series of indicator variables denote whether on a given day this college was -14,-13,… +13,+14 days away from the opening. The authors indicate that a counterfactual opening time is generated for those counties where a college was not yet open or did not have a college. It would be good to hear more about this approach. Are there alternatives available, and perhaps preferable, for example by matching similar control colleges to each college with an opening?
The authors conclude that the effects of college openings significantly affect the outcomes of interest, and the effects are stronger for colleges that do in person teaching than those doing only teaching only. The graphical results from the event study give a very clear picture (and are easier to interpret than the regression results): before the opening, the effects were essentially zero. The effects in terms of mobility and the reproduction number started appearing around 10 days before opening, and new cases increased as a result of openings at around a week after opening.
I was surprised to see that while the impact in terms of new cases was substantially higher for in person teaching, the impacts in terms of the reproduction ratio we in fact fairly close for these two teaching modalities. This perhaps merits some further discussion.
The authors are honest about the limitations of the research, in terms of not being able to consider the heterogeneity of COVID-19 policies at the college level, explicitly differentiating between imported cases and community transmission, or capturing spillovers to the local community, mainly due to data limitations. In addition, I wondered whether the data at hand would account for colleges actually discontinuing in person teaching after an outbreak[3] happened.
Overall, the paper provides suggestive evidence on the impact of college openings on COVID-19 transmission. The main strength of the paper is its empirical strategy, exploiting spatial and time variation on college openings. The range of sensitivity analyses conducted strengthens the results. However, given that the crucial challenge of any evaluation of COVID-19 policies is that it’s hard to figure out what is the true impact of a policy and what is natural variation that one cannot attribute to a given policy[2], it would be crucial that in the published version of the paper the authors carefully discuss the underlying assumptions of the methodological approaches employed.
[1] https://www.bmj.com/content/370/bmj.m3365
[2] https://coronavirus.jhu.edu/from-our-experts/evaluating-the-effectiveness-of-covid-19-policies-a-q-and-a-with-dr-elizabeth-stuart
[3] https://www.bmj.com/content/370/bmj.m3365
Creative Commons Attribution 4.0 International License (CC-BY 4.0)
