This study's claims on the effects of lockdowns and case-counts on food insecurity in Nigeria are undermined by significant methodological and statistical concerns, as well as questions regarding the comparability of state-level case counts used in the analysis.
RR: C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable unless the weaknesses are clearly understood and there are other theories and evidence to further support it.
The paper leverages the existing World Bank LSMS ISA survey to assess the effects of lockdowns and case counts on household food security in Nigeria. Special attention is paid to the mechanisms of estimated changes in food security status; the authors try and untangle effects on food security that may be due to changes in household labor market participation from those attributable to changes in food prices. The authors have a pre-COVID round of LSMS data that they build on with a phone survey. This gives them a nice pre-COVID status of households and the nationally representative nature of the sample distinguishes the study.
A challenge with this paper has to do with the “treatment”. The authors use both COVID lockdowns and case counts from the end of May 2020. The case count data is difficult to interpret. In particular, it is not clear how to interpret the variation in the case numbers across the states. Do they reflect differences in the extent of the pandemic? Or differences in testing per capita? Or something else? How many tests were being done at the time? What variability existed in the testing rates and the positivity rates across Nigeria at the time? What was the per capita testing rate? The testing data comes from the Nigerian Centre for Disease Control – is that comprehensive? Were private tests being done in parallel or is this an aggregation? What was the temporal variation in the cases within states? Were some states increasing their case numbers during this time and some declining? The authors argue that the case counts tell us something about the extent of the pandemic in that state but the reader needs to know much more about testing and positivity rates to understand what is being studied here.
It may be that the public numbers of the cases act as a signal to individuals within states and those signals lead to behavioral changes – Goolsbee and Syverson (2020) make a version of this argument. But I don’t think that’s what the authors are after.
Sticking to the lockdown indicator seems like a safer territory here in terms of interpretation, though I suspect variation in what a lockdown means and the extent to which it was enforced. Again, more detail here is necessary to understand what is going on. Interacting these treatment variables just leads to more confusion given that the underlying variables are not well described.
Given that multiple rounds of the LSMS exist for these households, can the authors use more rounds to get a pre-COVID trend for these households or to try and characterize or sort out seasonal fluctuations? The differences they document (Table 1) do seem very substantial. Perhaps zooming in on urban or rural households would help tighten the argument. The household data is exciting and the geographic scope distinguishes the data.