Review 3: "The infection fatality rate of COVID-19 inferred from seroprevalence data"

RR:C19 Evidence Scale rating by reviewer:

• Misleading. Serious flaws and errors in the methods and data render the study conclusions misinformative. The results and conclusions of the ideal study are at least as likely to conclude the opposite of its results and conclusions than agree. Decision-makers should not consider this evidence in any decision.

***************************************

Review of Ioannidis, “The infection fatality rate of COVID-19 inferred from seroprevalence data”

This paper:

• addresses an important topic, the estimation of the infection-to-fatality ratio (IFR) of COVID-19;

• reports results of a systematic review that identified 50 rows of data on the seroprevalence of COVID-19 in different locations at different times;

• links the serosurveys to COVID deaths counts and uses these as numerators and estimated infection counts as denominators to estimate the IFR for each location/time;

• presents these IFRs as evidence in support of the author's hypothesis (1) that COVID-19 is not as dangerous as people might believe and that non-pharmaceutical interventions (NPIs) like state-mandated social distancing orders are not warranted.

In the production of scientific knowledge, the systematic review and meta-analysis is often considered the most reliable method, and described as the top of the pyramid of evidence-based medicine.(2) Conducting such a study is not easy, and efforts to do so are valuable and should be encouraged. Unfortunately, the method used here to estimate IFR is flawed due to systematic bias introduced by internal migration (as I shall describe in detail below). This bias produces misleading conclusions when used to argue that IFR is lower in locales with lower-than-average cumulative death rates. The discussion section is therefore also fatally flawed. Even if the sources of bias in the IFR estimates were somehow addressed, the author's thesis, that NPIs are unneeded, ignores the emerging evidence of the long-term non-fatal health burden caused by COVID-19.

This systematic review has been useful in identifying 50 rows of data on population seroprevalence, which is more than previous work (3) (but less than subsequent (4)), but because of its flawed methods and misleading conclusions, I must recommend rejecting it as unsalvageable.

Insurmountable methodological flaws

The method of IFR estimation employed in this paper uses an estimated infection count derived from a serosurveys together with the cumulative death count a week after the survey to approximate the IFR with the formula

There are two major challenges in this approach: (1) misclassification bias and (2) migration bias. Both are most damaging when the cumulative infection count is low, which makes the estimates of IFR in locations with low COVID-19 burden differentially less accurate than the locations with high burden.

Like the author, I think of the IFR as a parameter about an individual: if a person selected uniformly at random is infected with the SARS-CoV-2 virus, what is the probability that they die from COVID-19? If we assume that all members of the population are equally likely to get infected, then this can be approximated accurately by the ratio of the cumulative death count to the cumulative infection count (however, as society learns who is most at risk of severe disease and acts to reduce infections in these vulnerable populations, this approximation will become less accurate).

Although not stated directly, an implicit assumption of the IFR estimation method in this paper is that

This assumption may fail for at least two reasons. First, as acknowledged by the author, seroprevalence testing is not perfect, and even if a test has specificity of 99%, it will still yield a non-negligible number of false positives in locations with low cumulative infections. It is possible for serosurvey papers to adjust for the sensitivity and specificity of the test statistically, but, as the author acknowledged, the evidence base for such adjustment is still emerging, which means that adjusting for the sensitivity of tests on individuals who have had asymptomatic or mildly symptomatic infections more of an art than a science at this point.

There is a second, and perhaps less familiar, reason that the seroprevalence rate might be very different than the cumulative infection rate: out-migration from COVID hotspots. Even with a perfectly accurate test, the seroprevalence count for a population is not necessarily close to the cumulative infection count, because of population change. Rather, 

$seroprevalence count \approx cumulativeinfection count-seropositive deaths+net migration of seropositives.$

If a substantial number of seropositive individuals have relocated from a COVID hotspot to a location with low seroprevalence, ignoring the net migration term will produce a substantial overestimate of cumulative infections. A hypothetical example (inspired by data from a New York Times analysis from May (5)) positing that 10% of residents of New York City have migrated out (including individuals who are seropositive and seronegative) and 1% of them relocated to Vermont illustrates this point:

The inaccuracy of estimating the cumulative infection count with seroprevalence count in NYC is not substantial: seroprevalence count = 6,000,000 – 30,000 – 600,000 = 5,370,000. The total error in a serosurvey here would likely be dominated by the sampling strategy.

In Vermont, on the other hand, an influx of seropositive people fleeing New York, even if it is only 1% of the total population leaving New York, would bias the seroprevalence count to be nearly 50% higher than the cumulative infection count: seroprevalence count = 12,000 - 60 + 6,000 = 17,940.  

Although it is possible to identify recent in-migrants during data collection in a serosurvey, it appears that this has not been done in the surveys identified in this systematic review. Without including some protection for migration bias, IFR estimates from locations with low seroprevalence must be considered inaccurate.

In conclusion

This important work must be rejected as unsalvageable. Future data collection in serosurveys should ask participants about their recent migration history, to permit developing more precise estimates of cumulative infection count.

In addition to the major concerns described above, I must comment that this paper does not follow the norms of systematic review. For example, it does not include a PRISMA checklist, it does not include a flow diagram, and the description of the search strategy appears to be incomplete and insufficient for replication.(6)

References

1. An Elite Group Of Scientists Tried To Warn Trump Against Lockdowns In March [Internet]. BuzzFeed News. [cited 2020 Aug 28]. Available from: https://www.buzzfeednews.com/article/stephaniemlee/ioannidis-trump-white-house-coronavirus-lockdowns

2. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. BMJ Evid-Based Med. 2016 Aug 1;21(4):125–7. 

3. A systematic review and meta-analysis of published research data on COVID-19 infection-fatality rates | medRxiv [Internet]. [cited 2020 Aug 28]. Available from: https://www.medrxiv.org/content/10.1101/2020.05.03.20089854v4

4. Arora RK, Joseph A, Wyk JV, Rocco S, Atmaja A, May E, et al. SeroTracker: a global SARS-CoV-2 seroprevalence dashboard. Lancet Infect Dis [Internet]. 2020 Aug 4 [cited 2020 Aug 28];0(0). Available from: https://www.thelancet.com/journals/laninf/article/PIIS1473-3099(20)30631-9/abstract

5. Quealy K. The Richest Neighborhoods Emptied Out Most as Coronavirus Hit New York City. The New York Times [Internet]. 2020 May 15 [cited 2020 Aug 28]; Available from: https://www.nytimes.com/interactive/2020/05/15/upshot/who-left-new-york-coronavirus.html

6. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009 Jul 21;6(7):e1000097.