Review 2: "Preprinting a pandemic: the role of preprints in the COVID-19 pandemic"

RR:C19 Evidence Scale rating by reviewer:

• Potentially informative: The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.

***************************************

Review:

This paper compares COVID-related and non-COVID-related preprints published on bioRxiv and medRxiv. Specifically, they calculate how often preprints are accessed, how the articles are shared on online platforms, and how preprints compare to their published articles. The data clearly demonstrates COVID-19 preprints are accessed, shared on social media, and referenced by journalists more often than non-COVID preprints. The study data, methods, and analysis do support the claim that: “Our results highlight the unprecedented role of preprints and preprint servers in the dissemination of COVID-19 science”. The authors also conclude that there is a “likely long-term impact of the pandemic on the scientific publishing landscape”, although there is not any specific data to support the second claim. The study is potentially informative.

Detailed Review

Introduction. The authors made a compelling case highlighting the gap between traditional publication processes and the urgent needs in disseminating scientific information regarding COVID. Current literature and context is well-described. The objective of this study is clearly stated.

Revision suggestion: Suggest removing the results and conclusion from the introduction(i.e., “We found that preprint servers... in this endeavour.”)

Results. Interesting and extensive analysis of the preprint repositories. It was particularly relevant to consider the issue of social media and misinformation as part of the analysis. The impact of the study could be enhanced by specifically investigating whether COVID-19 preprints address issues of health disparities or diversity, equity, and inclusion regarding study authors, study participants, or media headlines.

Revision suggestion: use sub-headings to organize content and help the readers navigate the many analyses.

Discussion. Thorough discussion. The main study finding is that preprints are playing a newly important role in science communication during the COVID-19 pandemic. This is well-supported by the research and clearly articulated in the discussion. However, the authors should specifically mention: what are the implications or recommendations to improve science communication that flow from this study finding? The paragraph about misinformation on twitter is interesting, but the authors could enhance the utility of the study by linking these results to the discussion about the speed at which new findings are shared, whether they are published, and how the preprint papers change by the time of publication.

Revision Suggestion: Two other study claims in the discussion are not well-supported by the data and methods. First, the discussion says “the pandemic has left what is likely to be a lasting imprint on the preprint and science publishing landscape”. None of the data speaks to changes on the publishing impact outside of the pandemic. They also conclude that rapid dissemination through preprints has not affected “quality of preprints that are subsequently published.” However, this study does not have data that speaks directly to the quality of the evidence.

Methods. The authors should be applauded for the major effort to build the study dataset from multiple information sources in order to comment on many aspects of the preprint lifecycle including eventual publication and mentions in social and traditional media. However, odds ratios are not the appropriate effect estimate to use. The risk / relative risk can be calculated instead and is preferred.

Figures & Tables.

Figure 1D. Would it be possible to align the timeline of panel D with panels A - C? Or enlarge the grey knots for dates.

Figure 2C & 2D. Spell out country name abbreviations in footnote; Or, if the goal is to show clustering by continent (2D), remove the country IDs and enlarge the knots.