RR\ID Peer Review Guidelines and Desired Content

I. Instructions for Peer Reviewers

RR\ID asks its peer reviewers to assess a manuscript with the same level of rigor as you would for a top journal, but we ask less work of you.  We are interested in your assessment of how well the paper’s conclusions are substantiated by the research that the authors have done. 

We ask you to grade that assessment using the RR\ID Strength of Evidence Scale and then to comment to defend your assessment with a written review.  Our reviews are typically around 500 words. We would like the focus to be on whether the overall conclusions and main claims are substantiated by the evidence presented.

While other journals might ask you to detail what modifications/enhancements the authors would need to do to make the paper of “publishable quality”, we do not.  You are welcome to do that--and both the authors and any journals that later consider publishing the paper will be grateful to you--but RR\ID is more than satisfied if you only comment on the degree to which the manuscript in its current form supports the results/conclusions that it presents.  Editorial suggestions (i.e., grammar, style, structure) are much less important for this review. We anticipate that most of the meritorious preprints will eventually be published through a formal editing process with elite journals.

II. RR\ID Strength of Evidence Scale¹

We ask reviewers to consider the following options for strength of evidence in the preprint and to choose one of the following that best represents their opinion of the manuscript. 

• Strong: The main study claims are very well-justified by the data and analytic methods used. There is little room for doubt that the study produced has very similar results and conclusions as compared with the hypothetical ideal study. The study’s main claims should be considered conclusive and actionable without reservation.

• Reliable: The main study claims are generally justified by its methods and data. The results and conclusions are likely to be similar to the hypothetical ideal study. There are some minor caveats or limitations, but they would/do not change the major claims of the study. The study provides sufficient strength of evidence on its own that its main claims should be considered actionable, with some room for future revision.

• Potentially informative: The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.

• Not informative: The flaws in the data and methods in this study are sufficiently serious that they do not substantially justify the claims made. It is not possible to say whether the results and conclusions would match that of the hypothetical ideal study. The study should not be considered as evidence by decision-makers.

• Misleading: Serious flaws and errors in the methods and data render the study conclusions misinformative. The results and conclusions of the ideal study are at least as likely to conclude the opposite of its results and conclusions than agree. Decision-makers should not consider this evidence in any decision.

[1] The Strength of Evidence Scale was developed by Noah A. Haber, ScD (Postdoctoral Fellow, Meta-Research Innovation Center, Stanford University) and by Emily R. Smith, ScD (Assistant Professor, Department of Global Health, Milken Institute School of Public Health, GWU).

III. Other areas to consider in your review:

We ask that reviewers consider a few key evaluation questions.

• Does the manuscript confirm previous work or refute the current understanding? Do the findings contribute to broader research understandings? Can the evidence and arguments presented support advancement of ID understanding within society? The degree of novelty can be considered here but is not the main driver of this indicator.

• How well does the manuscript position the work within the current literature/understanding? Does the manuscript cite current literature and discuss limitations? Is it steeped in reality with the potential to impact the implementation of policy and programs?

• Is there clarity regarding the recommended actions that result from the findings? Is the work clearly and accurately presented. That is, is it well-structured and well-written, with an ability to speak to key audiences?

• Do authors pay attention to ethics, diversity, and inclusion? Have the authors adequately discussed ethical concerns? When appropriate, have they been inclusive and taken into account equity, rights, and diversity?