Review 2: "Development of a Codebook of Online Anti-Vaccination Rhetoric to Manage COVID-19 Vaccine Misinformation"

RR:C19 Evidence Scale rating by reviewer:

• Reliable. The main study claims are generally justified by its methods and data. The results and conclusions are likely to be similar to the hypothetical ideal study. There are some minor caveats or limitations, but they would/do not change the major claims of the study. The study provides sufficient strength of evidence on its own that its main claims should be considered actionable, with some room for future revision.

***************************************

Review:

This is a timely, ambitious, and generally well-executed effort to understand the communication factors associated with vaccine hesitancy. The project is important as policymakers and health communicators seek to strategically manage vaccine hesitancy.

I am impressed with the scope of the project and especially the effort to incorporate a wide range of literature and theoretical concepts in addressing a complex set of attitudes. It is inevitable that when doing so, some of the nuances and details will be lost. There are a couple of places noted below where more detail and context should be added, but more generally, including a general statement about the complexities and history of vaccine hesitance would be helpful. Among other things, I suggest noting that VH is the reflection of a complex constellation of attitudes, varying by culture, age, experience, education as well as the nature, frequency, form, and channels of messages among other factors. This project, then, focuses on one component assumed to influence that constellation of attitudes: messages from online forums. Similarly, it would be helpful to include a general statement about the nature of persuasion – which is a very large and robust area of research that includes examining persuasive appeals, source credibility, frequency, intensity, prior exposure of messages, and existing attitudes, among other factors. These have all been shown to affect opinion formation.

The review of literature on vaccine hesitancy is generally well written, focused, and supports the goals of the project. I don’t agree, however, that most policymakers and risk communicators treat vaccine hesitancy as a uniform belief. Most risk communicators would recognize that this belief structure is highly variable.

I would like to see some reference to the inter-rater reliability in the identification of the codes. Currently, there is only oblique reference to the fact that “Codes were then compared and consolidated.” Typically, IRR is a simple measure of the degree of agreement between coders. It is also helpful to say something about the coders. Did they have training or background in persuasive appeals and narrative analysis?

I find the results quite interesting and would actually like more detailed descriptions of the codes and tropes. While space likely does not allow for this, often examples from the texts are included.

The terms “Narrative Tropes” and “Rhetorical Strategies” are used in the results in ways that do not appear to be conceptually distinct. Can the authors please clarify how these are conceptually distinct?

The discussion section provides some useful suggestions about persuasive strategies for communicating with the vaccine-hesitant, including the use of inoculating messages. The project falls short of detailing specific strategies for tailoring messages to counter the anti-vax messages identified. Doing would likely require empirical investigation and is beyond the scope of the current project. Given that the stated intent is to providing public health communicators and misinformation managers a codebook to help develop counter-messages it might be useful to say explicitly note that more is needed.

A section on limitations is needed addressing the limitations in data sets and interrater reliability.

In general, then, I find the project compelling and useful and recommend publication with these issues addressed.