Review 2: "Information Delivered by a Chatbot Has a Positive Impact on COVID-19 Vaccines Attitudes and Intentions"

RR:C19 Evidence Scale rating by reviewer:

• Reliable. The main study claims are generally justified by its methods and data. The results and conclusions are likely to be similar to the hypothetical ideal study. There are some minor caveats or limitations, but they would/do not change the major claims of the study. The study provides sufficient strength of evidence on its own that its main claims should be considered actionable, with some room for future revision.

***************************************

Review:

Against a backdrop of significant vaccine hesitation with regard to COVID-19 vaccines, it is essential to test approaches that respond to people's concerns about these vaccines. In this randomized, web-based study of 643 people, the authors tested the impact of a conversational robot (chatbot) on attitudes towards these vaccines and on the intention to vaccinate, compared to a control group. The latter was proposed a brief text, taken from an official French site, presenting the characteristics of these vaccines. The chatbot offered the participants in the intervention group a choice of six main questions, which were set in advance, such as "Are COVID-19 vaccines safe? It provided answers based on available scientific knowledge and allowed participants to choose again from new pre-determined questions. A total of 51 questions were prepared with their answers. Direct access to scientific sources was also provided. The results indicate that:

1) In comparison with the control group, participants in the Chatbot group developed more positive attitudes towards the COVID- 19 vaccines (+37%); the proportion of people who did not want to be vaccinated was reduced from 36% to 29% while the intention to vaccinate as soon as the vaccine is available increased from 17% to 21%.

2) The amount of change in attitudes was related to time spent interacting with the chatbot;

3) The participants whose initial attitudes were the most negative shifted the most towards positive attitudes; no backfire effect was observed.

This study is very interesting and its results are promising. The extent of the changes in attitudes towards COVID-19 vaccines is of the same order of magnitude as what has been obtained (albeit in another context) with motivational interviewing, for childhood vaccines (Gagneur 2019: 40% reduction in parents’ vaccine hesitancy). As the authors of the present study point out, the 'dose-response' relationship observed between the time spent on the chatbot and the extent of the change in attitudes suggests that the latter is well related to interactions with the chatbot.

The fact that the greatest change in attitudes was found among those with the most negative attitudes at the outset is interesting but raises questions. Did these people spend more time on the chatbot than others because they had more questions? It would be interesting to have information and the authors' point of view on this point.

Unfortunately, despite a second wave of survey, carried out one to two weeks after the first data collection, the authors were unable, for technical reasons, to verify whether the changes in attitudes and intention to be vaccinated were subsequently sustained at the individual level. This is a very important question in a changing context where the public is exposed to a lot of information. Of course, research will also be needed to verify the impact even on vaccination behavior.

The question also arises as to whether the interaction on the chatbot confers resistance to false information or whether it can correct it. A recent article demonstrates the harmful impact of exposure to misinformation on public attitudes and COVID-19 vaccination intentions (Loomba et al. 2021) and quantifies its extent (recent misinformation induced a decline in intent to vaccinate of more than 6 percentage points). The authors could comment on this.

Another aspect deserves further investigation, perhaps through qualitative approaches: it is the "mechanisms" by which these changes were achieved: what made people change their minds, compared to other information modalities. It is essential to understand this from the point of view of the transferability of this approach. The authors discuss a first hypothesis, namely the fact that people can interact with the chatbot for as long as they wish and go beyond brief information: the depth of the response is somehow personalized. The fact that they can choose their questions (as opposed to being provided with unsolicited information), the neutrality of the answers, and the lack of judgment in relation to people's concerns (which could be the pitfalls of information delivered in a face-to-face relationship) are possibly other advantages of the chatbot that might also explain the lack of reactance. The Authors’ discussion of these hypotheses would be welcome.

The generalizability of the results of this study is an issue: the sample is not representative of the general population (it is younger, more educated, and more male). This limitation should be discussed.

Finally, a central question remains: that of the “conversion rate” when a tool such as a chatbot is disseminated, i.e. the actual proportion of people who, on receiving the link, would open it and "enter" into a conversation. It can be assumed that not everyone is necessarily inclined to accept or enjoy interacting with a chatbot and the overall impact of the diffusion of a tool depends directly on this, even though the feedback from the participants was positive. Previous publications if any on these aspects should be mentioned.

References

Gagneur et al. Promoting vaccination in maternity wards ─ motivational interview technique reduces hesitancy and enhances intention to vaccinate, results from a multicentre non-controlled pre- and post-intervention RCT-nested study, Quebec, March 2014 to February 2015. Euro Surveill. 2019 Sep;24(36):1800641. doi: 10.2807/1560-7917.ES.2019.24.36.1800641.

Loomba S et al. Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nat Hum Behav. 2021 Feb 5. doi: 10.1038/s41562-021-01056-1.