RR:C19 Evidence Scale rating by reviewer:
***************************************
Review:
This paper provides an extensive analysis of how side effects and biometric measurements are associated with nAbs after the 2nd dose of mRNA COVID-19 vaccines. The study is well designed with 6 days of detailed questionnaires plus the biometric measurements, which is much more extensive than other studies. The statistical methods are appropriate with attention to multiplicity and the results are interesting and have implications for messaging.
I was impressed with the statistical approach the authors adopted. I felt it was thoughtful and complete. For example, I liked how they considered 1st percentile mean and 99th percentile of the 4 biomarkers. I worked a little on that type of data a while and wish I had thought of using the extreme percentiles as they should be sensitive to effects while the mean likely just dilute signal. Then the idea of identifying biomarkers by comparing dose 1 to dose 2 is clever.
Some of the other data analysis is more standard, but still they chose the right method and then were thorough in their evaluation, e.g. diagnostics and knowing to use the Kenward-Roger method for small samples.
They do have a multiple testing problem and are principled and thorough. Their strategy was to be expansive and consider 3 way interactions between vaccine, time point, and feature. This approach allows that the effect of vaccines might differ, the effects might be different at different time points and that there might be a kind of synergy where e.g. the effect of mRNA-1273 at time 2 was much greater BNT. This all makes sense as the vaccines have different doses and the time course of nAb response might be complex.
I also felt the design was impressive with such a thorough questionnaire and the extensive biomarker measurements. Really, I’d not seen anything like this. In the Moderna phase 3 trial COVE, there had been a paper written on this topic, but it just used data that had been designed to assess adverse effects, their results were more modest and less interesting. So I was taken with this really thoughtful design and its successful execution.
I also felt their study---linking symptoms to immune response was of interest both to clinicians/researches but also the broader community. I also felt use of the nAb assay is a preferred metric for immune response. And I liked their visualizations.
It is interesting to select biometric endpoints based on the biggest effect between dose 1 and dose 2. Because this doesn’t involved the nAb measurement it doesn’t impact the type I error rate.
I’ve seen it written as nAb more commonly than nAB.
The X-axes of Figure 2 and 3 show much larger changes in skin temp and heart rate following dose 2 compared to dose 1. It’s well known that the greater variation in X, the greater power one has to detect a non-zero slope. This could be mentioned as a potential reason why the evidence is weaker for dose 1.