RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
The manuscript describes the development and clinical performance evaluation of a lab-developed quantitative chemiluminescence assay utilizing the automated TOSOH AIA-CL analyzer system to detect IgG and total Ig formed against SARS-CoV-2 nucleocapsid and spike protein antigens in patient sera. This work builds upon previous suggestions by the group that an N-terminally truncated nucleocapsid protein antigen would help to increase the specificity of such type of serological assay. The spike protein epitope utilized in this study is its receptor binding domain, which is also utilized commonly in other serological tests for COVID-19. In brief, the authors determined that their assays exhibited increasing sensitivity with time (between symptom onset and testing) and was maximally sensitive and specific at 100% on or after 13 days post symptom onset. This work also compared the performance of the anti-NP IgG and total Ig assays with that of two commercially available qualitative chemiluminescence assays (the Roche Elecsys total Ig and Abbott Architect IgG assays), which resulted in strong correlations as assessed by the Pearson’s correlation coefficient. Overall, the rationale and design of the work are sound and were presented coherently. The main claims of the manuscript with regards to clinical performance are reliable and informative. There is, however, a lack of analytical validation data, which limits the generalizability that the assay “could be used as a reliable method for accurate detection of Covid-19 patients”, as detailed below. Specific comments, points of concern, and suggestions for revision are additionally included below.
1. In the Introduction, the authors should clarify that PCR is currently the standard diagnostic test for ongoing COVID-19 infection, not serological testing. Serological testing may provide a useful means to assess an individual’s immune response to viral exposure, infection, or vaccination, and thus generate important ancillary public health and epidemiological data; however, this testing modality is currently underutilized owing in large part to our still incomplete understanding of seroconversion of antibodies against SARS-CoV-2. At present, serological testing reveals an individual’s immune response (and in particular, antibody levels) to SARS-CoV-2, but not necessarily overall immunity, as further studies are needed.
2. The cutoff values of the four test systems (anti-NP and anti-SP/RBD IgG and total Ig) were established using 600 serum samples from non-infected, healthy donors and 17 samples from COVID-19 patients. A baseline index value of 1.0 was assigned to each test system using an ROC curve. The data described is presented in Figure S1, which was not included in the preprint for this review.
3. There is a glaring lack of analytical validation data expected for an assay panel destined for clinical use:
A. Was linearity of the chemiluminescence signal established? What is the analyte measuring range for these assays? What is the lower limit of quantitation?
B. What is the precision of these assays, as shown by repeatability (within-run) and reproducibility (between-day) data for these quantitative assays?
C. What is the robustness of these assays in the presence of common endogenous interferents such as in cases of hemolysis, lipemia, and icterus?
D. Is there cross-reactivity when the assays are performed on samples derived from patients with non-COVID-19 respiratory infections?
E. How do these assays compare with other quantitative assays?
4. The authors stated that the sensitivity of the anti-NP IgG assay was greater than that of the other assays in the “early phase of COVID-19”, but was this difference statistically significant? “Greater” in this context was subjective and stated without provision of actual comparison data.
5. Based on Figure 3, there does not appear to be a clear difference between each antibody titer among moderate, severe, and critical COVID-19 cases, which is an interesting finding. In several of the cases plotted in Figure 3, the measured index value at the early phases of disease was < 1.0; are these measurements not included in Figure 2?
6. The clinical performance of the four quantitative (anti-NP and anti-SP IgG and total Ig) test systems of the assay were evaluated using 1,000 and 202 serum samples collected from 1,000 healthy donors and 42 COVID-19 patients (at 7 or more days post symptom onset from two healthcare sites), respectively (of note, Table S1 shows a total of 44 COVID-19 patients but the manuscript only made specific mention that one patient with a history of anticancer therapy was excluded from the final study set). The results were then stratified according to the number of days post symptom onset, i.e. 7-9 days, 10-12 days, 13-20 days, 21-30 days, and 31+ days, as shown in Figure 2. For the 153 samples representing 13+ days post symptom onset, which were derived from 30 of the 42 COVID-19 patients, the sensitivity and specificity of all four test systems are 100%, as shown in Figure 4. Which were these 30 patients as described in Table S1, and specifically, did this subset include any of the mild and moderate cases? While the data shown by Figures 2 and 4 are impressive, additional granularization and stratification would potentially reveal additional information from this quantitative versus other already available qualitative assays.
7. The authors compared the performance of their quantitative anti-NP IgG and total Ig assays with the performance of the Roche total Ig and Abbott IgG assays, two commercially available qualitative serological assays, using 44 COVID-19 antibody-positive patient samples purchased from Biomex GmbH. Overall, the comparisons showed strong correlation as evidenced by the calculated Pearson’s correlation coefficients included in Table. However, correlation coefficients do not reveal exactly the extent of concordance between quantitative and qualitative assays. It would be both interesting and helpful to know the measured index values for these 44 samples using the quantitative anti-NP IgG and total Ig assays, as well as the anti-SP IgG and total Ig assays for further comparison. Conventionally, it would have been more appropriate to compare a quantitative assay with another quantitative assay and provide regression and Bland-Altman plots to assess the degree of agreement between the methods.
8. As shown in Table S1, in spite of the much larger number of healthy donors versus COVID-19 patients, the basic demographics between the two groups were also quite different—most notably, the mean age: 33.1 (7.6 standard deviation) versus 61.3 (16.5 SD) and 64.5 (17.2 SD) in the two COVID-19 cohorts. Furthermore, as the authors also pointed out in their Discussion, the majority of the COVID-19 cases from which samples were derived for the clinical performance evaluation were classified as moderate, severe, or critical (there were zero asymptomatic cases and 2 mild cases). Thus, the lack of adequate representation of asymptomatic and mild cases may limit the generalizability of the performance findings for the anti-NP and anti-SP IgG and total Ig quantitative assays.
9. Future applications of this work include utilizing the quantitative results to monitor vaccine response, to trend an individual’s immune response relative to the COVID-19 disease course, and to survey the efficacy of public health measures and epidemiological response.
10. Proofreading for correct grammar, spelling mistakes, and other errors in both text and figures is advised. For example, “Sencitivity” in Figure 4 is misspelled. Additionally, on page 17, the last paragraph states, “Since our assay measures total Ig in addition to IgG, it can possibly be more precise than assays that only detect a particular Ig type”; since precision measures how reproducible a test is, we believe the correct phrase should be “more sensitive” rather than “more precise.”