Review 2: "Validation of a Saliva-Based Test for the Molecular Diagnosis of SARS-CoV-2 Infection"

RR:C19 Evidence Scale rating by reviewer:

• Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.

***************************************

Review: The manuscript by Bulfoni et al. is a validation study of an RT-PCR based diagnostics test for SARS-CoV-2 using saliva as specimen mixed into a proprietary saline solution. Their protocol involved extracting total RNA from preserved saliva using Qiagen kit and an automated extractor, followed by a 1-step RT-PCR using Roche LightCycler master mix and primers that target the envelope (E) gene of SARS-CoV-2, run on a Roche LightCycler 480 instrument. Paired saliva and NP swab samples were collected, processed, and analyzed from 1161 test subjects to establish sensitivity, specificity, and percent agreement of their protocol to the conventional NP swab approach. Their main finding was that saliva could be used as biological matrix for identification of SARS-CoV-2 infection. As the world continues to struggle to mitigate the spread of COVID-19, and access to vaccines is highly heterogenous, effective testing for COVID-19 is a topic of intense interest throughout the scientific community and general public worldwide. For an assay to be effective, it has to have the following features: convenient and non-invasive sample collection method that can be done by non-healthcare workers with minimal PPE or through self-collection, high sensitivity and specificity, hours (not days) turnaround time, low cost, lack of supply chain bottlenecks (most notably, no need for RNA isolation), and massive scalability. Several studies have already shown that saliva is comparable to NP swab as specimen for SARS-CoV-2 testing. In the US alone, the FDA has already given emergency use approval (EUA) to various RT-PCR based tests that use saliva as biological matrix (https://www.fda.gov/medicaldevices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/in-vitrodiagnostics-euas-molecular-diagnostic-tests-sars-cov-2). All these tests have undergone thorough validation studies for their claims, and parameters such as collection, transport of sample, storage solutions, and processing have been clearly defined and backed with data in their respective instructions for use (IFUs) prior to FDA issuance of an EUA. Thus it is not fair to say in the manuscript that there are “no systematic studies are available yet, in particular concerning saliva collection and analysis validation based on both patients and asymptomatic or healthy groups.” As the authors have alluded to in their intro and discussion, not all saliva tests are the same, i.e. the sensitivity of RT-PCR using saliva samples varies depending on the collection technique (swabbing the tongue and/or cheek is different than collecting 1 mL of saliva drooled into a tube) as well as important differences in processing methods between different saliva tests. Hence, the statements addressing the scalability of their assay for mass testing should be clarified or addressed. Other major concerns: Table 3 showed that a positive concordance between saliva and NP swab was observed in 89 patients who showed symptoms at median 11 days prior to sample collection/testing. The total number of observations only add up to 152 out of the 156 patients admitted for COVID symptoms. Furthermore, according to their results in Table 3, out of the 152 samples tested, 109 tested positive in saliva, 94 tested positive in NP swabs, and 89 tested positive for both saliva and NP swab. However, their description in the paragraph after Table 3 says “Data regarding Ct values were available for 112/156 molecular tests based on saliva and for 71/156 NP swabs”, which does not match the numbers on Table 3 and makes it confusing for the readers. How would a sample be called positive if there were no Ct values measured? How come there were 94 positives in NP swabs but only 71 have available Ct values? The authors have to go back and double check their results. In addition, Figure 1 caption statement has to be modified to reflect the true number of paired Ct values that are compared in the plot. Same goes for Figure 2 and Table 4 description. Are there 89 or 71 samples that tested positive for both saliva and NP swab? They described their assay validation in the methods section, but no data could be found in the results section to support their claims on their assay LoD. While they did show that saliva collected in saline solution was superior to fresh saliva in terms of stability and outcome of assay by comparing 25 matched samples (saliva mixed with saline solution yielded 7 more positives compared to fresh saliva, Figure 3), it was not clear how many out of the 25 patient samples were true positives and true negatives. In addition, they described measuring the temporal stability of saliva and at different storage temperature (18-25 and 4C), and reported that “Our data indicates that viral RNA remains protected even at RT and at 4C up to 48h if collected in saline medium”, but the data backing up this claim could be found nowhere in the manuscript. Similarly, the entire section on “Validation of analytical procedure” has no data shown to back up their claims.

Minor concerns:

1) Their LoD has to be defined more clearly, instead of 5 copies per reaction, either convert this to 1 copy/uL (i.e. 5 copies per 5uL in each reaction) or 1000 copies/mL.

2) In their multiplex PCR assay, they described primers targeting the SARS-CoV-2 E gene, but did not mention any other targets for an internal control. Did they use primers for RnaseP, for example?

3) Because their RT-PCR based test only detects for the wild type E gene of SARS-CoV-2, the authors have to demonstrate that the primer recognition sites are not impacted by the presence of SARS-CoV-2 genetic variants in a patient sample. Adding another SARS-CoV-2 target to the multiplex PCR assay will help reduce the impact of a continuously mutating virus to this labdeveloped COVID-19 diagnostics test.

4) The final sentence in the discussion part lacks coherence and were not supported by data that they presented: “The test may also look for viral proteins to screen large numbers of asymptomatic people and in a future perspective it can be used also to characterize SARS-CoV-2 variants, which are considered more transmissible for their compartmentalization in saliva, their spread and high replication rate.