RR:C19 Evidence Scale rating by reviewer:
Potentially informative. The main claims made are not strongly justified by the methods and data, but may yield some insight. The results and conclusions of the study may resemble those from the hypothetical ideal study, but there is substantial room for doubt. Decision-makers should consider this evidence only with a thorough understanding of its weaknesses, alongside other evidence and theory. Decision-makers should not consider this actionable, unless the weaknesses are clearly understood and there is other theory and evidence to further support it.
***************************************
Review:
Summary:
From a large dataset, the authors selected a cohort of 16,277 patients who had positive COVID-19 PCR test. BERG’s Bayesian AI analytic was then used to generate 19 “causal” networks identifying 239 features as influencing mortality among the full cohort and sub-cohorts. A complete-case logistic regression with 25 main effect terms was used to identify features with low p-values. The authors used regularized logistic regression with elastic net penalization on 5 imputed datasets with 25 main terms, 300 pair-wise terms, and 7 squared terms. The authors found the MSE to be minimized when using only LASSO penalization. The authors then ran LASSO logistic regression across 10,000 bootstrapped datasets (generated from 10 imputed datasets) and looked at the median of the coefficient estimates and at the bootstrapped confidence intervals. Additionally, authors examined the proportion of these regressions in which each feature under examination was selected. The author’s primary conclusion identifies ondansetron “as the main factor associated with improved survival in mechanically ventilated COVID-19 patients.” Secondary conclusions confirm other beneficial medications and laboratory indicators consistent with previous literature.
Major Comments:
1. Why did the authors not adjust for multiple comparisons? Given the order of features under consideration, the number of significant findings in this paper could be consistent with chance. It is encouraging, as the authors point out, that most of the effect estimates found in their study are consistent at least in direction with existing literature. This does not, however, exclude the possibility that the findings regarding ondansetron occur in their data by chance. The coefficient estimates for the ondansetron and on ventilator interaction is large by comparison to other estimates given in the paper, so it’s possible the signal detected regarding ondansetron would not disappear after correcting for multiple comparisons, which could make the author’s conclusions stronger.
2. It is not surprising that the regressions in the second half of the paper agree with the feature selection made in the first half of the paper. Since the authors used the same data for feature selection and for estimation, I do not think it is fair to use the results from one (regression) as confirmation of the other. This is possibly the explanation for the dominance of LASSO over ridge regression in the glmnet applications.
Minor Comments:
1. Causality can be inferred from assumptions about networks, not based on the strength of prediction for a given network. As such, I caution against conclusions of causality such as those made by the authors regarding ondansetron treatment and COVID mortality.
2. I recommend authors include log-odds ratio estimates that incorporate main effect coefficients (e.g. the log-odds ratio comparing those with and without ondansetron among those ON a ventilator would have two terms: e^(-0.0797-0.365) = e^(- 0.4447)=0.641)