Review 1: "Accelerating Cough-Based Algorithms for Pulmonary Tuberculosis Screening: Results from the CODA TB DREAM Challenge"

Tauhidur Rahman

doi:doi:10.1162/2e3983f5.efc3eee9

Accelerating cough-based algorithms for pulmonary tuberculosis screening: Results from the CODA TB DREAM Challenge

by Devan Jaganath, Solveig K Sieberts, Mihaja Raberahona, Sophie Huddart, Larsson Omberg, Rivo Rakotoarivelo, Issa Lyimo, Omar Lweno, Devasahayam J. Christopher, Nguyen Viet Nhung, William Worodria, Charles Yu, Jhih-Yu Chen, Sz-Hau Chen, Tsai-Min Chen, Chih-Han Huang, Kuei-Lin Huang, Filip Mulier, Daniel Rafter, Edward S.C. Shih, Yu Tsao, Hsuan-Kai Wang, Chih-Hsun Wu, Christine Bachman, Stephen Burkot, Puneet Dewan, Sourabh Kulhare, Peter M. Small, Vijay Yadav, Simon Grandjean Lapierre, Grant Theron, and Adithya Cattamanchi

Show Description
Published on May 14, 2024
www.medrxiv.org

Description

Abstract Importance Open-access data challenges have the potential to accelerate innovation in artificial-intelligence (AI)-based tools for global health. A specimen-free rapid triage method for TB is a global health priority.Objective To develop and validate cough sound-based AI algorithms for tuberculosis (TB) through the Cough Diagnostic Algorithm for Tuberculosis (CODA TB) DREAM challenge.Design In this diagnostic study, participating teams were provided cough-sound and clinical and demographic data. They were asked to develop AI models over a four-month period, and then submit the algorithms for independent validation.Setting Data was collected using smartphones from outpatient clinics in India, Madagascar, the Philippines, South Africa, Tanzania, Uganda, and Vietnam.Participants We included data from 2,143 adults who were consecutively enrolled with at least two weeks of cough. Data were randomly split evenly into training and test partitions.Exposures Standard TB evaluation was completed, including Xpert MTB/RIF Ultra and culture. At least three solicited coughs were recorded using the Hyfe Research app.Main Outcomes and Measures We invited teams to develop models using 1) cough sound features only and/or 2) cough sound features with routinely available clinical data to classify microbiologically confirmed TB disease. Models were ranked by area under the receiver operating characteristic curve (AUROC) and partial AUROC (pAUROC) to achieve at least 80% sensitivity and 60% specificity.Results Eleven cough models were submitted, as well as six cough-plus-clinical models. AUROCs for cough models ranged from 0.69-0.74, and the highest performing model achieved 55.5% specificity (95% CI 47.7-64.2) at 80% sensitivity. The addition of clinical data improved AUROCs (range 0.78-0.83), five of the six submitted models reached the target pAUROC, and highest performing model had 73.8% (95% CI 60.8-80.0) specificity at 80% sensitivity. In post-challenge subgroup analyses, AUROCs varied by country, and was higher among males and HIV-negative individuals. The probability of TB classification correlated with Xpert Ultra semi-quantitative levels.Conclusions and Relevance In a short period, new and independently validated cough-based TB algorithms were developed through an open-source and transparent process. Open-access data challenges can rapidly advance and improve AI-based tools for global health.Key Points Question Can an open-access data challenge support the rapid development of cough-based artificial intelligence (AI) algorithms to screen for tuberculosis (TB)?Findings In this diagnostic study, teams were provided well-characterized cough sound data from seven countries, and developed and submitted AI models for independent validation. Multiple models that combined clinical and cough data achieved the target accuracy of at least 80% sensitivity and 60% specificity to classify microbiologically-confirmed TB.Meaning Cough-based AI models have promise to support point-of-care TB screening, and open-access data challenges can accelerate the development of AI-based tools for global health.

RR:C19 Evidence Scale rating by reviewer:

Reliable. The main study claims are generally justified by its methods and data. The results and conclusions are likely to be similar to the hypothetical ideal study. There are some minor caveats or limitations, but they would/do not change the major claims of the study. The study provides sufficient strength of evidence on its own that its main claims should be considered actionable, with some room for future revision.

***************************************

Review: This manuscript presents the results of the CODA TB DREAM Challenge, an open data challenge aimed at developing cough-based artificial intelligence (AI) algorithms for tuberculosis (TB) screening. The study addresses an important need to accelerate the development of AI tools for global health applications, particularly for TB diagnosis in low—and middle-income countries. Personally, I commend the authors for their innovative approach to bringing this idea of open innovation to the field of artificial intelligence and machine learning in public health research.

Things that I liked about this research:

The study design is robust, utilizing a large multi-country dataset of well-characterized cough sounds and clinical data from symptomatic individuals. The number of participants was also high (more than 2,000).
The challenge framework allowed for quick development and independent validation of multiple AI algorithms. It shows the usefulness and potential of open science programs in global health.
The authors made the training data and the winning team's write-up available to the public, which should be very helpful for future researchers.
The authors provide a comprehensive analysis of model performance, including subgroup analysis.

Things that I wish were better:

The target population was intentionally chosen by people who were already symptomatic. In general, if someone is so sick that the person needs to visit a hospital, that person needs a professional medical diagnosis, not a diagnosis based on any cough monitoring app. There is no mention of collecting data from the general population, which means the applicability of such technologies is still uncertain for general population cases. Thus, it is vague which target population demographic/niche this study is useful for, i.e. the outcomes of this study only prove that you can detect TB with certainty when the population is symptomatic. At this point, they just need a medical diagnosis; it is not clear what happens when the target population, including the general population, might benefit from this kind of technology, which can be deployed in a ubiquitous setting.
The data only contains mostly solicited coughs. I believe that instead of this, the study could've benefitted by using more unsolicited/natural cough sounds.
The recording time per cough (0.5 seconds) is too short. As the coughs were mostly collected in a solicited fashion, it could've been beneficial to collect coughs in longer time windows.
The ethical considerations of collecting data from under-developed countries could be discussed in more detail.

Overall, I think this study is a good initiative towards building an open and collaborative research culture. I have personally participated in this kind of competition and benefitted from it. Despite some questions about the study's applicability and usefulness in the real world, I agree with the article's primary claim that an open and collaborative environment is needed to accelerate and develop artificial intelligence in the domain of public health.

Review 1: "Accelerating Cough-Based Algorithms for Pulmonary Tuberculosis Screening: Results from the CODA TB DREAM Challenge"

Connections