(Press-News.org) Artificial intelligence (AI) has practically limitless applications in healthcare, ranging from auto-drafting patient messages in MyChart to optimizing organ transplantation and improving tumor removal accuracy. Despite their potential benefit to doctors and patients alike, these tools have been met with skepticism because of patient privacy concerns, the possibility of bias, and device accuracy.
In response to the rapidly evolving use and approval of AI medical devices in healthcare, a multi-institutional team of researchers at the UNC School of Medicine, Duke University, Ally Bank, Oxford University, Colombia University, and University of Miami have been on a mission to build public trust and evaluate how exactly AI and algorithmic technologies are being approved for use in patient care.
Together, Sammy Chouffani El Fassi, a MD candidate at the UNC School of Medicine and research scholar at Duke Heart Center, and Gail E. Henderson, PhD, professor at the UNC Department of Social Medicine, led a thorough analysis of clinical validation data for 500+ medical AI devices, revealing that approximately half of the tools authorized by the U.S. Food and Drug Administration (FDA) lacked reported clinical validation data. Their findings were published in Nature Medicine.
“Although AI device manufacturers boast of the credibility of their technology with FDA authorization, clearance does not mean that the devices have been properly evaluated for clinical effectiveness using real patient data,” said Chouffani El Fassi, who was first author on the paper. “With these findings, we hope to encourage the FDA and industry to boost the credibility of device authorization by conducting clinical validation studies on these technologies and making the results of such studies publicly available.”
Since 2016, the average number of medical AI device authorizations by the FDA per year has increased from 2 to 69, indicating tremendous growth in commercialization of AI medical technologies. The majority of approved AI medical technologies are being used to assist physicians with diagnosing abnormalities in radiological imagining, pathologic slide analysis, dosing medicine, and predicting disease progression.
Artificial intelligence is able to learn and perform such human-like functions by using combinations of algorithms. The technology is then given a plethora of data and sets of rules to follow, so that it can “learn” how to detect patterns and relationships with ease. From there, the device manufacturers need to ensure that the technology does not simply memorize the data previously used to train the AI, and that it can accurately produce results using never-before-seen solutions.
Regulation During a Rapid Proliferation of AI Medical Devices
Following the rapid proliferation of these devices and applications to the FDA, Chouffani El Fassi and Henderson et al. were curious about how clinically effective and safe the authorized devices are. Their team analyzed all submissions available on the FDA's official database, titled “Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices.”
“A lot of the devices that came out after 2016 were created new, or maybe they were similar to a product that already was on the market,” said Henderson. “Using these hundreds of devices in this database, we wanted to determine what it really means for an AI medical device to be FDA-authorized.”
Of the 521 device authorizations, 144 were labelled as “retrospectively validated,” 148 were “prospectively validated,” and 22 were validated using randomized controlled trials. Most notably, 226 of 521 FDA-approved medical devices, or approximately 43%, lacked published clinical validation data. A few of the devices used “phantom images” or computer-generated images that were not from a real patient, which did not technically meet the requirements for clinical validation.
Furthermore, the researchers found that the latest draft guidance, published by the FDA in September 2023, does not clearly distinguish between different types of clinical validation studies in its recommendations to manufacturers.
Types of Clinical Validation and A New Standard
In the realm of clinical validation, there are three different methods by which researchers and device manufacturers validate the accuracy of their technologies: retrospective validation, prospective validation, and subset of prospective validation called randomized controlled trials.
Retrospective validation involves feeding the AI model image data from the past, such as patient chest x-rays prior to the COVID-19 pandemic. Prospective validation, however, typically produces stronger scientific evidence because the AI device is being validated based on real-time data from patients. This is more realistic, according to the researchers, because it allows the AI to account for data variables that were not in existence when it was being trained, such as patient chest x-rays that were impacted by viruses during the COVID pandemic.
Randomized controlled trials are considered the gold standard for clinical validation. This type of prospective study utilizes random assignment controls for confounding variables that would differentiate the experimental and control groups, thus isolating the therapeutic effect of the device. For example, researchers could evaluate device performance by randomly assigning patients to have their CT scans read by a radiologist (control group) versus AI (experimental group).
Because retrospective studies, prospective studies, and randomized controlled trials produce various levels of scientific evidence, the researchers involved in the study recommend that the FDA and device manufactures should clearly distinguish between different types of clinical validation studies in its recommendations to manufacturers.
In their Nature Medicine publication, Chouffani El Fassi and Henderson et al. lay out definitions for the clinical validation methods which can be used as a standard in the field of medical AI.
“We shared our findings with directors at the FDA who oversee medical device regulation, and we expect our work will inform their regulatory decision making,” said Chouffani El Fassi. “We also hope that our publication will inspire researchers and universities globally to conduct clinical validation studies on medical AI to improve the safety and effectiveness of these technologies. We’re looking forward to the positive impact this project will have on patient care at a large scale.”
Algorithms Can Save Lives
Chouffani El Fassi is currently working with UNC cardiothoracic surgeons Aurelie Merlo and Benjamin Haithcock as well as the executive leadership team at UNC Health to implement an algorithm in their electronic health record system that automates the organ donor evaluation and referral process.
In contrast to the field’s rapid production of AI devices, medicine is lacking basic algorithms, such as computer software that diagnose patients using simple lab values in electronic health records. Chouffani El Fassi says this is because implementation is often expensive and requires interdisciplinary teams that have expertise in both medicine and computer science.
Despite the challenge, UNC Health is on a mission to improve the organ transplant space.
“Finding a potential organ donor, evaluating their organs, and then having the organ procurement organization come in and coordinate an organ transplant is a lengthy and complicated process,” said Chouffani El Fassi. “If this very basic computer algorithm works, we could optimize the organ donation process. A single additional donor means several lives saved. With such a low threshold for success, we look forward giving more people a second chance at life.”
Other contributors to the study include Adonis Abdullah, Ying Fang, Sarabesh Natarajan, Awab Bin Masroor, Naya Kayali, Simran Prakash, Manesh R. Patel, William Ratliff, Zeshan Hussain, Justin Castillo, Ihsan Yüksel, Haonan Chen, Pengyu Guo, Hamilton McInnis, James E Hancock, Barbara J Evans, and Aurelie Merlo.
- Written by Kendall Daniels
END
Almost half of FDA-approved AI medical devices are not trained on real patient data
2024-08-26
ELSE PRESS RELEASES FROM THIS DATE:
Does the extent of structural racism in a neighborhood affect residents’ risk of cancer from traffic-related air pollution?
2024-08-26
High levels of traffic-related air pollutants have been linked with elevated risks of developing cancer and other diseases. New research indicates that multiple aspects of structural racism—the ways in which societal laws, policies, and practices systematically disadvantage certain racial or ethnic groups—may contribute to increased exposure to carcinogenic traffic-related air pollution. The findings are published by Wiley online in CANCER, a peer-reviewed journal of the American Cancer Society.
Most studies suggesting that structural racism, which encompasses factors such as residential segregation and differences in economic status and homeownership, may influence ...
2 in 3 parents want help preventing their child from developing hereditary health conditions
2024-08-26
ANN ARBOR, Mich. – Among things many families don’t wish to pass down to their children and grandchildren: medical issues.
One in five parents say their child has been diagnosed with a hereditary condition, while nearly half expressed concerns about their child potentially developing such a condition, a new national poll suggests.
And two thirds of parents want their healthcare provider to suggest ways to prevent their child from developing a health problem that runs in the family, according to the University of Michigan Health C.S. Mott Children’s Hospital National ...
Could psychedelic-assisted therapy change addiction treatment?
2024-08-26
by Amy Norton
PISCATAWAY, NJ – After years of being seen as dangerous “party drugs,” psychedelic substances are receiving renewed attention as therapies for addiction -- but far more research is needed, according to a new special series of articles in the Journal of Studies on Alcohol and Drugs, published at Rutgers University.
Psychedelics are substances that essentially alter users’ perceptions and thoughts about their surroundings and themselves. For millennia, indigenous cultures have used plants with psychedelic properties in traditional medicine and spiritual rituals. And for a time in the mid-20th ...
Sustaining oyster farming with sturdier rafts
2024-08-26
Amid the rising human population and pressure on food supplies, the world can’t be everyone’s oyster. But perhaps there might be more oysters to eat if an Osaka Metropolitan University-led research team’s findings mean sturdy plastic rafts will be used in their farming.
Conventional oyster farming uses bamboo rafts with additional flotation devices such as Styrofoam. Though relatively affordable, these rafts can be damaged in typhoons. The OMU-led researchers propose a polyethylene raft that keeps costs manageable but is about five times more durable than a bamboo raft.
OMU Graduate School of Engineering Associate ...
People of lower socioeconomic status less likely to receive cataract surgery in private clinics
2024-08-26
Despite increased funding for cataract surgeries to private, for-profit clinics, access to surgery fell 9% for lower-income people, according to new research published in CMAJ (Canadian Medical Association Journal) https://www.cmaj.ca/lookup/doi/10.1503/cmaj.240414.
“Unexpectedly, despite new public funding for operations provided in private for-profit surgical centres, which was intended to fully cover all overhead costs and remove the need to charge patients, this disparity did not decrease, but instead grew ...
Tick-borne Powassan virus in a child
2024-08-26
With tick-borne viruses such as Powassan virus increasing in Canada, clinicians should consider these infections in patients with encephalitis, as a case study shows in CMAJ (Canadian Medical Association Journal) https://www.cmaj.ca/lookup/doi/10.1503/cmaj.240227.
Although rare, Powassan virus is serious, with a death rate of 10%–15% in people with encephalitis, and it can cause lingering health effects after infection. The virus can transmit within 15 minutes of tick attachment, and symptoms can develop 1–5 weeks later.
In this case study, a 9-year-old child with up-to-date vaccinations ...
Survey finds more than 3 in 4 Americans don’t feel they could help someone suffering an opioid overdose
2024-08-26
EMBARGOED UNTIL 12:01 A.M. AUGUST 26, 2024
NOTE TO EDITOR: (To download broadcast-quality video and other multimedia elements: https://bit.ly/3M2ljhX (password: naloxone)
COLUMBUS, Ohio – International Overdose Awareness Day is a worldwide campaign held each Aug. 31 that acknowledges the grief of family and friends left behind from those who have died from a drug overdose. This year’s campaign theme “Together we can” highlights the power of the community standing together to help end overdose.
However, a new survey of 1,000 Americans from The Ohio State University Wexner Medical ...
How to control your screentime use and make technology work for you
2024-08-26
Many of us feel that we, or our children, spend too much time staring at a screen. From gaming to social media use or ‘doomscrolling,’ it can sometimes feel that we are mindlessly spending hours going down a rabbit hole of technology.
However, according to Catherine Knibbs, a psychotherapist who specializes in cybertrauma and online harms, there are tangible steps we can all take to wrestle back control from the hands of the technology corporations.
In her new book, Managing Your Gaming and Social ...
Matching dinosaur footprints found on opposite sides of the Atlantic Ocean
2024-08-25
DALLAS (SMU) – An international team of researchers led by SMU paleontologist Louis L. Jacobs has found matching sets of Early Cretaceous dinosaur footprints on what are now two different continents.
More than 260 footprints were discovered in Brazil and in Cameroon, showing where land-dwelling dinosaurs were last able to freely cross between South America and Africa millions of years ago before the two continents split apart.
“We determined that in terms of age, these footprints were similar,” Jacobs said. “In their geological and plate ...
Turning bacteria into bioplastic factories
2024-08-23
In a world overrun by petroleum-based plastics, scientists are searching for alternatives that are more sustainable, more biodegradable and far less toxic to the environment.
Two new studies by biologists at Washington University in St. Louis highlight one potential source of game-changing materials: purple bacteria that, with a little encouragement, can act like microscopic factories for bioplastics.
A study led by graduate student Eric Conners found that two relatively obscure species of purple bacteria have the ability to produce polyhydroxyalkanoates (PHAs), natural ...