(Press-News.org) Artificial intelligence can transform medicine in a myriad of ways, including its promise to act as a trusted diagnostic aide to busy clinicians.
Over the past two years, proprietary AI models, also known as closed-source models, have excelled at solving hard-to-crack medical cases that require complex clinical reasoning. Notably, these closed-source AI models have outperformed open-source ones, so-called because their source code is publicly available and can be tweaked and modified by anyone.
Has open-source AI caught up?
The answer appears to be yes, at least when it comes to one such open-source AI model, according to the findings of a new NIH-funded study led by researchers at Harvard Medical School and done in collaboration with clinicians at Harvard-affiliated Beth Israel Deaconess Medical Center and Brigham and Women’s Hospital.
The results, published March 14 in JAMA Health Forum, show that a challenger open-source AI tool called Llama 3.1 405B performed on par with GPT-4, a leading proprietary closed-source model. In their analysis, the researchers compared the performance of the two models on 92 mystifying cases featured in The New England Journal of Medicine weekly rubric of diagnostically challenging clinical scenarios.
The findings suggest that open-source AI tools are becoming increasingly competitive and could offer a valuable alternative to proprietary models.
“To our knowledge, this is the first time an open-source AI model has matched the performance of GPT-4 on such challenging cases as assessed by physicians,” said senior author Arjun Manrai, assistant professor of biomedical informatics in the Blavatnik Institute at HMS. “It really is stunning that the Llama models caught up so quickly with the leading proprietary model. Patients, care providers, and hospitals stand to gain from this competition.”
The pros and cons of open-source and closed-source AI systems
Open-source AI and closed-source AI differ in several important ways. First, open-source models can be downloaded and run on a hospital’s private computers, keeping patient data in-house. In contrast, closed-source models operate on external servers, requiring users to transmit private data externally.
“The open-source model is likely to be more appealing to many chief information officers, hospital administrators, and physicians since there’s something fundamentally different about data leaving the hospital for another entity, even a trusted one,” said the study’s lead author, Thomas Buckley, a doctoral student in the new AI in Medicine track in the HMS Department of Biomedical Informatics.
Second, medical and IT professionals can tweak open-source models to address unique clinical and research needs, while closed-source tools are generally more difficult to tailor.
“This is key,” said Buckley. “You can use local data to fine-tune these models, either in basic ways or sophisticated ways, so that they’re adapted for the needs of your own physicians, researchers, and patients.”
Third, closed-source AI developers such as OpenAI and Google host their own models and provide traditional customer support, while open-source models place the responsibility for model setup and maintenance on the users. And at least so far, closed-source models have proven easier to integrate with electronic health records and hospital IT infrastructure.
Open-source AI versus closed-source AI: A scorecard for solving challenging clinical cases
Both open-source and closed-source AI algorithms are trained on immense datasets that include medical textbooks, peer-reviewed research, clinical-decision support tools, and anonymized patient data, such as case studies, test results, scans, and confirmed diagnoses. By scrutinizing these mountains of material at hyperspeed, the algorithms learn patterns. For example, what do cancerous and benign tumors look like on pathology slide? What are the earliest telltale signs of heart failure? How do you distinguish between a normal and an inflamed colon on a CT scan? When presented with a new clinical scenario, AI models compare the incoming information to content they’ve assimilated during training and propose possible diagnoses.
In their analysis, the researchers tested Llama on 70 challenging clinical NEJM cases previously used to assess GPT-4’s performance and described in an earlier study led by Adam Rodman, HMS assistant professor of medicine at Beth Israel Deaconess and co-author on the new research. In the new study, the researchers added 22 new cases published after the end of Llama’s training period to guard against the chance that Llama may have inadvertently encountered some of the 70 published cases during its basic training.
The open-source model exhibited genuine depth: Llama made a correct diagnosis in 70 percent of cases, compared with 64 percent for GPT-4. It also ranked the correct choice as its first suggestion 41 percent of the time, compared with 37 percent for GPT-4. For the subset of 22 newer cases, the open-source model scored even higher, making the right call 73 percent of the time and identifying the final diagnosis as its top suggestion 45 percent of the time.
“As a physician, I’ve seen much of the focus on powerful large language models center around proprietary models that we can’t run locally,” said Rodman. “Our study suggests that open-source models might be just as powerful, giving physicians and health systems much more control on how these technologies are used.”
Each year, some 795,000 patients in the United States die or suffer permanent disability due to diagnostic error, according to a 2023 report.
Beyond the immediate harm to patients, diagnostic errors and delays can place a serious financial burden on the health care system. Inaccurate or late diagnoses may lead to unnecessary tests, inappropriate treatment, and, in some cases, serious complications that become harder — and more expensive — to manage over time.
“Used wisely and incorporated responsibly in current health infrastructure, AI tools could be invaluable copilots for busy clinicians and serve as trusted diagnostic aides to enhance both the accuracy and speed of diagnosis,” Manrai said. “But it remains crucial that physicians help drive these efforts to make sure AI works for them.”
Authorship, funding, disclosures
Additional authors include Byron Crowe and Raja-Elie E. Abdulnour.
This project was supported by award K01HL138259 from the National Heart, Lung, and Blood Institute and a Harvard Medical School Dean’s Innovation Award.
Crowe reported receiving personal fees from Solera Health outside the submitted work. Rodman reported receiving grants from the Gordon and Betty Moore Foundation outside the submitted work.
END
Open-Source AI matches top proprietary model in solving tough medical cases
New analysis points to greater competition between AI diagnostic tools, a shift that stands to benefit patients and clinicians alike
2025-03-15
ELSE PRESS RELEASES FROM THIS DATE:
Good fences make good neighbors (with carnivores)
2025-03-15
A predator’s gotta eat, but sometimes what they eat harms people sharing the landscape, and that often leads to the carnivore’s death.
Fortified corrals are one strategy used in Tanzania to protect both livestock and vulnerable carnivore species. But then where do lions, leopards and hyenas go for dinner? Do they feed on the next herd over?
A new study led by Colorado State University has found that good fences truly do make good neighbors because fortified enclosures also benefit livestock keepers who live nearby. Instead of dining on easier meals next-door and negatively impacting neighbors who don’t ...
NRG Oncology trial supports radiotherapy alone following radical hysterectomy should remain the standard of care for early-stage, intermediate-risk cervical cancer
2025-03-15
Results from the NRG Oncology GOG-0263 phase III clinical trial testing the addition of cisplatin-based chemotherapy to adjuvant radiotherapy following radical hysterectomy and lymphadenectomy for patients with early-stage, intermediate-risk cervical carcinoma indicated that the addition of chemotherapy did not improve outcomes for patients and led to increased toxicity for patients. The outcomes of this trial support the use of the current standard of care using adjuvant radiotherapy alone following surgery. These results were ...
Introducing our new cohort of AGA Future Leaders
2025-03-14
We’re thrilled to announce the 16 distinguished early-career gastroenterologists and hepatologists selected for our 2025-2026 class of AGA Future Leaders. This AGA program cultivates effective leadership skills for professional advancement in AGA and within the field of digestive diseases.
Meet the AGA Future Leaders Class of 2025-2026
Lubin Arevalo, MD
Veroushka Ballester, MD, MS
Victor Chedid, MD, MS
Ryan Fawley, MD
Melissa Hershman, MD
Pichamol Jirapinyo, MD, MPH
Babu Pappu Mohan, MD
Carolyn Newberry, MD
Long ...
Sharks are dying at alarming rates, mostly due to fishing. Retention bans may help
2025-03-14
Despite the fear they may inspire in humans, sharks have far more reason to fear us. Nearly one-third of sharks are threatened with extinction globally, mostly as a result of fishing.
A team led by researchers at UC Santa Barbara discovered that mandates to release captured sharks won’t be enough to prevent the continued decline of these important ocean predators. These findings, published in Fish & Fisheries, highlight the importance of monitoring shark populations and combining different strategies for managing their numbers.
Some ...
Engineering excellence: Engineers with ONR ties elected to renowned scientific academy
2025-03-14
Three esteemed engineers with ties to the Office of Naval Research (ONR) have been elected to the prestigious National Academy of Engineering (NAE) Class of 2025. NAE members are among the world’s most accomplished engineers from business, academia and government.
“On behalf of the Office of Naval Research, I’m proud to extend my sincerest congratulations to these new members of the National Academy of Engineering,” said Chief of Naval Research Rear Adm. Kurt Rothenhaus. “Not only have these accomplished engineering professionals supported and conducted valuable naval-relevant research, they’re also enhancing the strength and prosperity of our nation by serving ...
New CRISPR-based diagnostic test detects pathogens in blood without amplification
2025-03-14
Bioengineering professor and The Grainger College of Engineering’s Dean, Rashid Bashir, led a team of researchers in a project that’s resulted in new technology that offers rapid, highly sensitive detection of multi-drug-resistant bacteria and other pathogens at low concentrations.
This research was featured in an article in the Proceedings of the National Academy of Sciences of the United States of America (PNAS).
Researchers designed a CRISPR-based test that rapidly detects low levels of pathogen genetic material in blood. This is done without the need for nucleic acid amplification.
In ...
Immunotherapy may boost KRAS-targeted therapy in pancreatic cancer
2025-03-14
PHILADELPHIA – Adding immunotherapy to a new type of inhibitor that targets multiple forms of the cancer-causing gene mutation KRAS kept pancreatic cancer at bay in preclinical models for significantly longer than the same targeted therapy by itself, according to researchers from the Perelman School of Medicine at the University of Pennsylvania and Penn Medicine’s Abramson Cancer Center. The results, published in Cancer Discovery, prime the combination strategy for future clinical trials.
Combatting the “undruggable” ...
Growing solar: Optimizing agrivoltaic systems for crops and clean energy
2025-03-14
Agrivoltaic systems, which combine solar power generation with agricultural practices, offer a promising solution to the growing demand for both renewable energy and food production. By integrating solar panels with crops, these systems not only address the land use conflict between agriculture and energy production, but they also provide important benefits such as reducing crop water stress and offering protection against extreme weather events. In addition, agrivoltaics can contribute to biodiversity by providing pollinator habitats and forage production. ...
Scientists discover how to reactivate cancer’s molecular “kill switch”
2025-03-14
Alternative RNA splicing is like a movie editor cutting and rearranging scenes from the same footage to create different versions of a film. By selecting which scenes to keep and which to leave out, the editor can produce a drama, a comedy, or even a thriller—all from the same raw material. Similarly, cells splice RNA in different ways to produce a variety of proteins from a single gene, fine-tuning their function based on need. However, when cancer rewrites the script, this process goes awry, fueling tumor growth and survival.
In a recent study reported in the Feb. 15 issue of Nature ...
YouTube influencers: gaming’s best friend or worst enemy?
2025-03-14
New INFORMS Marketing Science Study Key Takeaways:
YouTube influencers increase player engagement and playtime but often reduce game purchases, especially for story-driven games.
A unique event in YouTube’s history, the “Adpocalypse,” allowed researchers to measure the causal impact of influencer content, revealing its complex effects on game sales and usage.
Game developers must align their business models with influencer marketing, because games with in-game purchases benefit from exposure, while ...
LAST 30 PRESS RELEASES:
Study identifies candidates for therapeutic targets in pediatric germ cell tumors
Media alert: The global burden of CVD
Study illuminates contributing factors to blood vessel leakage
What nations around the world can learn from Ukraine
Mixing tree species does not always make forests more drought-resilient
Public confidence in U.S. health agencies slides, fueled by declines among Democrats
“Quantum squeezing” a nanoscale particle for the first time
El Niño spurs extreme daily rain events despite drier monsoons in India
Two studies explore the genomic diversity of deadly mosquito vectors
Zebra finches categorize their vocal calls by meaning
Analysis challenges conventional wisdom about partisan support for US science funding
New model can accurately predict a forest’s future
‘Like talking on the telephone’: Quantum computing engineers get atoms chatting long distance
Genomic evolution of major malaria-transmitting mosquito species uncovered
Overcoming the barriers of hydrogen storage with a low-temperature hydrogen battery
Tuberculosis vulnerability of people with HIV: a viral protein implicated
Partnership with Kenya's Turkana community helps scientists discover genes involved in adaptation to desert living
Decoding the selfish gene, from evolutionary cheaters to disease control
Major review highlights latest evidence on real-time test for blood – clotting in childbirth emergencies
Inspired by bacteria’s defense strategies
Research spotlight: Combination therapy shows promise for overcoming treatment resistance in glioblastoma
University of Houston co-leads $25 million NIH-funded grant to study the delay of nearsightedness in children
NRG Oncology PREDICT-RT study completes patient accrual, tests individualized concurrent therapy and radiation for high-risk prostate cancer
Taking aim at nearsightedness in kids before it’s diagnosed
With no prior training, dogs can infer how similar types of toys work, even when they don’t look alike
Three deadliest risk factors of a common liver disease identified in new study
Dogs can extend word meanings to new objects based on function, not appearance
Palaeontology: South American amber deposit ‘abuzz’ with ancient insects
Oral microbes linked to increased risk of pancreatic cancer
Soccer heading does most damage to brain area critical for cognition
[Press-News.org] Open-Source AI matches top proprietary model in solving tough medical casesNew analysis points to greater competition between AI diagnostic tools, a shift that stands to benefit patients and clinicians alike