(Press-News.org)
With hospitals already deploying artificial intelligence to improve patient care, a new study has found that using Chat GPT Plus does not significantly improve the accuracy of doctors’ diagnoses when compared with the use of usual resources.
The study, from UVA Health’s Andrew S. Parsons, MD, MPH and colleagues, enlisted 50 physicians in family medicine, internal medicine and emergency medicine to put Chat GPT Plus to the test. Half were randomly assigned to use Chat GPT Plus to diagnose complex cases, while the other half relied on conventional methods such as medical reference sites (for example, UpToDate©) and Google. The researchers then compared the resulting diagnoses, finding that the accuracy across the two groups was similar.
That said, Chat GPT alone outperformed both groups, suggesting that it still holds promise for improving patient care. Physicians, however, will need more training and experience with the emerging technology to capitalize on its potential, the researchers conclude.
For now, Chat GPT remains best used to augment, rather than replace, human physicians, the researchers say.
“Our study shows that AI alone can be an effective and powerful tool for diagnosis,” said Parsons, who oversees the teaching of clinical skills to medical students at the University of Virginia School of Medicine and co-leads the Clinical Reasoning Research Collaborative. “We were surprised to find that adding a human physician to the mix actually reduced diagnostic accuracy though improved efficiency. These results likely mean that we need formal training in how best to use AI.”
Chat GPT for Disease Diagnosis
Chatbots called “large language models” that produce human-like responses are growing in popularity, and they have shown impressive ability to take patient histories, communicate empathetically and even solve complex medical cases. But, for now, they still require the involvement of a human doctor.
Parsons and his colleagues were eager to determine how the high-tech tool can be used most effectively, so they launched a randomized, controlled trial at three leading-edge hospitals – UVA Health, Stanford and Harvard’s Beth Israel Deaconess Medical Center.
The participating docs made diagnoses for “clinical vignettes” based on real-life patient-care cases. These case studies included details about patients’ histories, physical exams and lab test results. The researchers then scored the results and examined how quickly the two groups made their diagnoses.
The median diagnostic accuracy for the docs using Chat GPT Plus was 76.3%, while the results for the physicians using conventional approaches was 73.7%. The Chat GPT group members reached their diagnoses slightly more quickly overall – 519 seconds compared with 565 seconds.
The researchers were surprised at how well Chat GPT Plus alone performed, with a median diagnostic accuracy of more than 92%. They say this may reflect the prompts used in the study, suggesting that physicians likely will benefit from training on how to use prompts effectively. Alternately, they say, healthcare organizations could purchase predefined prompts to implement in clinical workflow and documentation.
The researchers also caution that Chat GPT Plus likely would fare less well in real life, where many other aspects of clinical reasoning come into play – especially in determining downstream effects of diagnoses and treatment decisions. They’re urging additional studies to assess large language models’ abilities in those areas and are conducting a similar study on management decision-making.
“As AI becomes more embedded in healthcare, it's essential to understand how we can leverage these tools to improve patient care and the physician experience,” Parsons said. “This study suggests there is much work to be done in terms of optimizing our partnership with AI in the clinical environment.”
Following up on this groundbreaking work, the four study sites have also launched a bi-coastal AI evaluation network called ARiSE (AI Research and Science Evaluation) to further evaluate GenAI outputs in healthcare. Find out more information at the ARiSE website.
Findings Published
The researchers have published their results in the scientific journal JAMA Network Open. The research team consisted of Ethan Goh, Robert Gallo, Jason Hom, Eric Strong, Yingjie Weng, Hannah Kerman, Joséphine A. Cool, Zahir Kanjee, Parsons, Neera Ahuja, Eric Horvitz, Daniel Yang, Arnold Milstein, Andrew P.J. Olson, Adam Rodman and Jonathan H. Chen. Funding for this research was provided by the Gordon and Betty Moore Foundation. A full list of disclosures and funding sources is included in the paper.
To keep up with the latest medical research news from UVA, subscribe to the Making of Medicine blog at http://makingofmedicine.virginia.edu.
END
Extreme weather spurred by climate change, including droughts and heavy rains, may increase the risk of nitrates from fertilizers ending up in groundwater, according to a recent study from researchers at the University of California, Davis. The study found heavy rains after a drought caused nitrates to seep 33 feet under farm fields in as little as 10 days. The study was published in Water Resources Research.
“The conventional wisdom was that it could take several weeks to years for nitrates to move from the crop root zones to reach groundwater,” said corresponding author Isaya Kisekka, a professor in the Departments ...
Background and Aims
Liver cancer is a digestive system malignancy that poses a significant public health challenge globally. This study aimed to analyze and compare the epidemiological trends of liver cancer attributed to hepatitis B (LCHB) and alcohol use (LCAL) over the past 32 years.
Methods
Data on mortality and disability-adjusted life years for LCHB and LCAL in China, globally, and across five sociodemographic index regions were obtained from the Global Burden of Disease 2021 database and comprehensively ...
One only needs to glance at the news, social media, or even just out the window to understand the devastation caused by flooding. Recent back-to-back major hurricanes have brought catastrophic rainfall that has devastated communities across the southeastern United States.
With climate change, experts predict these extreme weather events will increasingly become the norm. Among the many ways that researchers are devising strategies to protect and assist vulnerable areas, one such effort involves increasing the speed and accuracy of damage assessments.
“Research ...
Stories on the discovery of vital fluid-transport systems in the human body are among the winners of the 2024 AAAS Kavli Science Journalism Awards. Winning journalists also did immersive stories on scientists and physicians at work – in the field, in the lab and in the emergency room.
Independent panels of science journalists select the winners of the awards, which are administered by the American Association for the Advancement of Science (AAAS) and endowed by The Kavli Foundation. There is a Gold Award ($5,000) and Silver Award ($3,500) for each of the eight categories. The global awards program drew entries ...
Mass General Brigham AI is hosting the Healthcare AI Challenge, a multi-institutional virtual, interactive series of events where healthcare professionals can explore and assess the latest AI healthcare technologies in real-world healthcare scenarios.
The Healthcare AI Challenge Collaborative is launching with a diverse set of healthcare institutions and their healthcare professionals, including Mass General Brigham; Emory Healthcare; the Department of Radiology at the University of Wisconsin School of Medicine and Public Health; and the Department of Radiology at the University of Washington School of Medicine. The American College of Radiology (ACR), ...
NEW YORK, NY— The American Federation for Aging Research (AFAR) is pleased to announce the recipients of its 2024 AFAR Grants for Junior Faculty. The AFAR Grants for Junior Faculty program provides up to $150,000 for a one- to two-year award to junior faculty (MDs and PhDs) to conduct research that will serve as the basis for longer term research efforts on the biology of aging. The major goal of this program is to assist in the development of the careers of early career investigators committed to pursuing careers in aging research. Selected through ...
Highlights:
An FDA-approved vaccine for smallpox and mpox is effective but causes side effects. The other requires multiple doses.
An experimental single-dose vaccine uses the horsepox virus to harness the benefits of both strategies.
Previous studies suggest that inoculation with horsepox elicits an antibody response to mpox and provides 100% protection in animal models.
New data show that the horsepox virus is significantly more attenuated, or weakened, compared to the virus used in the FDA-approved live virus vaccine.
Washington, D.C.—Vaccines that prevent smallpox and mpox come in 2 varieties. One uses a single shot of a live virus but ...
The Mass General Brigham Gene and Cell Therapy Institute, a hub of innovation dedicated to accelerating groundbreaking research, conducting clinical trials and developing FDA-approved treatments, today announced four winning project groups, consisting of seven individuals, of its second annual Spark Grant program. The program was created to fund gene and cell therapy projects by Mass General Brigham investigators that demonstrate tangible advancements towards clinical applications and commercialization outcomes, including licensing, partnerships, or new company creation. A total of $1,150,000 ...
CLEVELAND—Researchers at Case Western Reserve University have identified a new target to treat atherosclerosis, a condition where plaque clogs arteries and causes major cardiac issues, including stroke and heart attack.
In a new study, published in the journal Cell Reports, they identified an inflammation-reducing molecule—called itaconate (ITA)—that could be the foundation of a new approach to treat such a common and deadly disease.
Heart disease is the leading cause of death for ...
A research team led by Dr. Jung-Je Woo at the Gwangju Clean Energy Research Center of the Korea Institute of Energy Research (KIER) has successfully developed a cost-effective and eco-friendly technology for recycling cathode materials* from spent lithium-ion batteries.
*Cathode Materials: Materials that play a crucial role in generating electricity by storing and releasing lithium ions during battery charging and discharging.
With the recent rise in electric vehicles and mobile devices, managing spent batteries has become a critical global challenge. By 2040, the number of decommissioned electric vehicles is expected to exceed 40 million*, leading ...