(Press-News.org) OAK BROOK, Ill. – The latest version of ChatGPT passed a radiology board-style exam, highlighting the potential of large language models but also revealing limitations that hinder reliability, according to two new research studies published in Radiology, a journal of the Radiological Society of North America (RSNA).
ChatGPT is an artificial intelligence (AI) chatbot that uses a deep learning model to recognize patterns and relationships between words in its vast training data to generate human-like responses based on a prompt. But since there is no source of truth in its training data, the tool can generate responses that are factually incorrect.
“The use of large language models like ChatGPT is exploding and only going to increase,” said lead author Rajesh Bhayana, M.D., FRCPC, an abdominal radiologist and technology lead at University Medical Imaging Toronto, Toronto General Hospital in Toronto, Canada. “Our research provides insight into ChatGPT’s performance in a radiology context, highlighting the incredible potential of large language models, along with the current limitations that make it unreliable.”
ChatGPT was recently named the fastest growing consumer application in history, and similar chatbots are being incorporated into popular search engines like Google and Bing that physicians and patients use to search for medical information, Dr. Bhayana noted.
To assess its performance on radiology board exam questions and explore strengths and limitations, Dr. Bhayana and colleagues first tested ChatGPT based on GPT-3.5, currently the most commonly used version. The researchers used 150 multiple-choice questions designed to match the style, content and difficulty of the Canadian Royal College and American Board of Radiology exams.
The questions did not include images and were grouped by question type to gain insight into performance: lower-order (knowledge recall, basic understanding) and higher-order (apply, analyze, synthesize) thinking. The higher-order thinking questions were further subclassified by type (description of imaging findings, clinical management, calculation and classification, disease associations).
The performance of ChatGPT was evaluated overall and by question type and topic. Confidence of language in responses was also assessed.
The researchers found that ChatGPT based on GPT-3.5 answered 69% of questions correctly (104 of 150), near the passing grade of 70% used by the Royal College in Canada. The model performed relatively well on questions requiring lower-order thinking (84%, 51 of 61), but struggled with questions involving higher-order thinking (60%, 53 of 89). More specifically, it struggled with higher-order questions involving description of imaging findings (61%, 28 of 46), calculation and classification (25%, 2 of 8), and application of concepts (30%, 3 of 10). Its poor performance on higher-order thinking questions was not surprising given its lack of radiology-specific pretraining.
GPT-4 was released in March 2023 in limited form to paid users, specifically claiming to have improved advanced reasoning capabilities over GPT-3.5.
In a follow-up study, GPT-4 answered 81% (121 of 150) of the same questions correctly, outperforming GPT-3.5 and exceeding the passing threshold of 70%. GPT-4 performed much better than GPT-3.5 on higher-order thinking questions (81%), more specifically those involving description of imaging findings (85%) and application of concepts (90%).
The findings suggest that GPT-4’s claimed improved advanced reasoning capabilities translate to enhanced performance in a radiology context. They also suggest improved contextual understanding of radiology-specific terminology, including imaging descriptions, which is critical to enable future downstream applications.
“Our study demonstrates an impressive improvement in performance of ChatGPT in radiology over a short time period, highlighting the growing potential of large language models in this context,” Dr. Bhayana said.
GPT-4 showed no improvement on lower-order thinking questions (80% vs 84%) and answered 12 questions incorrectly that GPT-3.5 answered correctly, raising questions related to its reliability for information gathering.
“We were initially surprised by ChatGPT’s accurate and confident answers to some challenging radiology questions, but then equally surprised by some very illogical and inaccurate assertions,” Dr. Bhayana said. “Of course, given how these models work, the inaccurate responses should not be particularly surprising.”
ChatGPT’s dangerous tendency to produce inaccurate responses, termed hallucinations, is less frequent in GPT-4 but still limits usability in medical education and practice at present.
Both studies showed that ChatGPT used confident language consistently, even when incorrect. This is particularly dangerous if solely relied on for information, Dr. Bhayana notes, especially for novices who may not recognize confident incorrect responses as inaccurate.
“To me, this is its biggest limitation. At present, ChatGPT is best used to spark ideas, help start the medical writing process and in data summarization. If used for quick information recall, it always needs to be fact-checked,” Dr. Bhayana said.
###
“Performance of ChatGPT on a Radiology Board-style Examination: Insights into Current Strengths and Limitations” and “GPT-4 in Radiology: Improvements in Advanced Reasoning.” Collaborating with Dr. Bhayana were Satheesh Krishna, M.D., and Robert R. Bleakney, M.D.
In 2023, Radiology is celebrating its 100th anniversary with 12 centennial issues, highlighting Radiology’s legacy of publishing exceptional and practical science to improve patient care.
Radiology is edited by Linda Moy, M.D., New York University, New York, N.Y., and owned and published by the Radiological Society of North America, Inc. (https://pubs.rsna.org/journal/radiology)
RSNA is an association of radiologists, radiation oncologists, medical physicists and related scientists promoting excellence in patient care and health care delivery through education, research and technologic innovation. The Society is based in Oak Brook, Illinois. (RSNA.org)
For patient-friendly information on professions in radiology, visit RadiologyInfo.org.
END
HOUSTON – The University of Texas MD Anderson Cancer Center was awarded more than $5.7 million in grants from Break Through Cancer to support collaborative research teams working to discover novel molecular targets to eradicate minimal residual disease in acute myeloid leukemia (AML) and to treat clonal hematopoiesis, a precursor to AML.
MD Anderson received $2.7 million to fund research for the Targeting Clonal Hematopoiesis to Prevent AML TeamLab and $3 million for the Eradicating Minimal Residual Disease in AML TeamLab. The projects expand upon work initiated within MD Anderson’s Myelodysplastic Syndromes and Acute Myeloid ...
AUGUSTA, Ga. (May 16, 2023) – Magnetic stimulation of a group of nerves key to how our gut and brain communicate may help correct the conversation that goes awry in painful, debilitating diabetic gastroparesis, researchers say.
Patients come to Amol Sharma, MD, because their stomachs constantly hurt, they are always nauseous and they can’t or won’t eat or drink. Sometimes they can’t get out of the hospital because of nausea and vomiting.
“Gastroparesis is suspected in about 2% of the population, which is the about the population of Missouri, but only confirmed in .2% ...
The Austronesian language family is one of the largest in the world, comprising over 1,200 languages spoken from Madagascar to Hawaii. Dang Liu, Albert Min-Shan Ko and Mark Stoneking collected genome-wide data from 55 individuals from seven Taiwanese Austronesian groups and two Han-Taiwanese groups to study the genetic structure of Taiwan, the point of origin for all Austronesian-speaking peoples. There are over 20 different Indigenous groups in Taiwan, divided into “highland” and “lowland” peoples. Many lowland peoples have intermarried with Han people, and their languages are endangered or extinct. ...
Chinese electric vehicles (EVs) drive larger emissions reductions over time, due to increased operating efficiency and a greener electricity mix, according to a study. More than 10% of Chinese car sales are now electric, but the full life cycle of EVs still creates carbon emissions. Shaojun Zhang and colleagues conducted “cradle-to-grave” life cycle assessments for EVs in 2015 and 2020, including fuel-cycle and material-cycle phases, and compiled life-cycle projections for 2030. The authors considered factors including sources of electricity, vehicle fuel economy, major automotive metals, and battery ...
INDIANAPOLIS – One out of four adults, age 65 or older, falls every year in the U.S. Falls cause approximately 36,000 deaths annually in this age group, making it the leading cause of death from injury for older adults in the U.S.
A new study, led by Regenstrief Institute Research Scientist Briana Sprague, PhD, examines whether cognitive training – specifically, speed of processing, memory and reasoning training -- can lower the risk of falling. Significantly, the researchers found no effects of the training on likelihood of falling for those at low risk of falling. Dr. Sprague also is a faculty member at Indiana ...
An international research team led by Faviel A. López-Romero of the University of Vienna investigated how the jaw shape of sharks has changed over the course of evolution. Their conclusion: in the most widespread shark species, the jaws show relatively little variation in shape over millions of years; most variable jaws were found for deep-sea sharks. The results of this study were published in the journal Communications Biology.
One of the most prominent traits in sharks is the shape of their lower jaws, which bear also impressive teeth. With their jaws, sharks are able to feed on a wide variety of prey, which also places them among the Ocean's top predators. ...
PLYMOUTH MEETING, PA [May 16, 2023] — The National Comprehensive Cancer Network® (NCCN®) today announced the appointment of Katy Winckworth-Prejsnar, MPH, NCCN’s Senior Manager of Global Policy and Strategic Alliances, as Co-Chair of the Global Health Council (GHC)’s Non-Communicable Diseases (NCD) Roundtable. In this role, Winckworth-Prejsnar will help drive coordination between organizations worldwide that are working to improve policy and outcomes for cancer and other global health concerns. She will serve alongside Co-Chair Eliana Monteforte, Director of Special Projects, GHC.
“NCDs—including ...
Maybe you too have bought into the idea that men with numerous sexual partners are actually admired, while women with the same are condemned – the so-called sexual double standard. But that turns out to be a myth, according to a new survey.
“We haven’t found that women are subjected to the traditional double standards,” says Leif Edward Ottesen Kennair, a professor at the Norwegian University of Science and Technology's (NTNU) Department of Psychology.
On the contrary, men are judged a little ...
Every year, between 1,200 and 1,500 patients suffer a cardiac arrest in Norwegian hospitals. Rapid and sound treatment is absolutely essential in helping these patients survive.
Even if a patient suffers a cardiac arrest within the hospital's four walls, the prognosis is poor. Only one in four survives.
However, a new study suggests that easily available informaiton from the patient's own ECG could change the outcome.
Treatment the same for everyone
When a heart stops, doctors have to hurry, and the life-saving effort can last a long time.
But doctors rarely have a good idea of what the ...
BGI Genomics recently joined a mission business to South America in April 2023. Given that this continent ranks fourth in area and fifth in population worldwide, the economic and healthcare enhancement potential of this continent is compelling.
Every South American country faces different healthcare challenges and priorities. Still, the promise of precision medicine is clear: It offers an opportunity to shift the delivery of care from a legacy one-size-fits-all approach to applying the right treatment for the right patient at the right time.
To help deliver on precision medicine's potential, BGI Genomics considers genetic ...
