(Press-News.org) OAK BROOK, Ill. – Researchers evaluating the performance of ChatGPT-4 Vision found that the model performed well on text-based radiology exam questions but struggled to answer image-related questions accurately. The study’s results were published today in Radiology, a journal of the Radiological Society of North America (RSNA).
Chat GPT-4 Vision is the first version of the large language model that can interpret both text and images.
“ChatGPT-4 has shown promise for assisting radiologists in tasks such as simplifying patient-facing radiology reports and identifying the appropriate protocol for imaging exams,” said Chad Klochko, M.D., musculoskeletal radiologist and artificial intelligence (AI) researcher at Henry Ford Health in Detroit, Michigan. “With image processing capabilities, GPT-4 Vision allows for new potential applications in radiology.”
For the study, Dr. Klochko’s research team used retired questions from the American College of Radiology’s Diagnostic Radiology In-Training Examinations, a series of tests used to benchmark the progress of radiology residents. After excluding duplicates, the researchers used 377 questions across 13 domains, including 195 questions that were text-only and 182 that contained an image.
GPT-4 Vision answered 246 of the 377 questions correctly, achieving an overall score of 65.3%. The model correctly answered 81.5% (159) of the 195 text-only queries and 47.8% (87) of the 182 questions with images.
“The 81.5% accuracy for text-only questions mirrors the performance of the model’s predecessor,” he said. “This consistency on text-based questions may suggest that the model has a degree of textual understanding in radiology.”
Genitourinary radiology was the only subspecialty for which GPT-4 Vision performed better on questions with images (67%, or 10 of 15) than text-only questions (57%, or 4 of 7). The model performed better on text-only questions in all other subspecialties.
The model performed best on image-based questions in the chest and genitourinary subspecialties, correctly answering 69% and 67% of the image-containing questions, respectively. The model performed lowest on image-containing questions in the nuclear medicine domain, correctly answering only 2 of 10 questions.
The study also evaluated the impact of various prompts on the performance of GPT-4 Vision.
Original: You are taking a radiology board exam. Images of the questions will be uploaded. Choose the correct answer for each question.
Basic: Choose the single best answer in the following retired radiology board exam question.
Short instruction: This is a retired radiology board exam question to gauge your medical knowledge. Choose the single best answer letter and do not provide any reasoning for your answer.
Long instruction: You are a board-certified diagnostic radiologist taking an examination. Evaluate each question carefully and if the question additionally contains an image, please evaluate the image carefully in order to answer the question. Your response must include a single best answer choice. Failure to provide an answer choice will count as incorrect.
Chain of thought: You are taking a retired board exam for research purposes. Given the provided image, think step by step for the provided question.
Although the model correctly answered 183 of 265 questions with a basic prompt, it declined to answer 120 questions, most of which contained an image.
“The phenomenon of declining to answer questions was something we hadn’t seen in our initial exploration of the model,” Dr. Klochko said.
The short instruction prompt yielded the lowest accuracy (62.6%).
On text-based questions, chain-of-thought prompting outperformed long instruction by 6.1%, basic by 6.8%, and original prompting style by 8.9%. There was no evidence to suggest performance differences between any two prompts on image-based questions.
“Our study showed evidence of hallucinatory responses when interpreting image findings,” Dr. Klochko said. “We noted an alarming tendency for the model to provide correct diagnoses based on incorrect image interpretations, which could have significant clinical implications.”
Dr. Klochko said his study’s findings underscore the need for more specialized and rigorous evaluation methods to assess large language model performance in radiology tasks.
“Given the current challenges in accurately interpreting key radiologic images and the tendency for hallucinatory responses, the applicability of GPT-4 Vision in information-critical fields such as radiology is limited in its current state,” he said.
###
“Performance of GPT-4 with Vision on Text- and Image-based ACR Diagnostic Radiology In-Training Examination Questions.” Collaborating with Dr. Klochko were Nolan Hayden, M.D., Spencer Gilbert, B.S., Laila M. Poisson, Ph.D., and Brent Griffith, M.D.
Radiology is edited by Linda Moy, M.D., New York University, New York, N.Y., and owned and published by the Radiological Society of North America, Inc. (https://pubs.rsna.org/journal/radiology)
RSNA is an association of radiologists, radiation oncologists, medical physicists and related scientists promoting excellence in patient care and health care delivery through education, research and technologic innovation. The Society is based in Oak Brook, Illinois. (RSNA.org)
For patient-friendly information on medical imaging, visit RadiologyInfo.org.
END
A new study reveals nuanced findings on the neuropsychiatric risks of prenatal cannabis exposure. The research found a slight increase in the risk of ADHD and a heightened vulnerability to cannabis use in offspring. These results highlight the need for continued caution and further investigation into the long-term effects of cannabis use during pregnancy.
A new study led by Prof. Ilan Matok and Hely Bassalov PharmD from the Department of Clinical Pharmacy at the School of Pharmacy in the Faculty of Medicine at Hebrew University in collaboration ...
Gun-free zones have often been blamed for making schools, malls and other public areas more attractive to shooters; however, there have been no quantitative studies examining those claims. Now, in a first of its kind study published in The Lancet Regional Health Americas, researchers at UC Davis Health and other institutions have shown that gun-free zones may, in fact, reduce the risk of mass shootings.
"Our most significant finding is that gun-free zones don't attract active shooters,” said the study’s first author, Paul Reeping, ...
Darwin was puzzled by cooperation in nature—it ran directly against natural selection and the notion of survival of the fittest. But over the past decades, evolutionary mathematicians have used game theory to better understand why mutual cooperation persists when evolution should favour self-serving cheaters.
At a basic level, cooperation flourishes when the costs to cooperation are low or the benefits large. When cooperation becomes too costly, it disappears—at least in the realm of pure mathematics. ...
East Hanover, NJ – September 3, 2024 – Researchers at Kessler Foundation have published a new clinical protocol examining the combination of aerobic exercise and cognitive rehabilitation to improve learning and memory in individuals with multiple sclerosis (MS) who have mobility disability. The article, “Rationale and methodology for examining the combination of aerobic exercise and cognitive rehabilitation on new learning and memory in persons with multiple sclerosis and mobility disability: Protocol for a randomized controlled trial,” was published online and will appear in print in Contemporary Clinical Trials, ...
$75 million NIH grant could lead to the first effective drugs for a condition with few treatment options
A clinical trial that will test three drugs concurrently, and could include more, represents new hope for patients with progressive supranuclear palsy (PSP), an incurable neurodegenerative disorder that usually kills within seven years after symptoms start.
Researchers hope the trial, which will be led by UC San Francisco and conducted at up to 50 sites nationwide, will lead to the development of new therapies. There are currently no drugs to stall the disease’s deadly progression.
The ...
The Mass General Brigham Gene and Cell Therapy Institute (GCTI) today announced it has launched the RNA Therapeutics Core, a first-of-its-kind, state-of-the-art facility and resource to advance the use of RNA technologies within and beyond the Mass General Brigham research ecosystem. This new Core is dedicated to accelerating the exploration of novel therapeutic targets to effectively translate RNA-based medicines into clinical practice by leveraging advanced RNA vectors and delivery systems.
Until now, a Core of this kind has not existed within an academic setting. With this launch, the RNA Therapeutics Core enables ...
Valley fever is an emerging fungal disease in the western United States that most often causes flu-like symptoms, but can also cause dangerous or even deadly complications. By analyzing data on reported cases of Valley fever in California, which have increased dramatically over the last two decades, researchers from University of California San Diego and University of California, Berkeley, have identified seasonal patterns that could help individuals and public health officials better prepare for future surges in Valley fever cases. The findings also have important implications for how the changing climate can exacerbate the threat of infectious diseases. The findings are published in The ...
Natural selection is an important evolutionary force that enables humans to adapt to new environments and fight disease-causing pathogens. However, the unique footprints of natural selection in our genome can be buried beneath those left by other evolutionary forces. Thus, by leveraging information about multiple evolutionary forces, researchers can identify signatures of natural selection in the human genome, and ultimately determine its role in human adaptation and disease.
Low-cost DNA sequencing has ...
PULLMAN, Wash. -- When young adults first go off to college, more communication with parents generally leads to better relationships, but parents should avoid always initiating it, according to a study led by Washington State University researchers.
In a paper published in the journal Emerging Adulthood, WSU Assistant Professor Jennifer Duckworth and co-authors found that phone, text, video or in-person communication made first-year students feel better about the relationship with their parents. Students also felt better about the relationship when parents offered support or advice, and when they discussed important topics, such as studying and friendships. However, researchers found ...
A modeling study suggests that one-sided interspecies cooperation can spontaneously emerge and persist over time, despite only one species benefitting. Evolutionary game theory, and the prisoner’s dilemma in particular, are often used to model the evolution of cooperation within a single species. In the prisoner’s dilemma, both parties benefit by cooperating, but the greatest benefit is earned by a defector who plays with a cooperator. The temptation to cheat tends to push players towards defection, ...
