ChatGPT scores nearly 50 per cent on board certification practice test for ophthalmology, study shows

AI tool scored more than 10 per cent higher one month later

2023-04-27

(Press-News.org) A study of ChatGPT found the artificial intelligence tool answered less than half of the test questions correctly from a study resource commonly used by physicians when preparing for board certification in ophthalmology.

The study, published in JAMA Ophthalmology and led by St. Michael’s Hospital, a site of Unity Health Toronto, found ChatGPT correctly answered 46 per cent of questions when initially conducted in Jan. 2023. When researchers conducted the same test one month later, ChatGPT scored more than 10 per cent higher.

The potential of AI in medicine and exam preparation has garnered excitement since ChatGPT became publicly available in Nov. 2022. It’s also raising concern for the potential of incorrect information and cheating in academia. ChatGPT is free, available to anyone with an internet connection, and works in a conversational manner.

“ChatGPT may have an increasing role in medical education and clinical practice over time, however it is important to stress the responsible use of such AI systems,” said Dr. Rajeev H. Muni, principal investigator of the study and a researcher at the Li Ka Shing Knowledge Institute at St. Michael’s. “ChatGPT as used in this investigation did not answer sufficient multiple choice questions correctly for it to provide substantial assistance in preparing for board certification at this time.”

Researchers used a dataset of practice multiple choice questions from the free trial of OphthoQuestions, a common resource for board certification exam preparation. To ensure ChatGPT’s responses were not influenced by concurrent conversations, entries or conversations with ChatGPT were cleared prior to inputting each question and a new ChatGPT account was used. Questions that used images and videos were not included because ChatGPT only accepts text input.

Of 125 text-based multiple-choice questions, ChatGPT answered 58 (46 per cent) questions correctly when the study was first conducted in Jan. 2023. Researchers repeated the analysis on ChatGPT in Feb. 2023, and the performance improved to 58 per cent.

“ChatGPT is an artificial intelligence system that has tremendous promise in medical education. Though it provided incorrect answers to board certification questions in ophthalmology about half the time, we anticipate that ChatGPT’s body of knowledge will rapidly evolve,” said Dr. Marko Popovic, a co-author of the study and a resident physician in the Department of Ophthalmology and Vision Sciences at the University of Toronto.

ChatGPT closely matched how trainees answer questions, and selected the same multiple-choice response as the most common answer provided by ophthalmology trainees 44 per cent of the time. ChatGPT selected the multiple-choice response that was least popular among ophthalmology trainees 11 per cent of the time, second least popular 18 per cent of the time, and second most popular 22 per cent of the time.

“ChatGPT performed most accurately on general medicine questions, answering 79 per cent of them correctly. On the other hand, its accuracy was considerably lower on questions for ophthalmology subspecialties. For instance, the chatbot answered 20 per cent of questions correctly on oculoplastics and zero per cent correctly from the subspecialty of retina. The accuracy of ChatGPT will likely improve most in niche subspecialties in the future,” said Andrew Mihalache, lead author of the study and undergraduate student at Western University.

END

ELSE PRESS RELEASES FROM THIS DATE:

Ecology: Over 64% of suitable elephant habitat lost across Asia since 1700

2023-04-27

Habitats suitable for Asian elephants (Elephas maximus) across Asia have decreased by over 64% – equating to 3.3 million square kilometres of land – since the year 1700, estimates a study published in Scientific Reports. The authors suggest that habitat loss from 1700, after centuries of relative stability, coincides with the colonial-era use of land and subsequent agricultural intensification in South Asia. Asian elephants live in a range of habitats including grasslands and rainforests, but with increasing human use of land and habitat loss, elephants can come ...

Citing growing evidence of harm to child health and learning ability, advocates call for faster replacement of diesel school buses

2023-04-27

Health and environmental advocates today called on communities, school boards and governments at every level to accelerate the electrification of school buses, replacing tens of thousands of diesel-powered school buses spewing toxic fumes that can seriously harm child health and interfere with learning. Led by the Canadian Partnership for Children’s Health and Environment (CPCHE), a coalition of 34 organisations made the goal of all-electric school bus fleets in Canada the central focus of this year’s national Healthy Environments for Learning Day (April 27). The joint call for urgent action by ...

Association of SARS-CoV-2 vaccination or infection with Bell palsy

2023-04-27

About The Study: This systematic review and meta-analysis suggests a higher incidence of Bell palsy (BP) among SARS-CoV-2–vaccinated versus placebo groups. The occurrence of BP did not differ significantly between recipients of the Pfizer/BioNTech versus Oxford/AstraZeneca vaccines. SARS-CoV-2 infection posed a significantly greater risk for BP than SARS-CoV-2 vaccination. Authors: Amir Kheradmand, M.D., of Johns Hopkins University in Baltimore, and Mehran Rahimlou, Ph.D., of the Zanjan University of Medical Sciences in Zanjan, Iran, are the corresponding authors. To access the embargoed study: Visit our For The Media website at this link https://media.jamanetwork.com/ (doi:10.1001/jamaoto.2023.0160) Editor’s ...

Performance of an artificial intelligence chatbot in ophthalmic knowledge assessment

2023-04-27

About The Study: In this study that included 125 text-based multiple-choice questions provided by the OphthoQuestions free trial for ophthalmic board certification examination preparation, ChatGPT answered approximately half of the questions correctly. Medical professionals and trainees should appreciate the advances of AI in medicine while acknowledging that ChatGPT as used in this investigation did not answer sufficient multiple-choice questions correctly for it to provide substantial assistance in preparing for board certification at this time. Authors: Rajeev H. Muni, M.D., M.Sc., of St. Michael’s Hospital/Unity Health Toronto in Toronto, is the corresponding ...

Comparison between ChatGPT and Google search as sources of postoperative patient instructions

2023-04-27

About The Study: The findings of this study suggest that ChatGPT provides postoperative instructions that are helpful for patients with a fifth-grade reading level or different health literacy levels. However, ChatGPT generated instructions scored lower in understandability, actionability, and procedure-specific content than Google Search– and institution-specific instructions. Authors: Noel Ayoub, M.D., M.B.A., of the Stanford University School of Medicine in Stanford, California, is the corresponding author. To access the embargoed study: Visit our For The Media website at this link https://media.jamanetwork.com/ (doi:10.1001/jamaoto.2023.0704) Editor’s ...

Arthropods in high-diversity forests contribute to improved productivity

2023-04-27

An international team of researchers led by Prof. MA Keping from the Institute of Botany of the Chinese Academy of Sciences (IBCAS) has shown that forests with higher tree species richness tend to have greater arthropod diversity. The researchers showed that higher tree diversity promotes productivity through the suppression of herbivores by enemy arthropods. These findings, published recently in Nature Ecology & Evolution, underscore the importance of arthropod diversity as a mediator of the effects of tree diversity ...

Molecular autopsy sheds light on cause of sudden death of a child with COVID-19

2023-04-27

Researchers from Tokyo Medical and Dental University (TMDU) utilize an advanced DNA sequencing technique to reveal the factors that contributed to the death of a 5-year-old child with COVID-19 Tokyo, Japan – Sudden, unexplained child mortality is a tragedy; determining the cause of death is important for improving healthcare and providing loved ones with closure. Now, researchers from Japan have reported the use of an advanced DNA sequencing technique, whole-exome sequencing (WES), to determine why a young child died after a relatively mild infection. In a study published ...

Bioinformatics specialists in Saarbrücken explore the molecular mechanisms of ageing

2023-04-27

A team led by bioinformatics experts Andreas Keller and Fabian Kern from Saarland University together with researchers at Stanford University have gained new insights into manifestations of ageing at the molecular level. They found that the process of reading genetic information does not run as smoothly in older individuals as it does in younger ones. These changes in the transcription process are due to particular RNA molecules that influence the activity of individual genes and thus determine which proteins the body produces – physiological ...

University of Cincinnati research examines the role of genetics in opioid use disorder

2023-04-27

New research out of the University of Cincinnati examines the association between genetics and the presence of opioid use disorder (OUD). The study identified six single nucleotide polymorphisms (SNPs) or genetic variants that are linked to OUD. The study was published in the journal Clinical Pharmacology & Therapeutics. “We are trying to identify some of the genetic variants that might play into OUD,” says Caroline Freiermuth, MD, associate professor in the Department of Emergency ...

Light-based computing scheme reduces power needed to mine cryptocurrencies

2023-04-27

WASHINGTON — Researchers have developed a new light-based computing scheme that uses a photonic integrated circuit to reduce the energy necessary for cryptocurrency and blockchain applications. Mining cryptocurrencies like Bitcoin—a process of verifying transactions and adding new cryptocurrency to the blockchain—consumes up to 1% of the world’s energy. This energy expenditure is expected to grow as cryptocurrency and blockchain applications become increasingly mainstream. Cryptocurrencies are digital currencies created using encryption algorithms. These alternative currencies require ...

LAST 30 PRESS RELEASES:

Mathematical model illuminates how environment impacts life choices of salmon

Houston Methodist researchers shed light on increased rates of severe human infections caused by Streptococcus subspecies

Change of ownership in home health agencies may lead to increased Medicare spending and reduced staffing levels, according to UTHealth Houston research