(Press-News.org) Large language models (LLMs) are neural network–based computer programs that use a detailed statistical understanding of written language to perform many tasks, including text generation, summarization, software development, and prediction. However, LLMs can produce text that, while may seem correct, is not fact-based. This study investigates whether a popular LLM, ChatGPT-3.5, could produce high-quality, accurate, and bias-free summaries of medical research abstracts and determine the relevance of various journals and their articles to different medical specialties. Ten articles published in 2022 (not yet “seen” by ChatGPT, as ChatGPT was trained on data before 2022) were randomly sampled from each of 14 selected journals. ChatGPT was then prompted to summarize the abstract, “self-reflect” on the quality, accuracy, and bias of its own summaries, and evaluate its performance in classifying articles’ and journals' relevance to various areas of medicine (cardiology, pulmonary medicine, family medicine, internal medicine, public health, primary care, neurology, psychiatry, obstetrics and gynecology, and general surgery).
The quality of summaries, relevant classification of journal and article to medical specialty were also assessed by human physicians. The results include a total of 140 abstract summaries across 14 journals. ChatGPT produced summaries that were 70% shorter than the abstracts. The summaries were rated as high quality, high accuracy, and low bias by both ChatGPT and physician reviewers. Serious inaccuracies occurred in only four of the 140 summaries. Minor inaccuracies were noted in 20 of 140 articles and mostly related to the introduction of ambiguity in meaning or summarization of details that would have provided additional content but not completely changed the meaning. ChatGPT was able to classify journals to relevant medical specialties but was much less able to classify specific articles to relevant medical specialties. The summaries were found to have rare—but important—inaccuracies that preclude them from being considered a definitive source of truth.
What We Know: The availability of medical knowledge is increasing. However, due to the demands of their jobs, clinicians have little time to review academic literature, even within their own specialty. Large language models (eg, ChatGPT) could be helpful and save time, but they are not always accurate as they can include bias from their training models and the human feedback which reinforces their learning, and sometimes include information that is not fact-based.
What This Study Adds: Clinicians are strongly cautioned against solely relying on ChatGPT-based summaries to understand study methods and study results, especially in high-risk situations. Critical medical decisions should—for obvious reasons—remain based on a full evaluation of the full text of articles in context with available evidence from meta-analyses and professional guidelines. However, this study suggests ChatGPT can be useful as a screening tool to help busy clinicians and scientists more rapidly evaluate whether further review of an article is likely to be worthwhile.
Quality, Accuracy, and Bias in ChatGPT-Based Summarization of Medical Abstracts
Daniel J. Parente, MD, PhD, et al
Department of Family Medicine and Community Health, University of Kansas Medical Center, Kansas City, Kansas
PRE-EMBARGO LINK (Link expires at 5 p.m. EDT March 25, 2024)
PERMANENT LINK
END
ChatGPT’s potential and limits in summarizing medical research for clinicians
Large language models (LLMs) are neural network–based computer programs that use a detailed statistical understanding of written language to perform many tasks, including text generation, summarization, software development, and prediction.
2024-03-25
ELSE PRESS RELEASES FROM THIS DATE:
Pediatric health care disrupted by COVID-19 pandemic, compounded by existing barriers such as systemic racism
2024-03-25
This study explores the extent to which pediatric health care was interrupted during and as a result of the COVID-19 pandemic. This was measured based on three primary outcomes of interest: foregone care, foregone well-child or vaccination-related visits, and complete absence of well-child or vaccination-related visits. Researchers extracted data from a nationwide longitudinal survey known as CovEx (COVID Experiences Survey) that had been administered in two “waves'' to a cohort of parents of children between five and 12 years of age. Wave 1 took place October 8-November 13, 2020, and Wave 2 took place March 24-May 7, 2021, with an 82% retention rate of participants. Data was ...
Fathers’ perceptions of their teen sons’ readiness for sex linked to their likelihood to provide guidance on condom use
2024-03-25
The purpose of this study was to delve deeper into the link between fathers’ perceptions of their adolescent sons’ preparedness to engage in sexual intercourse, and how likely they were to provide their sons with guidance on using condoms correctly and consistently. Researchers recruited 191 father-son pairs from among Black and Latino residents of the South Bronx in New York City; adolescent participants ranged in age from 15 to 19 years old. Using a sequential mixed-methods explanatory design, all participants completed a confidential survey, after which a random ...
Survey on genital tucking among transgender and gender diverse individuals: Majority of respondents desire open conversation with health care providers
2024-03-25
Genital tucking is the practice of hiding or minimizing the appearance of one’s genitals and gonads. It is practiced by transgender women and gender diverse individuals who were assigned male at birth. In this study, researchers designed a 27-question survey related to such factors as the length of time individuals spent tucked; how commonly and frequently; and understandings of, concerns for, and positive/negative history of potential adverse effects. Only 23% of the 98 respondents reported discussing ...
Annual vaccination campaign with second dose protects high-risk groups from SARS-CoV-2 and may save health care costs
2024-03-25
Embargoed for release until 5:00 p.m. ET on Monday 25 March 2024
Annals of Internal Medicine Tip Sheet
@Annalsofim
Below please find summaries of new articles that will be published in the next issue of Annals of Internal Medicine. The summaries are not intended to substitute for the full articles as a source of information. This information is under strict embargo and by taking it into possession, media representatives are committing to the terms of the embargo not only on their own behalf, but also on behalf of the organization they represent.
----------------------------
1. Annual vaccination campaign with ...
DEI and antiracism curriculum must connect the classroom to the clinic to educate medical students about racism in medicine
2024-03-25
There's a growing awareness of the role medicine as an institution has played in creating and perpetuating health inequities facing historically marginalized groups, yet systemic racism and implicit biases continue to shape aspects of clinical practice such as care management decisions and patient communication.
Addressing medical racism has become an essential part of the medical education curriculum, though most published curricula treat health equity as a lecture topic separate from clinical practice. In this report, authors from the Emory University School of Medicine present a new approach to threading (cohesively incorporating) diversity, equity, and inclusion (DEI) and ...
A national quality improvement initiative provides insight into how to successfully implement change in primary care practices
2024-03-25
In 2015, the Agency for Healthcare Research and Quality (AHRQ) launched EvidenceNOW: Advancing Heart Health. This three-year initiative, AHRQ’s largest-ever investment in primary care research, was designed to provide external quality improvement support to help small and medium-size primary care practices implement the latest evidence and improve their delivery of cardiovascular care.
AHRQ awarded grants to primary care practices in seven regions across the U.S. to implement this initiative and to evaluate their own experiences with it, thereby accounting for regional differences in needs and existing resources. Additionally, ...
Annals of Family Medicine March/April 2024 Tip Sheet
2024-03-25
Collaboration Between Medical Department Staff and Their IT Colleagues on an EHR Optimization Project Leads to Improved Productivity
Between June and October 2021, members of Marshall University’s Department of Family and Community Health collaborated with their school’s Information Technology (IT) team to conduct a four-month, department-wide optimization project to improve their usage of the Electronic Health Record (EHR) software. The software was implemented across their institution ...
Annals of Family Medicine: Study shows Latino and Black fathers' perception of sons' sexual readiness is key to effective condom guidance
2024-03-25
Providence, R.I. -- A new study from Annals of Family Medicine reveals that a father's understanding of his son's readiness for sexual relationships plays a crucial role in the timing and effectiveness of discussions regarding condom use guidance. The findings offer valuable insights for family-focused health care providers, guiding them to encourage fathers to initiate conversations about sexual readiness and the importance of correct and consistent condom use with their adolescent sons.
In 2021, 47% of adolescent males in the 12th grade reported engaging in sexual activity, yet condom use has declined. These trends contribute to negative sexual health outcomes among ...
The future of independent primary care practices serving vulnerable populations depends on new policies that support health equity
2024-03-25
In recent years, the U.S. government has invested substantially in Federally Qualified Health Centers (FQHCs), which have become synonymous in policy discussions with primary care for the socially vulnerable. Conversely, no such investment has been made in independent practices serving socially vulnerable patients. As independent practices become less financially viable, this disparity could severely limit primary care options for socially vulnerable patients. This mixed-methods study considers the extent to which independent family physicians ...
Comparative study of type 2 diabetes medications show differences in medication acceptance, quality-of-life, insulin secretion and mortality
2024-03-25
Diabetes affects more than 1 in 10 — or more than 38 million — Americans. People with diabetes who keep their blood sugar levels in the near-normal range generally have a much lower risk of developing diabetes complications such as heart, kidney, and eye diseases. The challenge is that most people with diabetes require more than one medication to control blood sugar levels over time.
The Glycemia Reduction Approaches in Diabetes: A Comparative Effectiveness, or GRADE, Study was designed to compare ...
LAST 30 PRESS RELEASES:
Program takes aim at drinking, unsafe sex, and sexual assault on college campuses
Inability to pay for healthcare reaches record high in U.S.
Science ‘storytelling’ urgently needed amid climate and biodiversity crisis
KAIST Develops Retinal Therapy to Restore Lost Vision
Adipocyte-hepatocyte signaling mechanism uncovered in endoplasmic reticulum stress response
Mammals were adapting from life in the trees to living on the ground before dinosaur-killing asteroid
Low LDL cholesterol levels linked to reduced risk of dementia
Thickening of the eye’s retina associated with greater risk and severity of postoperative delirium in older patients
Almost one in ten people surveyed report having been harmed by the NHS in the last three years
Enhancing light control with complex frequency excitations
New research finds novel drug target for acute myeloid leukemia, bringing hope for cancer patients
New insight into factors associated with a common disease among dogs and humans
Illuminating single atoms for sustainable propylene production
New study finds Rocky Mountain snow contamination
Study examines lactation in critically ill patients
UVA Engineering Dean Jennifer West earns AIMBE’s 2025 Pierre Galletti Award
Doubling down on metasurfaces
New Cedars-Sinai study shows how specialized diet can improve gut disorders
Making moves and hitting the breaks: Owl journeys surprise researchers in western Montana
PKU Scientists simulate the origin and evolution of the North Atlantic Oscillation
ICRAFT breakthrough: Unlocking A20’s dual role in cancer immunotherapy
How VR technology is changing the game for Alzheimer’s disease
A borrowed bacterial gene allowed some marine diatoms to live on a seaweed diet
Balance between two competing nerve proteins deters symptoms of autism in mice
Use of antifungals in agriculture may increase resistance in an infectious yeast
Awareness grows of cancer risk from alcohol consumption, survey finds
The experts that can outsmart optical illusions
Pregnancy may reduce long COVID risk
Scientists uncover novel immune mechanism in wheat tandem kinase
Three University of Virginia Engineering faculty elected as AAAS Fellows
[Press-News.org] ChatGPT’s potential and limits in summarizing medical research for cliniciansLarge language models (LLMs) are neural network–based computer programs that use a detailed statistical understanding of written language to perform many tasks, including text generation, summarization, software development, and prediction.