PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

AI chatbots can run with medical misinformation, study finds, highlighting the need for stronger safeguards

Researchers suggest ways to reduce risk of chatbots repeating false medical details

2025-08-06
(Press-News.org) New York, NY [August 6, 2025] — A new study by researchers at the Icahn School of Medicine at Mount Sinai finds that widely used AI chatbots are highly vulnerable to repeating and elaborating on false medical information, revealing a critical need for stronger safeguards before these tools can be trusted in health care.

The researchers also demonstrated that a simple built-in warning prompt can meaningfully reduce that risk, offering a practical path forward as the technology rapidly evolves. Their findings were detailed in the August 2 online issue of Communications Medicine [https://doi.org/10.1038/s43856-025-01021-3].

As more doctors and patients turn to AI for support, the investigators wanted to understand whether chatbots would blindly repeat incorrect medical details embedded in a user’s question, and whether a brief prompt could help steer them toward safer, more accurate responses.

“What we saw across the board is that AI chatbots can be easily misled by false medical details, whether those errors are intentional or accidental,” says lead author Mahmud Omar, MD, who is an independent consultant with the research team. “They not only repeated the misinformation but often expanded on it, offering confident explanations for non-existent conditions. The encouraging part is that a simple, one-line warning added to the prompt cut those hallucinations dramatically, showing that small safeguards can make a big difference.”

The team created fictional patient scenarios, each containing one fabricated medical term such as a made-up disease, symptom, or test, and submitted them to leading large language models. In the first round, the chatbots reviewed the scenarios with no extra guidance provided. In the second round, the researchers added a one-line caution to the prompt, reminding the AI that the information provided might be inaccurate.

Without that warning, the chatbots routinely elaborated on the fake medical detail, confidently generating explanations about conditions or treatments that do not exist. But with the added prompt, those errors were reduced significantly.

“Our goal was to see whether a chatbot would run with false information if it was slipped into a medical question, and the answer is yes,” says co-corresponding senior author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai. “Even a single made-up term could trigger a detailed, decisive response based entirely on fiction. But we also found that the simple, well-timed safety reminder built into the prompt made an important difference, cutting those errors nearly in half. That tells us these tools can be made safer, but only if we take prompt design and built-in safeguards seriously.”

The team plans to apply the same approach to real, de-identified patient records and test more advanced safety prompts and retrieval tools. They hope their “fake-term” method can serve as a simple yet powerful tool for hospitals, tech developers, and regulators to stress-test AI systems before clinical use.

“Our study shines a light on a blind spot in how current AI tools handle misinformation, especially in health care,” says co-corresponding senior author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai and the Chief AI Officer for the Mount Sinai Health System. “It underscores a critical vulnerability in how today’s AI systems deal with misinformation in health settings. A single misleading phrase can prompt a confident yet entirely wrong answer. The solution isn’t to abandon AI in medicine, but to engineer tools that can spot dubious input, respond with caution, and ensure human oversight remains central. We’re not there yet, but with deliberate safety measures, it’s an achievable goal.”

The paper is titled “Large Language Models Demonstrate Widespread Hallucinations for Clinical Decision Support: A Multiple Model Assurance Analysis.”

The study’s authors, as listed in the journal, are Mahmud Omar, Vera Sorin, Jeremy D. Collins, David Reich, Robert Freeman, Alexander Charney, Nicholas Gavin, Lisa Stump, Nicola Luigi Bragazzi, Girish N. Nadkarni, and Eyal Klang.

This work was supported in part through the computational and data resources and staff expertise provided by Scientific Computing and Data at the Icahn School of Medicine at Mount Sinai and supported by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. The research was also supported by the Office of Research Infrastructure of the National Institutes of Health under award number S10OD026880 and S10OD030463.

-####-

About Mount Sinai's Windreich Department of AI and Human Health  

Led by Girish N. Nadkarni, MD, MPH—an international authority on the safe, effective, and ethical use of AI in health care—Mount Sinai’s Windreich Department of AI and Human Health is the first of its kind at a U.S. medical school, pioneering transformative advancements at the intersection of artificial intelligence and human health. 

The Department is committed to leveraging AI in a responsible, effective, ethical, and safe manner to transform research, clinical care, education, and operations. By bringing together world-class AI expertise, cutting-edge infrastructure, and unparalleled computational power, the department is advancing breakthroughs in multi-scale, multimodal data integration while streamlining pathways for rapid testing and translation into practice. 

The Department benefits from dynamic collaborations across Mount Sinai, including with the Hasso Plattner Institute for Digital Health at Mount Sinai—a partnership between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System—which complements its mission by advancing data-driven approaches to improve patient care and health outcomes. 

At the heart of this innovation is the renowned Icahn School of Medicine at Mount Sinai, which serves as a central hub for learning and collaboration. This unique integration enables dynamic partnerships across institutes, academic departments, hospitals, and outpatient centers, driving progress in disease prevention, improving treatments for complex illnesses, and elevating quality of life on a global scale. 

In 2024, the Department's innovative NutriScan AI application, developed by the Mount Sinai Health System Clinical Data Science team in partnership with Department faculty, earned Mount Sinai Health System the prestigious Hearst Health Prize. NutriScan is designed to facilitate faster identification and treatment of malnutrition in hospitalized patients. This machine learning tool improves malnutrition diagnosis rates and resource utilization, demonstrating the impactful application of AI in health care. 

For more information on Mount Sinai's Windreich Department of AI and Human Health, visit: ai.mssm.edu 

 

About the Hasso Plattner Institute at Mount Sinai 

At the Hasso Plattner Institute for Digital Health at Mount Sinai, the tools of data science, biomedical and digital engineering, and medical expertise are used to improve and extend lives. The Institute represents a collaboration between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System.  

Under the leadership of Girish Nadkarni, MD, MPH, who directs the Institute, and Professor Lothar Wieler, a globally recognized expert in public health and digital transformation, they jointly oversee the partnership, driving innovations that positively impact patient lives while transforming how people think about personal health and health systems. 

The Hasso Plattner Institute for Digital Health at Mount Sinai receives generous support from the Hasso Plattner Foundation. Current research programs and machine learning efforts focus on improving the ability to diagnose and treat patients. 

 

About the Icahn School of Medicine at Mount Sinai

The Icahn School of Medicine at Mount Sinai is internationally renowned for its outstanding research, educational, and clinical care programs. It is the sole academic partner for the seven member hospitals* of the Mount Sinai Health System, one of the largest academic health systems in the United States, providing care to New York City’s large and diverse patient population.  

The Icahn School of Medicine at Mount Sinai offers highly competitive MD, PhD, MD-PhD, and master’s degree programs, with enrollment of more than 1,200 students. It has the largest graduate medical education program in the country, with more than 2,600 clinical residents and fellows training throughout the Health System. Its Graduate School of Biomedical Sciences offers 13 degree-granting programs, conducts innovative basic and translational research, and trains more than 560 postdoctoral research fellows. 

Ranked 11th nationwide in National Institutes of Health (NIH) funding, the Icahn School of Medicine at Mount Sinai is among the 99th percentile in research dollars per investigator according to the Association of American Medical Colleges.  More than 4,500 scientists, educators, and clinicians work within and across dozens of academic departments and multidisciplinary institutes with an emphasis on translational research and therapeutics. Through Mount Sinai Innovation Partners (MSIP), the Health System facilitates the real-world application and commercialization of medical breakthroughs made at Mount Sinai.

------------------------------------------------------- 

* Mount Sinai Health System member hospitals: The Mount Sinai Hospital; Mount Sinai Brooklyn; Mount Sinai Morningside; Mount Sinai Queens; Mount Sinai South Nassau; Mount Sinai West; and New York Eye and Ear Infirmary of Mount Sinai


 


 

 

 

END



ELSE PRESS RELEASES FROM THIS DATE:

Nanoparticles for rheumatoid arthritis prevention, flare control

2025-08-06
As a chronic condition, rheumatoid arthritis (RA) can’t be cured, so treatment focuses on managing the disease and controlling its progression. Although current treatments help control RA symptoms in most people, they cannot prevent the onset of RA or painful flare-ups. Now, researchers publishing in ACS Central Science have developed nanoparticles that could slow disease progression and reduce flare severity, based on results from tests with human blood and mice models with RA-like disease. For a person diagnosed with RA, their immune system attacks tissue that makes up the joints, causing inflammation, swelling and pain. ...

Small-world networks can mitigate ESG controversies for multinationals, suggests new study

2025-08-06
Multinational enterprises (MNEs) can negate supplier-induced environmental, social and governance (ESG) controversies by setting up ‘small-world’ networks, according to new research co-authored by Bayes Business School. Globalising supply chains has become a dominant strategy for MNEs. Benefits include lower costs of labour and distribution, and greater scope for innovation. However, long-distance operations also carry associated risks. Contrasting ethical standards of suppliers in different areas of the world can lead to ...

Yonsei University researchers directly measure quantum metric tensor in real material

2025-08-06
Quantum distance refers to a measure of quantum mechanical similarity between two quantum states. A quantum distance of one means that the two quantum states are the same, whereas a quantum distance of zero implies that they are exactly the opposite. Physicists introduced this concept in the realm of theoretical science a long time ago, but its importance has been increasingly recognized in the field of physics only in recent times. In the last few years, many experimental physicists have tried to measure the quantum distance of electrons ...

TEER associated with improved survival in the elderly patients with AFMR

2025-08-06
The mitral valve controls blood flow between the heart’s left atrium and ventricle. When structural changes affect the atrium or valve, it may not close fully—causing blood to leak backwards, a condition known as atrial functional mitral regurgitation (AFMR). Common in frail or elderly patients, AFMR often leads to heart failure, and standard surgery is typically too risky due to other health issues.   Transcatheter edge-to-edge repair (TEER), a minimally invasive surgery, has emerged as a low-risk treatment option for various ...

Excessive screen time among youth may pose heart health risks

2025-08-06
Research Highlights: Increased time on electronic devices or watching TV among children and young adults was associated with higher cardiometabolic disease risk, such as high blood pressure, high cholesterol and insulin resistance, based on data from over 1,000 study participants in Denmark. The link between screen time and cardiometabolic risks was strongest among youth who slept fewer hours, suggesting that screen use may harm health by “stealing” time from sleep, researchers said. Researchers said the findings underscore the ...

Pure quantum state without the need for cooling

2025-08-06
  Three nano glass spheres cling to one another. They form a tower-like cluster, similar to when you pile three scoops of ice cream on top of one another – only much smaller. The diameter of the nano cluster is ten times smaller than that of a human hair. With the help of an optical device and laser beams, researchers at ETH Zurich have succeeded in keeping such objects almost completely motionless in levitation. This is significant when it comes to the future development of quantum sensors, which, together with quantum computers, constitute the most promising applications of quantum research. As part of their levitation ...

NHS Active 10 walking tracker users are more active after using the app

2025-08-06
Users of the NHS Active 10 app, designed to encourage people to become more active, immediately increased their amount of brisk and non-brisk walking upon using the app, according to researchers from the University of Cambridge. In a study published today in npj Digital Medicine, the researchers found that while activity levels then slowly declined over time, even after 30 months those users who were still using the app were more active than they had been beforehand. Lack of physical activity ...

Ultraviolet light reveals the aftermath of rare star collision

2025-08-06
University of Warwick astronomers have uncovered compelling evidence that a nearby white dwarf is in fact the remnant of two stars merging — a rare stellar discovery revealed through Hubble Space Telescope ultraviolet observations of carbon in the star’s hot atmosphere.  White dwarfs are the dense cores left behind when stars exhaust their fuel and collapse. They are Earth-sized stellar embers weighing typically half as much as the Sun, made up of carbon-oxygen cores with surface layers of helium and hydrogen. While white dwarfs ...

Growing shade trees can cut chocolate’s environmental impact

2025-08-06
University of Queensland research shows emissions from the global chocolate industry could be reduced by growing more shade trees over farms in the region that supplies 60 per cent of the world’s cocoa. Dr Wilma Blaser-Hart and Dr Simon Hart from UQ’s School of the Environment and Centre for Biodiversity and Conservation Science led a study which used satellite images and machine learning to examine farms in West Africa “Cocoa is naturally an understory tree in rainforests, but in monoculture farming systems it’s grown in the open,” ...

Seeing with fresh eyes: Snails as a system for studying sight restoration

2025-08-06
KANSAS CITY, MO—August 6, 2025—The eye of the apple snail is unusually similar to a human eye—but, unlike human eyes, it can regrow itself if injured or even amputated. New research from the Stowers Institute for Medical Research has established the apple snail as a novel research organism to study eye regeneration, with the potential to better understand and find treatments for eye conditions in humans like macular degeneration. The study, from the lab of Stowers President and Chief Scientific Officer Alejandro Sánchez Alvarado, Ph.D., published in Nature Communications on [date], describes a new system to study sensory organ ...

LAST 30 PRESS RELEASES:

Virtual reality shows promise in easing stress for cardiac patients, UCLA Health study finds

MBARI researchers deploy new imaging system to study the movement of deep-sea octopus

Scrambled RNA nudges millions of people towards type-2 diabetes

Big heart, acute senses key to explosive radiation of early fishes

Getting sticky: The highest-performing underwater adhesive hydrogel polymer

The health impact of wildfires in Los Angeles County and Maui

Replacing brain immune cells in mice slows neurodegeneration in Stanford Medicine study

Early heartbeats direct the heart’s own development and growth

Changes to El Niño occurrence causing widespread tropical insect and spider declines

Nanophotonics: An ultrafast light switch

Human instruction with artificial intelligence guidance provided best results in neurosurgical training

IHME’s 2025 Roux Prize awarded to rural health equity advocate – recognized for advancing access to oral health care in Nigeria

Archaeologists find oldest evidence of humans on ‘Hobbit’s’ island neighbor – who they were remains a mystery

Novel microwave dielectric ceramics enable high-performance 5G/6G communication devices

Revealed: New vaccine target to block malaria transmission

Air purifiers may reduce heart risks for people exposed to traffic pollution

Effective therapy for MDS is vastly underused, especially in female and non-white patients

Genetic rescue of endangered species may risk bad mutations slipping through

480 macrofungal species discovered in southeast Xizang, China: 8 new species, 115 edible, and 53 poisonous fungi

New study on hope among U.S. youth reveals key to safer schools this fall

AI chatbots can run with medical misinformation, study finds, highlighting the need for stronger safeguards

Nanoparticles for rheumatoid arthritis prevention, flare control

Small-world networks can mitigate ESG controversies for multinationals, suggests new study

Yonsei University researchers directly measure quantum metric tensor in real material

TEER associated with improved survival in the elderly patients with AFMR

Excessive screen time among youth may pose heart health risks

Pure quantum state without the need for cooling

NHS Active 10 walking tracker users are more active after using the app

Ultraviolet light reveals the aftermath of rare star collision

Growing shade trees can cut chocolate’s environmental impact

[Press-News.org] AI chatbots can run with medical misinformation, study finds, highlighting the need for stronger safeguards
Researchers suggest ways to reduce risk of chatbots repeating false medical details