In the retrospective study, the researchers analyzed records from seven Mount Sinai Health System hospitals, using both structured data, such as vital signs, and unstructured data, such as nurse triage notes, from more than 864,000 emergency room visits while excluding identifiable patient data. Of these visits, 159,857 (18.5 percent) led to the patient being admitted to the hospital.
The researchers compared GPT-4 against traditional machine-learning models such as Bio-Clinical-BERT for text and XGBoost for structured data in various scenarios, assessing its performance to predict hospital admissions independently and in combination with the traditional methods.
“We were motivated by the need to test whether generative AI, specifically large language models (LLMs) like GPT-4, could improve our ability to predict admissions in high-volume settings such as the Emergency Department,” says co-senior author Eyal Klang, MD, Director of the Generative AI Research Program in the Division of Data-Driven and Digital Medicine (D3M) at Icahn Mount Sinai. “Our goal is to enhance clinical decision-making through this technology. We were surprised by how well GPT-4 adapted to the ER setting and provided reasoning for its decisions. This capability of explaining its rationale sets it apart from traditional models and opens up new avenues for AI in medical decision-making.”
While traditional machine-learning models use millions of records for training, LLMs can effectively learn from just a few examples. Moreover, according to the researchers, LLMs can incorporate traditional machine-learning predictions, improving performance
“Our research suggests that AI could soon support doctors in emergency rooms by making quick, informed decisions about patient admissions. This work opens the door for further innovation in health care AI, encouraging the development of models that can reason and learn from limited data, like human experts do,” says co-senior author Girish N. Nadkarni, MD, MPH, Irene and Dr. Arthur M. Fishberg Professor of Medicine at Icahn Mount Sinai, Director of The Charles Bronfman Institute of Personalized Medicine, and System Chief of D3M. “However, while the results are encouraging, the technology is still in a supportive role, enhancing the decision-making process by providing additional insights, not taking over the human component of health care, which remains critical.”
The research team is investigating how to apply large language models to health care systems, with the goal of harmoniously integrating them with traditional machine-learning methods to address complex challenges and decision-making in real-time clinical settings.
“Our study informs how LLMs can be integrated into health care operations. The ability to rapidly train LLMs highlights their potential to provide valuable insights even in complex environments like health care,” says Brendan Carr, MD, MA, MS, a study co-author and emergency room physician who is Chief Executive Officer of Mount Sinai Health System. “Our study sets the stage for further research on AI integration in health care across the many domains of diagnostic, treatment, operational, and administrative tasks that require continuous optimization.”
The paper is titled “Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room.”
The remaining authors of the paper, all with Icahn Mount Sinai, are Benjamin S. Glicksberg, PhD; Dhaval Patel, BS; Ashwin Sawant, MD; Akhil Vaid, MD; Ganesh Raut, BS; Alexander W. Charney, MD, PhD; Donald Apakama, MD; and Robert Freeman, RN.
The work was supported by the National Heart Lung and Blood Institute NIH grant 5R01HL141841-05.
-####-
About the Icahn School of Medicine at Mount Sinai
The Icahn School of Medicine at Mount Sinai is internationally renowned for its outstanding research, educational, and clinical care programs. It is the sole academic partner for the eight- member hospitals* of the Mount Sinai Health System, one of the largest academic health systems in the United States, providing care to a large and diverse patient population.
Ranked 13th nationwide in National Institutes of Health (NIH) funding and among the 99th percentile in research dollars per investigator according to the Association of American Medical Colleges, Icahn Mount Sinai has a talented, productive, and successful faculty. More than 3,000 full-time scientists, educators, and clinicians work within and across 44 academic departments and 36 multidisciplinary institutes, a structure that facilitates tremendous collaboration and synergy. Our emphasis on translational research and therapeutics is evident in such diverse areas as genomics/big data, virology, neuroscience, cardiology, geriatrics, as well as gastrointestinal and liver diseases.
Icahn Mount Sinai offers highly competitive MD, PhD, and Master’s degree programs, with current enrollment of approximately 1,300 students. It has the largest graduate medical education program in the country, with more than 2,000 clinical residents and fellows training throughout the Health System. In addition, more than 550 postdoctoral research fellows are in training within the Health System.
A culture of innovation and discovery permeates every Icahn Mount Sinai program. Mount Sinai’s technology transfer office, one of the largest in the country, partners with faculty and trainees to pursue optimal commercialization of intellectual property to ensure that Mount Sinai discoveries and innovations translate into healthcare products and services that benefit the public.
Icahn Mount Sinai’s commitment to breakthrough science and clinical care is enhanced by academic affiliations that supplement and complement the School’s programs.
Through the Mount Sinai Innovation Partners (MSIP), the Health System facilitates the real-world application and commercialization of medical breakthroughs made at Mount Sinai. Additionally, MSIP develops research partnerships with industry leaders such as Merck & Co., AstraZeneca, Novo Nordisk, and others.
The Icahn School of Medicine at Mount Sinai is located in New York City on the border between the Upper East Side and East Harlem, and classroom teaching takes place on a campus facing Central Park. Icahn Mount Sinai’s location offers many opportunities to interact with and care for diverse communities. Learning extends well beyond the borders of our physical campus, to the eight hospitals of the Mount Sinai Health System, our academic affiliates, and globally.
-------------------------------------------------------
* Mount Sinai Health System member hospitals: The Mount Sinai Hospital; Mount Sinai Beth Israel; Mount Sinai Brooklyn; Mount Sinai Morningside; Mount Sinai Queens; Mount Sinai South Nassau; Mount Sinai West; and New York Eye and Ear Infirmary of Mount Sinai.
END
