“Our previous study showed that even the most advanced AI could produce the wrong codes, sometimes nonsensical ones, when left to guess,” says co-corresponding senior author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai. “This time, we gave the model a chance to reflect and review similar past cases. That small change made a big difference.”
Doctors in the United States spend hours every week assigning ICD codes—alphanumeric strings, used to describe everything from sprained ankles to heart attacks. But large language models, like ChatGPT, often struggle to assign these codes correctly. To address this, the researchers tried a “lookup-before-coding” method that first prompts the AI to describe a diagnosis in plain language and then choose the most fitting code from a list of real-world examples. The approach delivered greater accuracy, fewer mistakes, and performance on par with or better than humans.
The team utilized 500 Emergency Department patient visits at Mount Sinai Health System hospitals. For each case, they fed the physician’s note to nine different AI models, including small open-source systems. First, the models generated an initial ICD diagnostic description. Using a retrieval method, each description was matched to 10 similar ICD descriptions from a database of more than 1 million hospital records, along with how often those diagnoses occurred. In a second step, the model used this retrieved information to select the most accurate ICD description and code.
Emergency physicians and two independent AI systems evaluated the coding results independently, without information about whether the codes were generated by AI or clinicians.
Across the board, models that used the retrieval step outperformed those that didn’t, and even did better than physician-assigned codes in many cases. Surprisingly, even small open-source models performed well when allowed to “look up” examples.
“This is about smarter support, not automation for automation’s sake,” says co-corresponding senior author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and Chief AI Officer for the Mount Sinai Health System. “If we can cut the time our physicians spend on coding, reduce billing errors, and improve the quality of our data, all with an affordable and transparent system, that’s a big win for patients and providers alike.”
The authors emphasize that this retrieval-enhanced method is designed to support, not replace, human oversight. While it’s not yet approved for billing and was tested specifically on primary diagnosis codes from emergency visits discharged home, it shows encouraging potential for clinical use. The researchers see immediate uses, such as suggesting codes in electronic records or flagging errors before billing.
The investigators are now integrating the method into Mount Sinai’s electronic health records system for pilot testing. They hope to expand it to other clinical settings and to include secondary and procedural codes in future versions.
“The big picture here is AI’s potential to transform how we care for patients. When technology relieves the administrative burden of our physicians and other providers, they have more time for direct patient care. That’s good for clinicians, that's good for patients and it’s good for health systems of every size,” says David L. Reich MD, Chief Clinical Officer of the Mount Sinai Health System and President of The Mount Sinai Hospital. “Using AI in this way improves our ability to provide attentive and compassionate care by spending more time with patients. This strengthens the foundation of hospitals and health systems everywhere.”
The paper is titled “Assessing Retrieval-Augmented Large Language Models for Medical Coding.”
The study’s authors, as listed in the journal, are Eyal Klang, Idit Tessler, Donald U. Apakama, Ethan Abbott, Benjamin S Glicksberg, Arnold Monique, Akini Moses, Ankit Sakhuja, Ali Soroush, Alexander W. Charney, David L. Reich, Jolion McGreevy, Nicholas Gavin, Brendan Carr, Robert Freeman, and Girish N Nadkarni.
This work was supported by the Clinical and Translational Science Awards (CTSA) grant UL1TR004419 from the National Center for Advancing Translational Sciences. Research reported in this publication was also supported by the Office of Research Infrastructure of the National Institutes of Health under grant award numbers S10OD026880 and S10OD030463.
For more Mount Sinai artificial intelligence news, visit: https://icahn.mssm.edu/about/artificial-intelligence.
-####-
About Mount Sinai's Windreich Department of AI and Human Health
Led by Girish N. Nadkarni, MD, MPH—an international authority on the safe, effective, and ethical use of AI in health care—Mount Sinai’s Windreich Department of AI and Human Health is the first of its kind at a U.S. medical school, pioneering transformative advancements at the intersection of artificial intelligence and human health.
The Department is committed to leveraging AI in a responsible, effective, ethical, and safe manner to transform research, clinical care, education, and operations. By bringing together world-class AI expertise, cutting-edge infrastructure, and unparalleled computational power, the department is advancing breakthroughs in multi-scale, multimodal data integration while streamlining pathways for rapid testing and translation into practice.
The Department benefits from dynamic collaborations across Mount Sinai, including with the Hasso Plattner Institute for Digital Health at Mount Sinai—a partnership between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System—which complements its mission by advancing data-driven approaches to improve patient care and health outcomes.
At the heart of this innovation is the renowned Icahn School of Medicine at Mount Sinai, which serves as a central hub for learning and collaboration. This unique integration enables dynamic partnerships across institutes, academic departments, hospitals, and outpatient centers, driving progress in disease prevention, improving treatments for complex illnesses, and elevating quality of life on a global scale.
In 2024, the Department's innovative NutriScan AI application, developed by the Mount Sinai Health System Clinical Data Science team in partnership with Department faculty, earned Mount Sinai Health System the prestigious Hearst Health Prize. NutriScan is designed to facilitate faster identification and treatment of malnutrition in hospitalized patients. This machine learning tool improves malnutrition diagnosis rates and resource utilization, demonstrating the impactful application of AI in health care.
For more information on Mount Sinai's Windreich Department of AI and Human Health, visit: ai.mssm.edu
About the Hasso Plattner Institute at Mount Sinai
At the Hasso Plattner Institute for Digital Health at Mount Sinai, the tools of data science, biomedical and digital engineering, and medical expertise are used to improve and extend lives. The Institute represents a collaboration between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System.
Under the leadership of Girish Nadkarni, MD, MPH, who directs the Institute, and Professor Lothar Wieler, a globally recognized expert in public health and digital transformation, they jointly oversee the partnership, driving innovations that positively impact patient lives while transforming how people think about personal health and health systems.
The Hasso Plattner Institute for Digital Health at Mount Sinai receives generous support from the Hasso Plattner Foundation. Current research programs and machine learning efforts focus on improving the ability to diagnose and treat patients.
About the Icahn School of Medicine at Mount Sinai
The Icahn School of Medicine at Mount Sinai is internationally renowned for its outstanding research, educational, and clinical care programs. It is the sole academic partner for the seven member hospitals* of the Mount Sinai Health System, one of the largest academic health systems in the United States, providing care to New York City’s large and diverse patient population.
The Icahn School of Medicine at Mount Sinai offers highly competitive MD, PhD, MD-PhD, and master’s degree programs, with enrollment of more than 1,200 students. It has the largest graduate medical education program in the country, with more than 2,600 clinical residents and fellows training throughout the Health System. Its Graduate School of Biomedical Sciences offers 13 degree-granting programs, conducts innovative basic and translational research, and trains more than 560 postdoctoral research fellows.
Ranked 11th nationwide in National Institutes of Health (NIH) funding, the Icahn School of Medicine at Mount Sinai is among the 99th percentile in research dollars per investigator according to the Association of American Medical Colleges. More than 4,500 scientists, educators, and clinicians work within and across dozens of academic departments and multidisciplinary institutes with an emphasis on translational research and therapeutics. Through Mount Sinai Innovation Partners (MSIP), the Health System facilitates the real-world application and commercialization of medical breakthroughs made at Mount Sinai.
END