But that wasn’t the only impact, according to a comprehensive new study by Penn researchers that digs deeper into the complicated issue of race and ethnicity in health care algorithms. Removing race from the kidney function algorithm also appeared to reduce chemotherapy access, reduce eligibility for Black patients in clinical trials and affect medication dosing.
The new paper, published this month in the Annals of Internal Medicine, paints a nuanced picture of algorithms in health care—a ubiquitous, but often unseen, force in clinical decision making—and how their use can impact racial and ethnic disparities. The research team, led by Shazia Mehmood Siddique, MD, an assistant professor of Gastroenterology in Penn’s Perelman School of Medicine, found that algorithms can mitigate, perpetuate, and exacerbate racial and ethnic disparities, regardless of whether they explicitly use race or ethnicity as an input.
“Intentionality matters,” said Siddique, who also serves as director for research for Penn Medicine’s Center for Evidence-Based Practice (CEP) and also the Penn Center for Healthcare Improvement and Patient Safety (CHIPS). “Racial and ethnic disparities cannot be an afterthought.”
The researchers defined algorithms as mathematical equations that combined various data points and inputs, such as sex and age, to inform clinical care. Algorithms are embedded throughout health care, Siddique said, to help providers make complex clinical decisions, such as whether a patient should be diagnosed with a disease or is eligible for a particular treatment, and to help health systems determine how to allocate resources, such as care management and critical care services. Importantly, algorithms are often embedded in electronic health records and patients and clinicians themselves are often not always aware they are being utilized.
To patients, it might seem as if medical criteria such as risk scores or treatment thresholds are based entirely on objective factors, said study co-author Brian Leas, a senior research analyst at CEP. But algorithms introduce a social component to clinical decision making. “How algorithms are constructed is a choice made by the developers,” he said. “It’s a decision to put certain factors together into a formula, and those decisions can be made differently.”
Traditionally, health care algorithms were developed by researchers in academic settings. More recently, however, health systems, insurance companies, and electronic health record companies have begun to develop their own algorithms, while artificial intelligence tools have rapidly assumed a major role in fueling new algorithms —a shift that researchers said heightens the need for scrutiny.
In 2020, health care algorithms caught the attention of four U.S. senators, including New Jersey’s Cory Booker. Citing the detrimental kidney disease algorithm, as well as another that unjustly lowered settlement benefits for concussion-related injuries for Black players in the National Football League, the lawmakers called on the Agency for Healthcare Research and Quality in the U.S. Department of Health and Human Services to conduct a review of race-based clinical algorithms in medical practice. The agency subsequently commissioned the Penn team’s study, which was conducted in collaboration with ECRI, a nonprofit healthcare research organization focused on patient safety and reducing preventable harm.
In a systematic review of 63 studies, the researchers found that there’s no silver bullet fix for issues associated with algorithms. Instead, they identified several strategies to mitigate disparities in health care algorithms, including adding a non-race variable, using data that reflects diverse racial and ethnic groups when developing algorithms, and swapping race with another more precise variable, such as genetic data or social factors that may impact care.
Health care algorithms are most successful at reducing disparities when they intentionally reduce documented inequalities, Siddique said. In some cases, this meant including race as an algorithm component. For instance, a prostate cancer screening algorithm was found to overtest Black men, which led to unnecessary biopsies and complications. Adding Black race as an input in the algorithm mitigated the disparity.
But sometimes, as with the kidney disease algorithm, reducing disparities means compromising other outcomes. When race was removed as a variable from an algorithm for lung cancer screening eligibility, disparities in eligibility for Hispanic and Asian Americans improved, while disparities deepened for Black patients. “A value judgment is needed when we are making a decision about these tradeoffs in outcomes,” Siddique said. “Is it worth this downside that we’re going to see for this potential benefit? How can we further refine the algorithm to minimize disparities across groups?” Researchers found that instead of removing race from the lung cancer eligibility algorithm, refining the model by allowing race and ethnicity to remain if a patient’s only reason for ineligibility was low life expectancy due to their race or ethnicity ultimately improved lung cancer eligibility disparities across the board.
Race is often used in algorithms as a proxy for another variable, such as ancestry, a specific gene, social determinants of health or even the effects of systemic racism, Siddique said. The problem, she added, is that it is often unclear why race is being used in an algorithm. “We need algorithm developers to be clear about what race is being used as a proxy for, because clinicians may have no idea,” Siddique said. “If there is no transparency about it, then it can perpetuate the false assumption that race is biologic.”
A better option: replace race with a more precise variable. Siddique is now studying, for instance, whether replacing race with country of origin in a liver cancer screening guideline would reduce disparities. (While similar to algorithms, guidelines are non-mathematical, evidence-based recommendations typically developed by medical associations to help guide best clinical practices.)
The conversation about algorithms in healthcare—and their impact on racial and ethnic disparities—is just beginning. In response to the Penn-ECRI research, a diverse panel of health care experts convened late last year to offer guidance on mitigating algorithm bias. The group recommended guiding principles to support health care equity in the algorithm development and review process.
In a commentary in Health Affairs published in October 2023, Siddique and others promoted a “race-aware” -- rather than race-based -- approach to algorithms, calling for increased diversity in clinical trials, a focus on precision medicine and improved education on the factors that shape health outcomes.
The study was funded by the Agency for Healthcare Quality and Research.
###
Penn Medicine is one of the world’s leading academic medical centers, dedicated to the related missions of medical education, biomedical research, excellence in patient care, and community service. The organization consists of the University of Pennsylvania Health System and Penn’s Raymond and Ruth Perelman School of Medicine, founded in 1765 as the nation’s first medical school.
The Perelman School of Medicine is consistently among the nation's top recipients of funding from the National Institutes of Health, with $550 million awarded in the 2022 fiscal year. Home to a proud history of “firsts” in medicine, Penn Medicine teams have pioneered discoveries and innovations that have shaped modern medicine, including recent breakthroughs such as CAR T cell therapy for cancer and the mRNA technology used in COVID-19 vaccines.
The University of Pennsylvania Health System’s patient care facilities stretch from the Susquehanna River in Pennsylvania to the New Jersey shore. These include the Hospital of the University of Pennsylvania, Penn Presbyterian Medical Center, Chester County Hospital, Lancaster General Health, Penn Medicine Princeton Health, and Pennsylvania Hospital—the nation’s first hospital, founded in 1751. Additional facilities and enterprises include Good Shepherd Penn Partners, Penn Medicine at Home, Lancaster Behavioral Health Hospital, and Princeton House Behavioral Health, among others.
Penn Medicine is an $11.1 billion enterprise powered by more than 49,000 talented faculty and staff.
END
