PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

ChatGPT is biased against resumes with credentials that imply a disability — but it can improve

2024-06-21
(Press-News.org) While seeking research internships last year, University of Washington graduate student Kate Glazko noticed recruiters posting online that they’d used OpenAI’s ChatGPT and other artificial intelligence tools to summarize resumes and rank candidates. Automated screening has been commonplace in hiring for decades. Yet Glazko, a doctoral student in the UW’s Paul G. Allen School of Computer Science & Engineering, studies how generative AI can replicate and amplify real-world biases — such as those against disabled people. How might such a system, she wondered, rank resumes that implied someone had a disability?

In a new study, UW researchers found that ChatGPT consistently ranked resumes with disability-related honors and credentials — such as the “Tom Wilson Disability Leadership Award” — lower than the same resumes without those honors and credentials. When asked to explain the rankings, the system spat out biased perceptions of disabled people. For instance, it claimed a resume with an autism leadership award had “less emphasis on leadership roles” — implying the stereotype that autistic people aren’t good leaders.

But when researchers customized the tool with written instructions directing it not to be ableist, the tool reduced this bias for all but one of the disabilities tested. Five of the six implied disabilities — deafness, blindness, cerebral palsy, autism and the general term “disability” — improved, but only three ranked higher than resumes that didn't mention disability.

The team presented its findings June 5 at the 2024 ACM Conference on Fairness, Accountability, and Transparency in Rio de Janeiro.

“Ranking resumes with AI is starting to proliferate, yet there’s not much research behind whether it’s safe and effective,” said Glazko, the study’s lead author. “For a disabled job seeker, there’s always this question when you submit a resume of whether you should include disability credentials. I think disabled people consider that even when humans are the reviewers.”

Researchers used one of the study’s authors’ publicly available curriculum vitae (CV), which ran about 10 pages. The team then created six enhanced CVs, each implying a different disability by including four disability-related credentials: a scholarship; an award; a diversity, equity and inclusion (DEI) panel seat; and membership in a student organization.

Researchers then used ChatGPT’s GPT-4 model to rank these enhanced CVs against the original version for a real “student researcher” job listing at a large, U.S.-based software company. They ran each comparison 10 times; in 60 trials, the system ranked the enhanced CVs, which were identical except for the implied disability, first only one quarter of the time.

“In a fair world, the enhanced resume should be ranked first every time,” said senior author Jennifer Mankoff, a UW professor in the Allen School. “I can’t think of a job where somebody who’s been recognized for their leadership skills, for example, shouldn’t be ranked ahead of someone with the same background who hasn’t.”

When researchers asked GPT-4 to explain the rankings, its responses exhibited explicit and implicit ableism. For instance, it noted that a candidate with depression had “additional focus on DEI and personal challenges,” which “detract from the core technical and research-oriented aspects of the role.”

“Some of GPT’s descriptions would color a person’s entire resume based on their disability and claimed that involvement with DEI or disability is potentially taking away from other parts of the resume,” Glazko said. “For instance, it hallucinated the concept of ‘challenges’ into the depression resume comparison, even though ‘challenges’ weren’t mentioned at all. So you could see some stereotypes emerge.”

Given this, researchers were interested in whether the system could be trained to be less biased. They turned to the GPTs Editor tool, which allowed them to customize GPT-4 with written instructions (no code required). They instructed this chatbot to not exhibit ableist biases and instead work with disability justice and DEI principles.

They ran the experiment again, this time using the newly trained chatbot. Overall, this system ranked the enhanced CVs higher than the control CV 37 times out of 60. However, for some disabilities, the improvements were minimal or absent: The autism CV ranked first only three out of 10 times, and the depression CV only twice (unchanged from the original GPT-4 results).

“People need to be aware of the system’s biases when using AI for these real-world tasks,” Glazko said. “Otherwise, a recruiter using ChatGPT can’t make these corrections, or be aware that, even with instructions, bias can persist.”

Researchers note that some organizations, such as ourability.com and inclusively.com, are working to improve outcomes for disabled job seekers, who face biases whether or not AI is used for hiring. They also emphasize that more research is needed to document and remedy AI biases. Those include testing other systems, such as Google’s Gemini and Meta’s Llama; including other disabilities; studying the intersections of the system’s bias against disabilities with other attributes such as gender and race; exploring whether further customization could reduce biases more consistently across disabilities; and seeing whether the base version of GPT-4 can be made less biased.

“It is so important that we study and document these biases,” Mankoff said. “We’ve learned a lot from and will hopefully contribute back to a larger conversation — not only regarding disability, but also other minoritized identities — around making sure technology is implemented and deployed in ways that are equitable and fair.”

Additional co-authors were Yusuf Mohammed, a UW undergraduate in the Allen School; Venkatesh Potluri, a UW doctoral student in the Allen School; and Ben Kosa, who completed this research as a UW undergraduate in the Allen School and is an incoming doctoral student at University of Wisconsin–Madison. This research was funded by the National Science Foundation; by donors to the UW’s Center for Research and Education on Accessible Technology and Experiences (CREATE); and by Microsoft.

For more information, contact Glazko at glazko@cs.washington.edu and Mankoff at jmankoff@cs.washington.edu.

END


ELSE PRESS RELEASES FROM THIS DATE:

Simple test for flu could improve diagnosis and surveillance

2024-06-21
Fewer than one percent of people who get the flu every year get tested, in part because most tests require trained personnel and expensive equipment. Now researchers have developed a low-cost paper strip test that could allow more patients to find out which type of flu they have and get the right treatment.  The test, developed by a team from the Broad Institute of MIT and Harvard and Princeton University, and supported by the US Centers for Disease Control and Prevention, uses CRISPR to distinguish between the two main types of seasonal flu, influenza A and B, as well as seasonal ...

UT Health San Antonio researcher awarded five-year, $2.53 million NIH grant to study alcohol-assisted liver disease

2024-06-21
SAN ANTONIO, June 21, 2024 – Liver transplants associated with alcohol-related disease are growing at a rapid pace, shifting research to address pathologies behind the ailments in light of a limited supply of organ donors. At the forefront is Mengwei Zang, MD, PhD, an internationally recognized leader in chronic liver disease research at The University of Texas Health Science Center at San Antonio (UT Health San Antonio) who was just awarded a groundbreaking five-year, $2.53 million grant from the National ...

Giving pre-med students hands-on clinical training

Giving pre-med students hands-on clinical training
2024-06-21
A group of pre-medical students received valuable hands-on clinical training during a workshop in the new Smart Hospital at The University of Texas at Arlington. The Clinical Experience Workshop allowed 10 pre-med students to participate in experiential activities and to interact one-on-one with “patients” portrayed by students from the UTA Department of Theatre Arts. “This was a clinical opportunity for pre-med students with no clinical background to be immersed in clinical medicine, learn basic skills, and experience actual patient encounters with simulated patients ...

CAMH research suggests potential targets for prevention and early identification of psychotic disorders

2024-06-21
A new study by the Centre for Addiction and Mental Health (CAMH), entitled Mental Health Service Use Before First Diagnosis of a Psychotic Disorder and published in JAMA Psychiatry, found that nearly 75 per cent of young Ontarians with a psychotic disorder had at least one mental health service visit within the three years prior to their first diagnosis of the disorder. The retrospective cohort study—one of the largest of its kind—suggests that youth with a psychotic disorder are nearly four times as likely to have a previous mental health-related hospital ...

Mapping the heart to prevent damage caused by a heart attack

Mapping the heart to prevent damage caused by a heart attack
2024-06-21
Scientists at the Victor Chang Cardiac Research Institute in Australia have produced a first of its kind integrated map of heart cells which unlocks the process of cardiac fibrosis – a major cause of heart failure.  The discovery opens new avenues to develop targeted drugs to prevent scarring damage caused after a heart attack.  During and after a heart attack, the heart’s muscles are damaged leading to the formation of scar tissue which lacks the elasticity and contractility of healthy heart muscle. This damage is permanent and can affect ...

Study challenges popular idea that Easter islanders committed ‘ecocide’

Study challenges popular idea that Easter islanders committed ‘ecocide’
2024-06-21
Some 1,000 years ago, a small band of Polynesians sailed thousands of miles across the Pacific to settle one of the world’s most isolated places—a small, previously uninhabited island they named Rapa Nui. There, they erected hundreds of “moai,” or gigantic stone statues that now famously stand as emblems of a vanished civilization. Eventually, their numbers ballooned to unsustainable levels; they chopped down all the trees, killed off the seabirds, exhausted the soils and in the end, ruined their environment. Their population and civilization collapsed, with just a few thousand people remaining when ...

Chilling discovery: Study reveals evolution of human cold and menthol sensing protein, offering hope for future non-addictive pain therapies.

Chilling discovery: Study reveals evolution of human cold and menthol sensing protein, offering hope for future non-addictive pain therapies.
2024-06-21
Chronic pain affects millions worldwide, and current treatments often rely on opioids, which carry risks of addiction and overdose.  Non-addictive alternatives could revolutionize pain management, and new research targeting the human protein which regulates cold sensations, brings scientists closer to developing pain medications that don't affect body temperature and don't carry the risks of addiction.  Research published in Science Advances on June 21, led by Wade Van Horn, professor in Arizona State University’s School of Molecular Sciences and Biodesign ...

Elena Beccalli, new rector of Università Cattolica del Sacro Cuore, takes office on 1st July

2024-06-21
Elena Beccalli will be rector of the Università Cattolica del Sacro Cuore on 1st July for the four-year term 2024–2028. After being appointed by the University's Board of Directors, which convened today, Thursday 20 June 2024, Professor Beccalli succeeds Professor Franco Anelli. She is the first woman appointed to this role in the history of our university.   The decision of the Board of Directors follows the appointment of Professor Elena Beccalli, Dean of the School of Banking, Finance, and Insurance ...

Pacific Northwest Research Institute uncovers hidden DNA mechanisms of rare genetic diseases

Pacific Northwest Research Institute uncovers hidden DNA mechanisms of rare genetic diseases
2024-06-21
Seattle, WA — June 21, 2024 — Researchers at the Pacific Northwest Research Institute (PNRI) and collaborating institutions have made a groundbreaking discovery that could significantly advance our understanding of genomic disorders. Their latest study, funded by the National Institutes of Health[1] and published in the journal Cell Genomics, reveals how specific DNA rearrangements called inverted triplications contribute to the development of various genetic diseases. Understanding the Study Genomic disorders occur when there are changes or mutations in DNA that disrupt normal biological functions. These can lead ...

Empowering older adults: Wearable tech made easier with personalized support

2024-06-21
(Toronto, June 20, 2024) A new review in the Journal of Medical Internet Research, published by JMIR Publications, found that community-dwelling older adults are more likely to continue using wearable monitoring devices (WMDs), like trackers, pedometers, and smartwatches, if they receive support from health care professionals or peers. The research team from The Hong Kong Polytechnic University, led by Dr. Arkers Kwan Ching Wong, reviewed data from 3 randomized controlled trials involving over 150 older adults. The evaluation showed that the interventions that focused on increasing awareness of being monitored and used collaborative goal-setting and feedback tools, such as the SystemCHANGE ...

LAST 30 PRESS RELEASES:

USC Stem Cell mouse study sheds light on the secret to maintaining a youthful immune system

Suicide risk highest on Mondays and New Year’s Day

Gene signature shows promise to improve survival for breast cancer patients

Investigation finds “unexplained” millions in drug industry payments to the NHS

Maternal antibodies interfere with malaria vaccine responses

Teaching must be made more attractive as a profession to tackle shortages

Airbnb rentals linked to increased crime rates in London neighborhoods – study

UK budget 'blindness' risks handing green economy future to China, report argues

Marri trees a lifeline for many native bee species in biodiversity hotspot

Treatments used for HER2-positive breast cancers could help patients with rare gastrointestinal cancer

Little-studied RNA might be key to regulating genetic disorders like epilepsy, autism

UB researchers show why cannabis policies should shift to a harm reduction, health promotion approach to safeguard public health

Live well, think well: Research shows healthy habits tied to brain health

Could poor sleep in middle age speed up brain aging?

Fossils unveil how southern Europe’s ecosystem changed through Glacial-Interglacial Stages

Your ability to balance on one leg may be a reliable indicator of neuromuscular aging, with men and women showing significant declines over the decades

Most young adults in the UK consider non-consensual condom removal during sex to be wrong and a violation of consent, with almost 9 in 10 seeing it as a form of sexual assault, per survey of 18-25-yea

Under climate change scenarios, 30-44% more land in Ethiopia might become suitable for growing arabica coffee by 2080, although some cultivated areas might also become unsuitable, per modelling study

Cockroaches and maggots might be able to turn an invasive seaweed into a high quality compost, finds a new experimental study which provides hope for the environment and the circular economy

Implantable device may prevent death from opioid overdose

Half of young adults support prison time for non-consensual condom removal

‘Paleo-robots’ to help scientists understand how fish started to walk on land

Study: Robotic automation, AI will speed up scientific progress in science laboratories

Paleontologists discover Colorado ‘swamp dweller’ that lived alongside dinosaurs

Repeated COVID vaccines enhance mucosal immunity against the virus

MD Anderson expands arts experience program to enhance healing and well-being for patients

Students at Historically Black Colleges and Universities (HBCUs) face barriers to medical school admission, study finds

Symbiosis in ancient Corals

Researchers receive grant to study invasive autumn olive

New research shows urine tests may detect early diseases

[Press-News.org] ChatGPT is biased against resumes with credentials that imply a disability — but it can improve