PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

Women may pay a "MOM PENALTY" when AI is used in hiring, new research from NYU Tandon School of Engineering suggests

Researchers uncover bias in popular Large Language Models when used to evaluate potential job candidates

2023-12-13
(Press-News.org) Maternity-related employment gaps may cause job candidates to be unfairly screened out of positions for which they are otherwise qualified, according to new research from NYU Tandon School of Engineering.  

A research team led by Siddharth Garg, Institute Associate Professor of Electrical and Computer Engineering, examined bias in Large Language Models (LLMs) – advanced AI systems trained to understand and generate human language –  when used in hiring processes. 

The team will present its findings in a paper presented at NeurIPS 2023 R0-FoMo Workshop on December 15. Akshaj Kumar Veldanda, PhD candidate in Department of Electrical and Computer Engineering, is the paper's lead researcher. 

AI algorithms have come under scrutiny recently when used in employment.  President Biden's October 2023 AI executive order underscored the pressing need to address potential bias when employers rely on AI to help with hiring. New York City has enacted a first-of-its-kind law requiring regular audits to assess the transparency and fairness of algorithmic hiring decisions.

“Our research is helping develop a robust auditing methodology that can uncover hiring biases in LLMs, aiding researchers and practitioners in efforts to intervene before discrimination occurs,” said Garg. “Our study unearths some of the very biases that the New York City law intends to prohibit.”

In the study, researchers assessed the ability of three popular LLMs, namely ChatGPT (GPT-3.5), Bard, and Claude, to disregard irrelevant personal attributes such as race or political affiliations — factors that are both legally and ethically inappropriate to consider — while evaluating job candidates' resumes.

To do this, researchers added “sensitive attributes” to experimental resumes, including race and gender signaled through first and last names associated with either Black or white men or women; language indicating periods of absence from employment for parental duties, affiliation with either the Democratic or Republican party, and disclosure of pregnancy status.

After being fed the resumes, the LLMs were presented with two queries that human resource professionals could reasonably use in hiring: identifying whether the information presented on a resume aligns it with a specific job category – such as “teaching” or “construction” – and summarizing resumes to include only information relevant for employment.

While race and gender did not trigger biased results in the resume-matching experiment, the other sensitive attributes did, meaning at least one of the LLMs erroneously factored them into whether it included or excluded a resume from a job category.

Maternity- and paternity employment gaps triggered pronounced biased results. Claude performed the worst on that attribute, most frequently using it to wrongly assign a resume either inside or outside its correct job category. ChatGPT also showed consistently biased results on that attribute, although less frequently than Claude. 

“Employment gaps for parental responsibility, frequently exercised by mothers of young children, are an understudied area of potential hiring bias,” said Garg. “This research suggests those gaps can wrongly weed out otherwise qualified candidates when employers rely on LLMs to filter applicants.”

Both political affiliation and pregnancy triggered incorrect resume classification as well, with Claude once again performing the worst and ChatGPT coming in behind it.

Bard performed strongest across the board, exhibiting a remarkably consistent lack of bias across all sensitive attributes. 

“Claude is the most prone to bias in our study, followed by GPT-3.5. But Bard’s performance shows that bias is not a fait accompli,” said Garg “LLMs can be trained to withstand bias on attributes that are infrequently tested against, although in the case of Bard it could be biased along sensitive attributes that were not in this study.”

When it comes to producing resume summaries, researchers found stark differences between models. GPT-3.5 largely excludes political affiliation and pregnancy status from the generated summaries, whereas Claude is more likely to include all sensitive attributes. Bard frequently refuses to summarize, but is more likely to include sensitive information in cases where it generates the summaries. In general, job category classification on summaries – rather than full resumes – improves fairness of all LLMs including Claude, potentially because summaries make it easier for a model to attend to relevant information.

“The summary experiment also points to the relative weakness of Claude compared to the other LLMs tested,” said Garg. “This study overall tells us that we must continue to interrogate the soundness of using LLMs in employment, ensuring we ask LLMs to prove to us they are unbiased, not the other way around. But we also must embrace the possibility that LLMs can, in fact, play a useful and fair role in hiring.”

Methodology and notes

The study began by using a publicly released dataset of 2484 resumes from livecareer.com, available via Kaggle, spanning 24 job categories, which were anonymized to remove personal information. Due to limitations with state-of-the-art language model APIs, the evaluation initially focused on a subset of three job categories: Information Technology (IT), Teacher, and Construction. This yielded a “raw" resume corpus containing 334 resumes. Researchers subsequently evaluated across all 24 job categories for both Bard and Claude. The researchers manually inspected a sample of the resumes to ensure they matched their ground-truth job categories and had relevant information, such as experience and educational qualifications.

Sensitive attributes like race, gender, maternity/paternity-based employment gaps, pregnancy status, and political affiliation were introduced to the resumes using a specific approach, including that of Sendhil Mullainathan, a behavioral economist and professor at Harvard University who produced seminal research on hiring bias using racially stereotypical names of job candidates. Language added for other sensitive attributes aligned with standard recommendations related to resume creation. 

For job category classifications, researchers pose a binary classification problem to the LLM to identify whether a resume belongs to that job category or not. Researchers then evaluated the accuracy, true positive and true negative rates using ground-truth labels from its dataset. 

For the summary task, the LLM was asked to briefly summarize a specific resume and keep the most important information for employment.  Researchers evaluated bias by identifying whether sensitive attributes were retained and by using summaries for the classification task, mimicking a scenario in which the resume itself is too long for classification analysis. Classification on summaries improves fairness of LLMs, including Claude. 

Because ChatGPT, Bard, and Anthropic (Claude) are black box models – meaning they arrive at conclusions or decisions without providing any explanations as to how they were reached –  in-depth examination of biases are hindered. To gain a deeper understanding, the researchers conducted an evaluation on Alpaca, a white-box model that provides such explanations. The team observed that Alpaca exhibits biases in classification tasks as well. The team employed an existing method called Contrastive Input Decoding (CID) to explain the biases within the Alpaca model. Through this approach, researchers observed that:

For maternity leave, some responses offered the following reason for rejection: "Including personal information about maternity leave is not relevant to the job and could be seen as a liability." For pregnancy status, CID rejected candidates because "She is pregnant" or "Because of her pregnancy."  For political affiliation, CID analysis indicated that certain candidates were unsuitable because, "The candidate is a member of the Republican party, which may be a conflict of interest for some employers." It is important to note that CID does only sometimes offer these reasons, potentially because CID picks one of the potentially many reasons for rejection. Nonetheless, these results suggest that CID could be an effective tool to analyze bias even on larger models, given white-box access.

 

About the New York University Tandon School of Engineering

The NYU Tandon School of Engineering dates to 1854, the founding date for both the New York University School of Civil Engineering and Architecture and the Brooklyn Collegiate and Polytechnic Institute. A January 2014 merger created a comprehensive school of education and research in engineering and applied sciences as part of a global university, with close connections to engineering programs at NYU Abu Dhabi and NYU Shanghai. NYU Tandon is rooted in a vibrant tradition of entrepreneurship, intellectual curiosity, and innovative solutions to humanity’s most pressing global challenges. Research at Tandon focuses on vital intersections between communications/IT, cybersecurity, and data science/AI/robotics systems and tools and critical areas of society that they influence, including emerging media, health, sustainability, and urban living. We believe diversity is integral to excellence, and are creating a vibrant, inclusive, and equitable environment for all of our students, faculty and staff. For more information, visit engineering.nyu.edu.

END


ELSE PRESS RELEASES FROM THIS DATE:

Study presents new pathway for electrochemically controlling ion selectivity

2023-12-13
A new study by researchers at the University of Illinois Urbana-Champaign advances fundamental knowledge about the role of solvation in ion binding and presents a new pathway for electrochemically controlling ion selectivity. The study was published in JACS Au. The team, led by Chemical & Biomolecular Engineering professor Xiao Su and recently graduated Ph.D. student Raylin Chen, is building on their prior work exploring electrochemical separations of ions, which has revealed that a critical mechanism for binding ions is solvation. Here, the researchers set out to control solvation of a polymer and use that to bind different ...

Poor diet quality during adolescence is linked to serious health risks

2023-12-13
Philadelphia, December 13, 2023 – Diet quality among adolescents in the United States is among the worst across all age groups, putting young people at risk for heart attack, stroke, and diabetes, among other cardiometabolic diseases later in life. The research brief shared in the Journal of Nutrition Education and Behavior, published by Elsevier, used the Healthy Eating Index-2015 and medical testing to assess a group of youth aged 10-16 years. This study examined data from the Translational Investigation of Growth and Everyday Routine in Kids cohort. This study measured physical activity, sleep, and overall dietary guidelines for youth living in metropolitan areas ...

Stuart Parkin to receive American Physical Society’s highest award for contributions to spintronics and data storage

2023-12-13
The American Physical Society (APS) has awarded Stuart Parkin of the Max Planck Institute of Microstructure Physics the 2024 APS Medal for Exceptional Achievement in Research. Parkin will be recognized “for major discoveries in spintronics leading to a revolution in data storage and memory” at a ceremony during the APS Annual Leadership Meeting in January 2024. “Stuart Parkin is a luminary whose incisive experiments and major discoveries in spintronics led to a revolution in data storage and memory,” said APS President-Elect Young-Kee Kim, who chaired the medal’s selection committee. “His indomitable ...

New research shows that US renters are hit the hardest when a hurricane strikes

2023-12-13
WASHINGTON, DC, December 13, 2023 –With a severe shortage of affordable housing in the United States, renters living along the East and Gulf coasts are uniquely vulnerable to hurricane disasters. Two new studies based on data from 2009 to 2018 show that renters living along the East and Gulf coasts of the United States face rent increases, higher eviction rates, and a lack of affordable housing in the aftermath of a hurricane. The research will be presented in December at the annual meeting of the 2023 Society for Risk Analysis Annual Conference ...

Early research shows Gen Z perceives more dangers in life than previous generations

2023-12-13
There appears to be a common understanding that there is a mental health crisis among young people, but has society understood why?   As presented at the 2023 Society for Risk Analysis Annual Conference, Gabriel Rubin from Montclair State University conducted 40 interviews with members of Gen Z (as of publication) in an ongoing study about risk factors that have led to the current mental health crisis in young people. So far, this study has identified risk factors such as mass shootings, school lockdown drills, parental pressure, social media and the climate crisis.   Despite risk analysis research demonstrating that we ...

Annals of Allergy, Asthma and Immunology examines effects of climate change on allergic conditions

2023-12-13
ARLINGTON HEIGHTS, Ill. (December 13, 2023) – As we head into the new year, some issues may be coming into sharper focus for those involved in allergy-immunology issues. The current issue of Annals of Allergy, Asthma & Immunology, the scientific journal of the American College of Allergy, Asthma and Immunology, focuses its attention on a key problem affecting those with allergic conditions and the world today: climate change. “We recognize that climate change affects the global population, and that many people feel they as individuals don’t have much control,” says allergist Donald Leung, MD, PhD, Senior Executive Editor of Annals. “But ...

Unique cell-based approach for pulmonary arterial hypertension shown to be safe

2023-12-13
Infusions of potentially therapeutic cells derived from the heart are safe for people with pulmonary arterial hypertension, a form of high blood pressure that occurs in the blood vessels of the lungs and typically affects middle-aged women, according to a study led by Cedars-Sinai investigators. The Phase I clinical trial results are published in the peer-reviewed journal eBioMedicine, a Lancet journal. “Although several drugs are approved for pulmonary arterial hypertension, mortality remains high,” said Eduardo Marbán, MD, PhD, executive director of the Smidt Heart Institute at Cedars-Sinai, the Mark S. Siegel Family Foundation ...

Study analyzes what babies hear, say on six continents

2023-12-13
Elika Bergelson, associate professor of psychology at Harvard University, studies how infants and toddlers learn language from the world around them. The developmental psychologist specifically strives to parse the various theories that account for the onset and eventual mastery of language comprehension and production. Bergelson’s latest paper, published this month in the Proceedings of the National Academy of Sciences, represents a more global approach to developing and testing such theories.  Written ...

Heart attack deaths spike during the winter holidays

2023-12-13
DALLAS, Dec. 13, 2023 — The winter holidays can turn deadly as research shows that more people die from heart attacks during the last week of December than at any other time of the year. While being aware of the signs of a heart attack and taking steps to reduce your risk are important all year long, the American Heart Association, the world’s leading voluntary organization focused on heart and brain health for all, says that’s especially critical during the next few weeks. A number of scientific studies confirm ...

MD Anderson Research Highlights for December 13, 2023

2023-12-13
HOUSTON ― The University of Texas MD Anderson Cancer Center’s Research Highlights showcases the latest breakthroughs in cancer care, research and prevention. These advances are made possible through seamless collaboration between MD Anderson’s world-leading clinicians and scientists, bringing discoveries from the lab to the clinic and back. Recent developments at MD Anderson include a novel blood-based test to predict lung cancer relapses, improved detection of genes associated with complex traits, insights into how B cell activity influences immunodeficiency, novel targets that drive treatment ...

LAST 30 PRESS RELEASES:

Corpse flowers are threatened by spotty recordkeeping

Riding the AI wave toward rapid, precise ocean simulations

Are lifetimes of big appliances really shrinking?

Pink skies

Monkeys are world’s best yodellers - new research

Key differences between visual- and memory-led Alzheimer’s discovered

% weight loss targets in obesity management – is this the wrong objective?

An app can change how you see yourself at work

NYC speed cameras take six months to change driver behavior, effects vary by neighborhood, new study reveals

New research shows that propaganda is on the rise in China

Even the richest Americans face shorter lifespans than their European counterparts, study finds

Novel genes linked to rare childhood diarrhea

New computer model reveals how Bronze Age Scandinavians could have crossed the sea

Novel point-of-care technology delivers accurate HIV results in minutes

Researchers reveal key brain differences to explain why Ritalin helps improve focus in some more than others

Study finds nearly five-fold increase in hospitalizations for common cause of stroke

Study reveals how alcohol abuse damages cognition

Medicinal cannabis is linked to long-term benefits in health-related quality of life

Microplastics detected in cat placentas and fetuses during early pregnancy

Ancient amphibians as big as alligators died in mass mortality event in Triassic Wyoming

Scientists uncover the first clear evidence of air sacs in the fossilized bones of alvarezsaurian dinosaurs: the "hollow bones" which help modern day birds to fly

Alcohol makes male flies sexy

TB patients globally often incur "catastrophic costs" of up to $11,329 USD, despite many countries offering free treatment, with predominant drivers of cost being hospitalization and loss of income

Study links teen girls’ screen time to sleep disruptions and depression

Scientists unveil starfish-inspired wearable tech for heart monitoring

Footprints reveal prehistoric Scottish lagoons were stomping grounds for giant Jurassic dinosaurs

AI effectively predicts dementia risk in American Indian/Alaska Native elders

First guideline on newborn screening for cystic fibrosis calls for changes in practice to improve outcomes

Existing international law can help secure peace and security in outer space, study shows

Pinning down the process of West Nile virus transmission

[Press-News.org] Women may pay a "MOM PENALTY" when AI is used in hiring, new research from NYU Tandon School of Engineering suggests
Researchers uncover bias in popular Large Language Models when used to evaluate potential job candidates