Only Amazon MTurk’s ‘master’ workers provide reliable research data quality
2025-07-16
(Press-News.org)
A new study led by Dr. Vadim Axelrod, of the Gonda (Goldschmied) Multidisciplinary Brain Research Center at Bar-Ilan University, has revealed serious concerns about the quality of data collected on Amazon Mechanical Turk’s (MTurk) — a platform widely used for behavioral and psychological research.
MTurk, an online crowdsourcing marketplace where individuals complete small tasks for payment, has served as a key resource for researchers for over 15 years. Despite previous concerns about participant quality, the platform remains popular within the academic community. Dr. Axelrod’s team set out to rigorously assess the current quality of data produced by MTurk participants.
The study, involving over 1,300 participants across main and replication experiments, employed a straightforward but powerful method: repeating identical questionnaire items to measure response consistency. “If a participant is reliable, their answers to repeated questions should be consistent," added Dr. Axelrod. In addition, the study included different types of “attentional catch” questions that should be easily answered by any attentive respondent.
The findings, just published by Royal Society Open Science, were stark: the majority of participants from MTurk’s general worker pool failed the attention checks and demonstrated highly inconsistent responses, even when the sample was limited to users with a 95% or higher approval rating.
“It’s hard to trust the data of someone who claims a runner isn’t tired after completing a marathon in extremely hot weather or that a cancer diagnosis would make someone glad,” Dr. Axelrod noted. “The participants did not lack the knowledge to answer such attentional catch questions — they just weren’t paying sufficient attention. The implication is that their responses to the main questionnaire may be equally random.”
By contrast, Amazon’s elite “Master” workers — selected by Amazon based on high performance across previous tasks — consistently produced high-quality data. The authors recommend using Master workers for future research, taking into consideration that these participants are much more experienced and far fewer in number.
“Reliable data is the foundation of any empirical science,” said Dr. Axelrod. “Researchers need to be fully informed about the reliability of their participant pool. Our findings suggest that caution is warranted when using MTurk’s general pool for behavioral research.”
END
ELSE PRESS RELEASES FROM THIS DATE:
2025-07-15
Glaciers hold layers of history preserved in ice, offering unique insights into Earth's past that can also help us interpret the future. Trapped amidst the frozen water are microscopic deposits of dust, pollen, and even pollutants that scientists can use to examine environmental changes through time. DRI’s Ice Core Lab has used this technique to highlight atmospheric lead pollution and economic turbulence in Ancient Rome. Now, their latest study found that a glacier in the French Alps dates back to the last Ice Age – the oldest known glacier ice in the region. Serving as a record that spans through the development of agriculture ...
2025-07-15
Yoga, Tai Chi, walking and jogging may be the best forms of exercise to improve sleep quality and ease insomnia, suggest the findings of a comparative pooled data analysis published in the online journal BMJ Evidence Based Medicine.
The findings back the use of exercise as a primary treatment strategy for poor sleep patterns, say the researchers.
Characterised by difficulties falling and staying asleep, and early morning awakening, the prevalence of insomnia ranges from 4-22%, note the researchers. It is associated with heightened risks of various mental and physical health conditions, including dementia and cardiovascular disease.
Drug ...
2025-07-15
Medical tourism for bariatric and weight reduction surgery needs urgent regulation to protect recipients’ health, especially as the data show that tourist numbers are increasing despite the advent of weight loss drugs, say experts in a commentary published online in BMJ Global Health.
The high prevalence of obesity coupled with healthcare resource constraints and increased globalisation have resulted in more people accessing obesity treatment abroad, amid the rapid growth of services to meet this demand, note Dr Jessica McGirr of the Obesity Research and Care Group RCSI University of Medicine and Health Sciences, Dublin, Ireland and ...
2025-07-15
Most up-to-date report estimates that development assistance for health (DAH) will further decline by 2030, following steep cutbacks from major donors in 2025, including the US, UK, France, and Germany.
The drop comes after DAH fell more than 50% in 2025 from an all-time high of $80 billion in 2021.
Sub-Saharan Africa could be hardest hit by cuts in 2025.
The potential increase in infectious diseases in low- and middle-income countries poses a global health concern that could impact all nations, including those in high-income regions, due to the risk ...
2025-07-15
Exercise could ease symptoms for people with Chronic Lymphocytic Leukaemia, but support and adequate guidance is lacking
Personalised exercise guidance is needed for people living with Chronic Lymphocytic Leukaemia (CLL), the most common type of blood cancer, according to a new study led by the University of Surrey.
The research, published on eJHaem, surveyed 128 CLL patients across the UK, and found that while nearly 80% expressed a desire to participate in exercise programs, 70% reported never receiving any related advice from their healthcare providers.
Dr David ...
2025-07-15
University of Cambridge media release
Lost English legend decoded, solving Chaucerian mystery and revealing a medieval preacher’s meme
UNDER STRICT EMBARGO UNTIL 19:01 US EDT ON TUESDAY 15TH JULY 2025 / 00:01 UK TIME (BST) ON WEDNESDAY 16TH JULY 2025
A medieval literary puzzle which has stumped scholars including M.R. James for 130 years has finally been solved. Cambridge scholars now believe the Song of Wade, a long lost treasure of English culture, was a chivalric romance not a monster-filled epic. The discovery solves the most famous mystery ...
2025-07-15
The stigma of having alopecia causes more depression and anxiety than the disease itself, new research has found.
The study, led by King’s College London is the largest of its kind and published in the British Journal of Dermatology, found that the impact on the quality of life on people with alopecia is determined by the negative perceptions and stigma associated with having the illness.
Supported by an academic research grant by Pfizer, the study surveyed 596 adults with alopecia areata and found that over 80% reported ...
2025-07-15
For the first time, a new study has revealed how and when we make eye contact—not just the act itself—plays a crucial role in how we understand and respond to others, including robots.
Led by cognitive neuroscientist Dr Nathan Caruana, researchers from the HAVIC Lab at Flinders University asked 137 participants to complete a block-building task with a virtual partner.
They discovered that the most effective way to signal a request was through a specific gaze sequence: looking at an object, making eye contact, then looking back at the same object. ...
2025-07-15
Innovative virtual eye clinics in shopping centres could significantly reduce waiting times for routine eye appointments, UCL-led research suggests in a first-of-its-kind study.
The research, published in the journal BMJ Open, evaluates the impact of a community-based ‘pop-up’ eye clinic set up in Brent Cross shopping centre in north London in September 2021, at the height of the pandemic, on reducing the post-COVID-19 appointment backlog.
The researchers looked at data from 69,257 appointments attended by 39,357 patients with stable glaucoma and medical retinal conditions at sites ...
2025-07-15
The University of Birmingham, UK, and Brazilian-based CBMM, have partnered on a project to ensure the future supply of Niobium, a non-critical but rare and important metal and an essential component for a carbon recycling technology that could radically reduce emissions from energy- and carbon-intensive foundation industries.
CBMM, a global leader in the production of Niobium products, will work with Birmingham researchers led by Professor Yulong Ding to improve the efficiency of production, and reduce the ...
LAST 30 PRESS RELEASES:
[Press-News.org] Only Amazon MTurk’s ‘master’ workers provide reliable research data quality
