PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

AI can predict study results better than human experts

2024-11-27
(Press-News.org) Large language models, a type of AI that analyses text, can predict the results of proposed neuroscience studies more accurately than human experts, finds a new study led by UCL (University College London) researchers.

The findings, published in Nature Human Behaviour, demonstrate that large language models (LLMs) trained on vast datasets of text can distil patterns from scientific literature, enabling them to forecast scientific outcomes with superhuman accuracy.

The researchers say this highlights their potential as powerful tools for accelerating research, going far beyond just knowledge retrieval.

Lead author Dr Ken Luo (UCL Psychology & Language Sciences) said: “Since the advent of generative AI like ChatGPT, much research has focused on LLMs' question-answering capabilities, showcasing their remarkable skill in summarising knowledge from extensive training data. However, rather than emphasising their backward-looking ability to retrieve past information, we explored whether LLMs could synthesise knowledge to predict future outcomes.

“Scientific progress often relies on trial and error, but each meticulous experiment demands time and resources. Even the most skilled researchers may overlook critical insights from the literature. Our work investigates whether LLMs can identify patterns across vast scientific texts and forecast outcomes of experiments.”

The international research team began their study by developing BrainBench, a tool to evaluate how well large language models (LLMs) can predict neuroscience results.

BrainBench consists of numerous pairs of neuroscience study abstracts. In each pair, one version is a real study abstract that briefly describes the background of the research, the methods used, and the study results. In the other version, the background and methods are the same, but the results have been modified by experts in the relevant neuroscience domain to a plausible but incorrect outcome.

The researchers tested 15 different general-purpose LLMs and 171 human neuroscience experts (who had all passed a screening test to confirm their expertise) to see whether the AI or the person could correctly determine which of the two paired abstracts was the real one with the actual study results.

All of the LLMs outperformed the neuroscientists, with the LLMs averaging 81% accuracy and the humans averaging 63% accuracy. Even when the study team restricted the human responses to only those with the highest degree of expertise for a given domain of neuroscience (based on self-reported expertise), the accuracy of the neuroscientists still fell short of the LLMs, at 66%. Additionally, the researchers found that when LLMs were more confident in their decisions, they were more likely to be correct.* The researchers say this finding paves the way for a future where human experts could collaborate with well-calibrated models.

The researchers then adapted an existing LLM (a version of Mistral, an open-source LLM) by training it on neuroscience literature specifically. The new LLM specialising in neuroscience, which they dubbed BrainGPT, was even better at predicting study results, attaining 86% accuracy (an improvement on the general-purpose version of Mistral, which was 83% accurate).

Senior author Professor Bradley Love (UCL Psychology & Language Sciences) said: “In light of our results, we suspect it won’t be long before scientists are using AI tools to design the most effective experiment for their question. While our study focused on neuroscience, our approach was universal and should successfully apply across all of science.

“What is remarkable is how well LLMs can predict the neuroscience literature. This success suggests that a great deal of science is not truly novel, but conforms to existing patterns of results in the literature. We wonder whether scientists are being sufficiently innovative and exploratory.”

Dr Luo added: “Building on our results, we are developing AI tools to assist researchers. We envision a future where researchers can input their proposed experiment designs and anticipated findings, with AI offering predictions on the likelihood of various outcomes. This would enable faster iteration and more informed decision-making in experiment design.”

The study was supported by the Economic and Social Research Council (ESRC), Microsoft, and a Royal Society Wolfson Fellowship, and involved researchers in UCL, University of Cambridge, University of Oxford, Max Planck Institute for Neurobiology of Behavior (Germany), Bilkent University (Turkey) and other institutions in the UK, US, Switzerland, Russia, Germany, Belgium, Denmark, Canada, Spain and Australia.

 

* When presented with two abstracts, the LLM computes the likelihood of each, assigning a perplexity score to represent how surprising each is based on its own learned knowledge as well as the context (background and method). The researchers assessed LLMs' confidence by measuring the difference in how surprising/perplexing the models found real versus fake abstracts – the greater this difference, the greater the confidence, which correlated with a higher likelihood the LLM had picked the correct abstract.

END


ELSE PRESS RELEASES FROM THIS DATE:

Brain stimulation effectiveness tied to learning ability, not age

2024-11-27
As we age, our cognitive and motor functions deteriorate, which in turn affects our independence and overall quality of life. Research efforts to ameliorate or even completely abolish this have given rise to technologies that show a lot of promise. Among these is non-invasive brain stimulation: a term encompassing a set of techniques that can affect brain functions externally and noninvasively, without the need for surgery or implants. One such promising technique, in particular, is anodal transcranial direct current stimulation (atDCS), which uses ...

Making a difference: Efficient water harvesting from air possible

Making a difference: Efficient water harvesting from air possible
2024-11-27
Harvesting water from the air and decreasing humidity are crucial to realizing a more comfortable life for humanity. Water-adsorption polymers have been playing a key part in atmospheric water harvesting and desiccant air conditioning, but desorption so that the polymers can be efficiently reused has been an issue. Now, Osaka Metropolitan University researchers have found a way to make desorption of these polymers more efficient. Usually, heat of around 100°C is required to desorb these polymers, but Graduate School of Engineering student Daisuke Ikegawa, Assistant Professor Arisa Fukatsu, ...

World’s most common heart valve disease linked to insulin resistance in large national study

2024-11-27
A large new population study of men over 45 indicates insulin resistance may be an important risk factor for the development of the world’s most common heart valve disease – aortic stenosis (AS).    Published today in the peer-reviewed journal Annals of Medicine, the findings are believed to be the first to highlight this previously unrecognised risk factor for the disease.    It is hoped that by demonstrating this link between AS and insulin resistance – when cells fail to respond effectively to insulin and the body makes more than necessary to maintain normal glucose ...

Study unravels another piece of the puzzle in how cancer cells may be targeted by the immune system

2024-11-27
Effective immunity hinges on the ability to sense infection and cellular transformation. In humans, there is a specialised molecule on the surface of cells termed MR1. MR1 allows sensing of certain small molecule metabolites derived from cellular and microbial sources; however, the breadth of metabolite sensing is unclear. Published in PNAS, researchers at the Monash University Biomedicine Discovery Institute have identified a form of Vitamin B6 bound to MR1 as a means of engaging tumour-reactive immune cells. The work involved an international collaborative team co-led by researchers from the University of Melbourne. According ...

Long-sought structure of powerful anticancer natural product solved by integrated approach

Long-sought structure of powerful anticancer natural product solved by integrated approach
2024-11-27
    A collaborative effort by the research groups of Professor Haruhiko Fuwa from Chuo University and Professor Masashi Tsuda from Kochi University has culminated in the structure elucidation and total synthesis of anticancer marine natural products, iriomoteolide-1a and -1b. These natural products were originally isolated from the marine dinoflagellate collected off the Iriomote Island, Okinawa, Japan.   Because of its potent anticancer activity, iriomoteolide-1a is an intriguing natural product that attract immense attention from the chemical community around the globe. ...

World’s oldest lizard wins fossil fight

World’s oldest lizard wins fossil fight
2024-11-27
A storeroom specimen that changed the origins of modern lizards by millions of years has had its identity confirmed. The tiny skeleton, unearthed from Triassic-aged rocks in a quarry near Bristol, is at least 205 million years old and the oldest modern-type lizard on record. Recently, the University of Bristol team’s findings came under question, but fresh analysis, published today in Royal Society Open Science, proves that the fossil is related to modern anguimorphs such as anguids and monitors. The discovery ...

Simple secret to living a longer life

Simple secret to living a longer life
2024-11-27
If everyone in the United States population was as active as the top 25 per cent, individuals over the age of 40 could add five years to their life, according to a new study led by Griffith University researchers.   Physical activity has long been known to be good for health, however estimates have varied regarding how much benefit could be gained from a defined amount of activity, both for individuals and for populations.  This latest study used accelerometry to gain an accurate view of the population’s physical activity levels instead of relying on survey responses as per other studies, and found the benefits were around twice as strong ...

Same plant, different tactic: Habitat determines response to climate

Same plant, different tactic: Habitat determines response to climate
2024-11-27
Plants need light to grow, but too much light can induce damage to the photosynthetic complex known as photosystem II. It is known that plants adapted to growing under full sun repair this light-induced damage more. But this repair activity slows down in colder temperatures. An Osaka Metropolitan University-led international research team has now found some clues to how plants survive in colder regions. Graduate School of Science Associate Professor Riichi Oguchi and colleagues from Australia, Austria, and Japan grew Arabidopsis thaliana (commonly ...

Drinking plenty of water may actually be good for you

2024-11-27
Public health recommendations generally suggest drinking eight cups of water a day. And many people just assume it’s healthy to drink plenty of water.   Now researchers at UC San Francisco have taken a systematic look at the available evidence. They concluded that drinking enough water can help with weight loss and prevent kidney stones, as well as migraines, urinary tract infections and low blood pressure.   “For such a ubiquitous and simple intervention, the evidence hasn’t been clear and the benefits were not well-established, so we wanted to take a closer look,” said senior and corresponding author Benjamin Breyer, MD, MAS, the Taube ...

Men at high risk of cardiovascular disease face brain health decline 10 years earlier than women

2024-11-27
Men with cardiovascular disease risk factors, including obesity, face brain health decline a decade earlier—from their mid 50s to mid 70s—than similarly affected women who are most susceptible from their mid 60s to mid 70s, suggest the findings of a long term study, published online in the Journal of Neurology Neurosurgery & Psychiatry.   The most vulnerable regions of the brain are those involved in processing auditory information, aspects of visual perception, emotional processing and memory, with ...

LAST 30 PRESS RELEASES:

Wrong trees in the wrong place can make cities hotter at night, study reveals

New gene therapy reverses heart failure in large animal model

Young children less likely than adults to see discrimination as harmful

Tiny poops in the ocean may help solve the carbon problem

Study offers insight into chloroplast evolution

Advancing the synthesis of two-dimensional gold monolayers

Human disruption is driving ‘winner’ and ‘loser’ tree species shifts across Brazilian forests

A novel heme-model compound that treats lethal gas poisoning

Shape-changing device helps visually impaired people perform location task as well as sighted people - EMBARGO: Tuesday 10 December (10:00 UK time)

AI predicts that most of the world will see temperatures rise to 3°C much faster than previously expected

Second round of FRONTIERS Science Journalism Residency Program awards grants to ten journalists

The inequity of wildfire rescue resources in California

Aerosol pollutants from cooking may last longer in the atmosphere – new study

Breakthrough in the precision engineering of four-stranded β-sheets

Family income predicts adult problems more than neighborhood poverty

Leading stress expert Ron de Kloet on hormone's dual nature: From protection to harm

Almost half of young vapers are able to stop with quitline help

After a divisive election, most U.S. adults ready to avoid politics this holiday

Food insecurity in LA County remains well above national average, despite slight decline

People with a positive attitude are built differently

AML, sickle cell disease research among highlights of UC ASH abstracts

Dozens of presentations advance multiple myeloma research at the 2024 American Society for Hematology (ASH) meeting

ASH 2024: Study shows that genetic mutations accumulate in smokers with myelodysplastic syndromes and worsen outcomes

Nature inspires self-assembling helical polymer

Could US-style summer holiday programs boost Aussie kids’ health?

Towards safer, higher performance batteries through network topology optimization

ASH: Triplet combination regimens demonstrate high response rates in multiple leukemias

Toxoplasma gondii parasite uses unconventional method to make proteins for evasion of drug treatment

US e-scooter/e-bike injuries have tripled since 2019, fuelled by alcohol/substance use

Cost stops 1 in 6 US adults with asthma from taking meds as prescribed, study suggests

[Press-News.org] AI can predict study results better than human experts