PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

Audio-guided self-supervised learning for disentangled visual speech representations

Audio-guided self-supervised learning for disentangled visual speech representations
2025-01-07
(Press-News.org)

Learning visual speech representations from talking face videos is an important problem for several speech-related tasks, such as lip reading, talking face generation, audio-visual speech separation, and so on. The key difficulty lies in tackling speech-irrelevant factors presented in the videos, such as lighting, resolution, viewpoints, head motion, and so on.

To solve the problems, a research team led by Shuang YANG publishes their new research on 15 December 2024 in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature.

The team proposes to disentangle speech-relevant and speech-irrelevant facial movements from videos in a self-supervised learning manner. The proposed method can learn discriminative disentangled speech representations from videos and can benefit the lip reading task by a straightforward method like knowledge distillation. Both qualitative and quantitative results on the popular visual speech datasets LRW and LRS2-BBC show the effectiveness of their method.

In the research, the researchers observe the speech process and find that speech-relevant and speech-irrelevant facial movements are differences in the frequency of occurrence. Specifically, speech-relevant facial movements always occur at a higher frequency than speech-irrelevant ones. Moreover, the researchers find that the speech-relevant facial movements are consistently synchronized with the accompanying audio speech signal.

Based on the new observations above, the researchers introduce a novel two-branch network to decompose the visual changes between two frames in the same video into speech-relevant and speech-irrelevant components. For speech-relevant branch, they introduce the high-frequency audio signal to guide the learning of speech-relevant cues. For the speech-irrelevant branch, they introduce an information bottleneck to restrict the capacity from acquiring high-frequency and fine-grained speech-relevant information.

Future work can focus on exploring more explicit auxiliary tasks and constraints beyond the reconstruction task to capture speech cues from videos. Meanwhile, it's also a nice try to combine multiple types of knowledge representations to enhance the obtained speech representations.  

DOI: 10.1007/s11704-024-3787-8

END


[Attachments] See images for this press release:
Audio-guided self-supervised learning for disentangled visual speech representations Audio-guided self-supervised learning for disentangled visual speech representations 2 Audio-guided self-supervised learning for disentangled visual speech representations 3

ELSE PRESS RELEASES FROM THIS DATE:

From logs to security: How process analysis is transforming access control

From logs to security: How process analysis is transforming access control
2025-01-07
Researchers at the University of Electro-Communications have developed a groundbreaking framework for improving system security by analyzing business process logs. This framework focuses on ensuring that role-based access control (RBAC) rules-critical to managing who can access specific system resources-are correctly implemented. Noncompliance with these rules, whether due to error or malicious activity, can result in unauthorized access and pose significant risks to organizations.   RBAC is a widely used access control model that relies on predefined roles assigned to users. However, as business processes become more complex, ensuring ...

Dronedarone inhibits the proliferation of esophageal squamous cell carcinoma through the CDK4/CDK6-RB1 axis in vitro and in vivo

Dronedarone inhibits the proliferation of esophageal squamous cell carcinoma through the CDK4/CDK6-RB1 axis in vitro and in vivo
2025-01-07
Esophageal squamous cell carcinoma (ESCC) is a severe health threat, being a predominant subtype of esophageal cancer and contributing significantly to cancer-related mortality globally. Despite advancements in combination therapies, patient prognosis remains poor, highlighting an urgent need for novel treatment strategies. In this context, a study explores the potential of dronedarone, an FDA-approved drug, in inhibiting ESCC proliferation through the CDK4/CDK6-RB1 axis, both in vitro and in vivo. The research reveals that dronedarone, ...

Photonic nanojet-regulated soft microalga-robot

Photonic nanojet-regulated soft microalga-robot
2025-01-07
Micro/nanorobots hold exciting prospects for executing different tasks in complex microenvironments due to their small size, high flexibility, controllability, and environmental adaptability. However, traditional rigid micro/nanorobots are still difficult to perform different biomedical tasks in complex and unstructured narrow microenvironments due to their limited flexibility and insufficient deformability. To address this problem, in a new paper published in PhotoniX, a team of scientists led by Professor Hongbao Xin from Institute of Nanophotonics, Jinan University, China, has developed a new soft microalga robot (saBOT). They innovatively used microalga, ...

How do directional connections shape complex dynamics in neuronal networks?

How do directional connections shape complex dynamics in neuronal networks?
2025-01-07
Uncovering the relationship between structure (connectivity) and function (neuronal activity) is a fundamental question across many areas of biology. However, investigating this directly in animal brains is challenging because of the immense complexity of their neural connections and the invasive surgeries that are typically needed. Lab-grown neurons with artificially-controlled connections have the possibility of becoming a useful alternative to animal testing, particularly as we learn how to accurately characterize their behaviour. A research team at Tohoku University used microfluidic devices to reveal how directional connections shape the complex dynamics ...

Drug-resistant hookworms put pets and people at risk

Drug-resistant hookworms put pets and people at risk
2025-01-07
Canine hookworms are becoming increasingly resistant to drugs across Australia, according to new research. Scientists at The University of Queensland and The University of Sydney have identified widespread resistance to benzimidazole-based dewormers which are commonly used to treat gastrointestinal parasites in dogs. Dr Swaid Abdullah from UQ’s School of Veterinary Science said almost 70 per cent of the hookworm samples studied showed genetic mutations that can cause drug resistance. “This is a big problem, as hookworm infections ...

New strontium isotope map of Sub-Saharan Africa is a powerful tool for archaeology, forensics, and wildlife conservation

2025-01-07
A team of researchers led by UC Santa Cruz recently released a sophisticated new map that reveals, for the first time, the unique “geologic fingerprints” for most of the African continent.  The map will help archaeologists, conservation scientists, and forensics experts match artifacts and plant, animal, and human remains found at locations around the world back to their most likely region of origin within Africa, offering new insights on issues ranging from the history of the transatlantic slave trade to modern wildlife trafficking and human migration patterns.  The research team’s ...

‘Sandwich carers’ experience decline in mental and physical health

2025-01-07
People who care for both their children and older family members – also known as ‘sandwich carers’ – suffer from deterioration in both their mental and physical health over time, finds a new study by UCL researchers. The research, published in Public Health, analysed data from around 2,000 sandwich carers and 2,000 non-sandwich carers from the UK Household Longitudinal Study between 2009 and 2020. Sandwich carers juggle the responsibilities of caring for ageing parents or older relatives while raising dependent children ...

A new way to determine whether a species will successfully invade an ecosystem

2025-01-06
CAMBRIDGE, MA -- When a new species is introduced into an ecosystem, it may succeed in establishing itself, or it may fail to gain a foothold and die out. Physicists at MIT have now devised a formula that can predict which of those outcomes is most likely. The researchers created their formula based on analysis of hundreds of different scenarios that they modeled using populations of soil bacteria grown in their laboratory. They now plan to test their formula in larger-scale ecosystems, including forests. This approach could also be helpful in predicting whether probiotics or fecal microbiota treatments (FMT) would successfully combat infections of the human GI tract. “People ...

A change in the weather in the U.S. Corn Belt

2025-01-06
A change in the weather in the U.S. Corn Belt Intensive farming and shallow groundwater affect precipitation patterns The sweeping land use changes and irrigation of the U.S. Corn Belt, along with the influence of the area’s shallow groundwater, have significantly altered precipitation patterns in that vital agricultural region, new research shows. The study, published in the Proceedings of the National Academy of Sciences, focuses on “precipitation recycling” — a process in which the moisture released to the atmosphere by plants, soils, lakes, and other features of the landscape returns to the same area in the form of rain. By using advanced ...

How we classify flood risk may give developers, home buyers a false sense of security

2025-01-06
Common methods of communicating flood risk may create a false sense of security, leading to increased development in areas threatened by flooding. This phenomenon, called the “safe development paradox,” is described in a new paper from North Carolina State University. Lead author Georgina Sanchez, a research scholar in NC State’s Center for Geospatial Analytics, said this may be an unintended byproduct of how the Federal Emergency Management Agency classifies areas based on their probability of dangerous flooding. Known as flood mapping, this classification system describes areas in terms of their likelihood of being flooded each year. These ...

LAST 30 PRESS RELEASES:

Dual scalable annealing processors: overcoming capacity and precision limits

Genetic secrets of rice pave way for future farming and conservation

A vast molecular cloud, long invisible, is discovered near solar system

Extreme monsoon changes threaten the Bay of Bengal's role as a critical food source

New pests and diseases will cut UK tree growth

Elucidating the double duty of sleep in memory processing

Many paths to an angry bird

Balancing nature restoration and land use: a path to sustainable growth in the EU

With AI, researchers can now identify the smallest crystals

Every dose counts: Safeguarding the success of vaccination in Europe

Can exercise and rehab services be integrated into breast cancer care?

Simple test could better predict your risk of heart disease

Global study links consumption of ultraprocessed foods to preventable premature deaths

Accurate and rapid arthritis diagnosis in just 10 minutes

Hospital-based outbreak detection system saves lives

AACR: Topical treatment offers relief from painful skin rash caused by targeted cancer therapy

Buprenorphine treatment in pregnancy and maternal-infant outcomes

Donor lungs safely preserved up to 20 hours out-of-body prior to transplantation

Experts at ISHLT report urgent need for pediatric heart support devices

DCD heart transplantation reaches 10-year mark, now up to 30% of transplant volumes

Immunotherapy before and after surgery improves outcomes in head and neck cancer

Donor hearts are traveling longer distances with machine perfusion

Six leading organizations unite to launch the pediatric heart transplant alliance

Effect of coupled wing motion on the aerodynamic performance during different flight stages of pigeon

Cercus electric stimulation enables cockroach with trajectory control and spatial cognition training

Day-long conference addresses difficult to diagnose lung disease

First-ever cardiogenic shock academy features simulation lab

Thirty-year mystery of dissonance in the “ringing” of black holes explained

Less intensive works best for agricultural soil

Arctic rivers project receives “national champion” designation from frontiers foundation

[Press-News.org] Audio-guided self-supervised learning for disentangled visual speech representations