PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

'Audeo' teaches artificial intelligence to play the piano

2021-02-04
(Press-News.org) Anyone who's been to a concert knows that something magical happens between the performers and their instruments. It transforms music from being just "notes on a page" to a satisfying experience.

A University of Washington team wondered if artificial intelligence could recreate that delight using only visual cues -- a silent, top-down video of someone playing the piano. The researchers used machine learning to create a system, called Audeo, that creates audio from silent piano performances. When the group tested the music Audeo created with music-recognition apps, such as SoundHound, the apps correctly identified the piece Audeo played about 86% of the time. For comparison, these apps identified the piece in the audio tracks from the source videos 93% of the time.

The researchers presented Audeo Dec. 8 at the NeurIPS 2020 conference.

"To create music that sounds like it could be played in a musical performance was previously believed to be impossible," said senior author Eli Shlizerman, an assistant professor in both the applied mathematics and the electrical and computer engineering departments. "An algorithm needs to figure out the cues, or 'features,' in the video frames that are related to generating music, and it needs to 'imagine' the sound that's happening in between the video frames. It requires a system that is both precise and imaginative. The fact that we achieved music that sounded pretty good was a surprise."

Audeo uses a series of steps to decode what's happening in the video and then translate it into music. First, it has to detect which keys are pressed in each video frame to create a diagram over time. Then it needs to translate that diagram into something that a music synthesizer would actually recognize as a sound a piano would make. This second step cleans up the data and adds in more information, such as how strongly each key is pressed and for how long.

"If we attempt to synthesize music from the first step alone, we would find the quality of the music to be unsatisfactory," Shlizerman said. "The second step is like how a teacher goes over a student composer's music and helps enhance it."

The researchers trained and tested the system using YouTube videos of the pianist Paul Barton. The training consisted of about 172,000 video frames of Barton playing music from well-known classical composers, such as Bach and Mozart. Then they tested Audeo with almost 19,000 frames of Barton playing different music from these composers and others, such as Scott Joplin.

Once Audeo has generated a transcript of the music, it's time to give it to a synthesizer that can translate it into sound. Every synthesizer will make the music sound a little different -- this is similar to changing the "instrument" setting on an electric keyboard. For this study, the researchers used two different synthesizers.

"Fluidsynth makes synthesizer piano sounds that we are familiar with. These are somewhat mechanical-sounding but pretty accurate," Shlizerman said. "We also used PerfNet, a new AI synthesizer that generates richer and more expressive music. But it also generates more noise."

Audeo was trained and tested only on Paul Barton's piano videos. Future research is needed to see how well it could transcribe music for any musician or piano, Shlizerman said.

"The goal of this study was to see if artificial intelligence could generate music that was played by a pianist in a video recording -- though we were not aiming to replicate Paul Barton because he is such a virtuoso," Shlizerman said. "We hope that our study enables novel ways to interact with music. For example, one future application is that Audeo can be extended to a virtual piano with a camera recording just a person's hands. Also, by placing a camera on top of a real piano, Audeo could potentially assist in new ways of teaching students how to play."

INFORMATION:

Kun Su and Xiulong Liu, both doctoral students in electrical and computer engineering, are co-authors on this paper. This research was funded by the Washington Research Foundation Innovation Fund as well as the applied mathematics and electrical and computer engineering departments.

For more information, contact Shlizerman at shlizee@uw.edu.



ELSE PRESS RELEASES FROM THIS DATE:

In symbiosis: Plants control the genetics of microbes

In symbiosis: Plants control the genetics of microbes
2021-02-04
Researchers from the University of Ottawa have discovered that plants may be able to control the genetics of their intimate root symbionts - the organism with which they live in symbiosis - thereby providing a better understanding of their growth. In addition to having a significant impact on all terrestrial ecosystems, their discovery may lead to improved eco-friendly agricultural applications. We talked to research lead Nicolas Corradi, Associate Professor in the Department of Biology and Research Chair in Microbial Genomics at the University of Ottawa, ...

Experimental vaccine blunts the deadliest of synthetic opioids

2021-02-04
LA JOLLA, CA--As the opioid epidemic raged on with an even greater force during COVID-19, the Scripps Research laboratory of chemist Kim Janda, PhD, has been working on new therapeutic interventions that may be able to prevent the bulk of deaths from opioid overdose. Janda and his team have developed experimental vaccines that have shown in rodents to blunt the deadly effects of fentanyl--which has been driving the boom in opioid deaths--as well as its even more fatal cousin, carfentanil, a growing source of overdoses and a chemical terrorist threat. "Synthetic opioids are not only extremely deadly, but also addictive and easy to manufacture, making them a formidable public health threat, especially when the coronavirus crisis is negatively impacting mental health," says Janda, the Ely ...

Switching nanolight on and off

Switching nanolight on and off
2021-02-04
A team of researchers led by Columbia University has developed a unique platform to program a layered crystal, producing imaging capabilities beyond common limits on demand. The discovery is an important step toward control of nanolight, which is light that can access the smallest length scales imaginable. The work also provides insights for the field of optical quantum information processing, which aims to solve difficult problems in computing and communications.   "We were able to use ultrafast nano-scale microscopy to discover a new way to control our crystals with light, turning elusive photonic properties on and off at will," said Aaron Sternbach, postdoctoral researcher ...

Scientists establish multiple primate models of SARS-CoV-2 airborne infection

2021-02-04
Army scientists evaluated three nonhuman primate species as potential models of SARS-CoV-2 airborne infection, according to results published online this week in PLOS ONE. Their work demonstrates that any of these species may be useful for testing vaccines and therapies in response to the COVID-19 pandemic, which has resulted in over 104 million cases and more than 2 million deaths worldwide in the past year. Given the global impact of COVID-19, experts are working rapidly to develop medical countermeasures, and testing in animal models is critically important to evaluate the efficacy of these products. Recent studies suggest that aerosol ...

New test provides fast and accurate diagnosis of liposarcomas

New test provides fast and accurate diagnosis of liposarcomas
2021-02-04
Philadelphia, February 4, 2021 - Researchers have leveraged the latest advances in RNA technology and machine learning methods to develop a gene panel test that allows for highly accurate diagnosis of the most common types of liposarcoma. It quickly and reliably distinguishes benign lipomas from liposarcomas and can be performed in laboratories at a lower cost than current "gold standard" tests. The new assay is described in The Journal of Molecular Diagnosis, published by Elsevier. "Liposarcomas are a type of malignant cancer that is difficult to diagnose because, even under a microscope, it is hard to differentiate liposarcomas from benign tumors or other types ...

New study examines addiction medicine treatment in Vietnam

2021-02-04
An assessment published this week in the journal The Lancet HIV provides new insight about an initiative to integrate treatment of opioid use disorder along with HIV in Vietnam. The study marks one of the first scientifically robust assessments of a new model of treating HIV in lower or middle income countries where injection drug use is a major cause of HIV infection. It also suggests the importance of building support for peer and community connections to tackle the opioid epidemic that continues to ravage the United States in the midst of the COVID-19 pandemic. The study was led by scientists and physicians at Hanoi Medical University and Oregon Health & Science University. "Our study suggests that countries that ...

Imaging the first moments of a body plan emerging in the embryo

2021-02-04
WOODS HOLE, Mass. -- Egg cells start out as round blobs. After fertilization, they begin transforming into people, dogs, fish, or other animals by orienting head to tail, back to belly, and left to right. Exactly what sets these body orientation directions has been guessed at but not seen. Now researchers at the Marine Biological Laboratory (MBL) have imaged the very beginning of this cellular rearrangement, and their findings help answer a fundamental question. "The most interesting and mysterious part of developmental biology is the origin of the body axis in animals," said researcher Tomomi Tani. An MBL scientist in the Eugene Bell Center at the time of the research, Tani is now with Japan's National Institute of Advanced Industrial Science and Technology. The work by Tani and ...

Mysterious organic scum boosts chemical reaction efficiency, may reduce chemical waste

Mysterious organic scum boosts chemical reaction efficiency, may reduce chemical waste
2021-02-04
CHAMPAIGN, Ill. -- Chemical manufacturers frequently use toxic solvents such as alcohols and benzene to make products like pharmaceuticals and plastics. Researchers are examining a previously overlooked and misunderstood phenomenon in the chemical reactions used to make these products. This discovery brings a new fundamental understanding of catalytic chemistry and a steppingstone to practical applications that could someday make chemical manufacturing less wasteful and more environmentally sound. The study led by University of Illinois Urbana-Champaign researcher David Flaherty, University of Minnesota, Twin Cities researcher Matthew Neurock and Virginia Tech researcher Ayman Karim is published in the journal Science. Combining ...

An optical coating like no other

An optical coating like no other
2021-02-04
For more than a century, optical coatings have been used to better reflect certain wavelengths of light from lenses and other devices or, conversely, to better transmit certain wavelengths through them. For example, the coatings on tinted eyeglasses reflect, or "block out," harmful blue light and ultraviolet rays. But until now, no optical coating had ever been developed that could simultaneously reflect and transmit the same wavelength, or color. In a paper in Nature Nanotechnology, researchers at the University of Rochester and Case Western Reserve University describe a new class of optical coatings, so-called Fano Resonance Optical Coatings (FROCs), that can be used on filters to reflect and transmit colors of remarkable purity. In addition, ...

Comb-like etching regulated growth for large-area graphene nanoribbon arrays

Comb-like etching regulated growth for large-area graphene nanoribbon arrays
2021-02-04
The rapid development of silicon-based transistors leads to its integration getting closer to the limit of Moore's law. Graphene is expected to become the next generation of mainstream chip materials due to its ultra-high carrier mobility. However, it is difficult to obtain a high on/off current ratio for intrinsic graphene-based transistor owing to the absence of bandgap. Graphene nanoribbons, which possess a tunable bandgap and unique optoelectrical properties, have attracted extensive attention and exploration. Nowadays, the preparation of graphene nanoribbons is underdeveloped, and common strategies include the clip of carbon materials (graphene films, carbon nanotubes, or graphite) ...

LAST 30 PRESS RELEASES:

Chung-Ang University develops chloride-resistant Ru nanocatalysts for sustainable hydrogen production from seawater

Afghanistan’s August 2025 earthquake reveals the cost of international isolation, UN scientists warn

Shortlist announced for Panmure House Prize

Small nuclear RNA base editing a safer alternative to CRISPR, UC San Diego researchers find

Can Hayabusa2 touchdown? New study reveals space mission’s target asteroid is tinier and faster than thought

Millisecond windows of time may be key to how we hear, study finds

Graz University of Technology opens up new avenues in lung cancer research with digital cell twin

Exoplanets are not water worlds

Study shows increasing ‘healthy competition’ between menu options nudges patients towards greener, lower-fat hospital food choices

New insights into melanoma plasticity uncover a critical role of iron metabolism

A graphene sandwich — deposited or transferred?

New light-powered motor fits inside a strand of hair

Oil rig study reveals vital role of tiny hoverflies

Children’s Hospital of Philadelphia researchers boost widespread use of dental varnish across pediatric network

iRECODE: A new computational method that brings clarity to single-cell analysis

New NUS-MOH study: Singapore’s healthcare sector carbon emissions 18% lower than expected, a milestone in the city-state’s net zero journey

QUT scientists create material to turn waste heat into clean power

Major new report sets out how to tackle the ‘profound and lasting impact’ of COVID-19 on cardiovascular health

Cosmic crime scene: White dwarf found devouring Pluto-like icy world

Major report tackles Covid’s cardiovascular crisis head-on

A third of licensed GPs in England not working in NHS general practice

ChatGPT “thought on the fly” when put through Ancient Greek maths puzzle

Engineers uncover why tiny particles form clusters in turbulent air

GLP-1RA drugs dramatically reduce death and cardiovascular risk in psoriasis patients

Psoriasis linked to increased risk of vision-threatening eye disease, study finds

Reprogramming obesity: New drug from Italian biotech aims to treat the underlying causes of obesity

Type 2 diabetes may accelerate development of multiple chronic diseases, particularly in the early stages, UK Biobank study suggests

Resistance training may improve nerve health, slow aging process, study shows

Common and inexpensive medicine halves the risk of recurrence in patients with colorectal cancer

SwRI-built instruments to monitor, provide advanced warning of space weather events

[Press-News.org] 'Audeo' teaches artificial intelligence to play the piano