(Press-News.org) A badly dubbed foreign film makes a viewer yearn for subtitles; even subtle discrepancies between words spoken and facial motion are easy to detect. That's less likely with a method developed by Disney Research that analyzes an actor's speech motions to literally put new words in his mouth.
The researchers found that the facial movements an actor makes when saying "clean swatches," for instance, are the same as those for such phrases as "likes swats," "then swine," or "need no pots."
Sarah Taylor and her colleagues at Disney Research Pittsburgh and the University of East Anglia, report that this approach, based on "dynamic visemes," or facial movements associated with speech sounds, produces far more alternative word sequences than approaches that use conventional visemes, which are static lip shapes associated with sounds.
They will present their findings April 23 at ICASSP 2015, the IEEE International Conference on Acoustics, Speech and Signal Processing in Brisbane, Australia.
"This work highlights the extreme level of ambiguity in visual-only speech recognition," Taylor said. But whereas a lip reader battles against ambiguity, using context to figure out the most likely words that were spoken, the Disney team exploited that ambiguity to find alternative words.
"Dynamic visemes are a more accurate model of visual speech articulation than conventional visemes and can generate visually plausible phonetic sequences with far greater linguistic diversity," she added.
The relationship between what a person sees and what a person hears is complex, Taylor noted. In the "clean swatches" example, for instance, the alternative word sequences can vary in the number of syllables yet remain visually consistent with the video.
Speech redubbing, such as translating movies, television shows and video games into another language, or removing offensive language from a TV show, typically involves meticulous scripting to select words that match lip motions and re-recording by a skilled actor. Automatic speech redubbing, as explored in this study, is a largely unexplored area of research.
With conventional static visemes, a lip shape is assumed to represent a small number of different sounds and the mapping of those units to distinctive sounds is incomplete, which the researchers found limited the number of alternative word sequences that could be automatically generated.
Dynamic visemes represents the set of speech-related lip motions and their mapping to sequences of spoken sounds. The researchers exploit this more general mapping to identify a large number of different word sequences for a given facial movement.
"The method using dynamic visemes produces many more plausible alternative word sequences that are perceivably better than those produced using a static viseme approaches," Taylor said.
INFORMATION:
In addition to Taylor, the research team included Barry-John Theobald of East Anglia's School of Computing Sciences, and Disney's Iain Matthews.
For more information and a video, visit the project web site, http://www.disneyresearch.com/publication/a-mouth-full-of-words-visually-consistent-acoustic-redubbing.
About Disney Research
Disney Research is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Vice Presidents Jessica Hodgins and Markus Gross manage Disney Research facilities in Los Angeles, Pittsburgh, Zürich, and Boston and work closely with the Pixar and ILM research groups in the San Francisco Bay Area. Research topics include computer graphics, animation, video processing, computer vision, robotics, wireless & mobile computing, human-computer interaction, displays, behavioral economics, and machine learning.
Leading doctors today [Monday 20 April, 2015] warn that medical and public recognition of sepsis--thought to contribute to between a third and a half of all hospital deaths--must improve if the number of deaths from this common and potentially life-threatening condition are to fall.
In a new Commission, published in The Lancet Infectious Diseases, Professor Jonathan Cohen and colleagues outline the current state of research into this little-understood condition, and highlight priority areas for future investigation.
Sepsis--sometimes misleadingly called "blood poisoning"--is ...
The college graduation rate for students who have lived in foster care is 3 percent, among the lowest of any demographic group in the country. And this rate is unlikely to improve unless community colleges institute formal programs to assist foster youth both financially and academically, concludes a new study by researchers at University of the Pacific.
The findings will be presented during the annual meeting of the American Educational Research Association in Chicago on Sunday, April 19.
"Informal programs are less likely to work since foster youth lack guidance ...
PHILADELPHIA-- Once again, researchers at Penn's Abramson Cancer Center have extended the reach of the immune system in the fight against metastatic melanoma, this time by combining the checkpoint inhibitor tremelimumab with an anti-CD40 monoclonal antibody drug. The first-of-its-kind study found the dual treatments to be safe and elicit a clinical response in patients, according to new results from a phase I trial to be presented at the AACR Annual Meeting 2015 on Sunday, April 19.
Researchers include first author David L. Bajor, MD, instructor of Medicine in the division ...
PHILADELPHIA -- Women diagnosed with breast cancer are often told not to eat soyfoods or soy-based supplements because they can interfere with anti-estrogen treatment. But new research being presented at the American Association for Cancer Research (AACR) Annual Meeting 2015 could eventually impact that advice, because in animals, a long history of eating soyfoods boosts the immune response against breast tumors, reducing cancer recurrence.
The study, conducted at Georgetown Lombardi Comprehensive Cancer Center, could offer good news to some women whose diet has long ...
PHILADELPHIA, April 19, 2015 - Broccoli sprout extract protects against oral cancer in mice and proved tolerable in a small group of healthy human volunteers, the University of Pittsburgh Cancer Institute (UPCI), partner with UPMC CancerCenter, announced today at the American Association for Cancer Research (AACR) Annual Meeting in Philadelphia.
The promising results will be further explored in a human clinical trial, which will recruit participants at high risk for head and neck cancer recurrence later this year. This research is funded through Pitt's Specialized Program ...
PHILADEPHIA, April 19, 2015 - Identifying molecular changes that occur in tissue after chemotherapy could be crucial in advancing treatments for ovarian cancer, according to research from Magee-Womens Research Institute and Foundation (MWRIF) and the University of Pittsburgh Cancer Institute (UPCI), partner with UPMC CancerCenter, presented today at the American Association for Cancer Research (AACR) Annual Meeting 2015.
For years now, intraperitoneal chemotherapy, a treatment which involves filling the abdominal cavity with chemotherapy drugs after surgery, has been ...
PHILADELPHIA -Genetically modified versions of patients' own immune cells successfully traveled to tumors they were designed to attack in an early-stage trial for mesothelioma and pancreatic and ovarian cancers at the Perelman School of Medicine at the University of Pennsylvania. The data adds to a growing body of research showing the promise of CAR T cell technology. The interim results will be presented at the American Association for Cancer Research (AACR) Annual Meeting 2015, April 18-22.
"The goal of this phase I trial was to study the safety and feasibility of ...
Using mobile apps in preschool classrooms may help improve early literacy skills and boost school readiness for low-income children, according to research by NYU's Steinhardt School of Culture, Education, and Human Development.
"Guided use of an educational app may be a source of motivation and engagement for children in their early years," said Susan B. Neuman, professor of childhood and literacy education at NYU Steinhardt and the study's author. "The purpose of our study was to examine if a motivating app could accelerate children's learning, which it did."
Neuman ...
Schools must track the academic progress of homeless students with as much care as they track special education, Title I and English language learner students, according to researchers at University of the Pacific.
"In an age of accountability, schools focus their efforts and attention on the students they are mandated to report on," said Ronald Hallett, associate professor of education and lead author of the study. "We need to realign our policies and procedures if we are going to improve academic outcomes for homeless and highly mobile students."
Hallett and his colleagues ...
Meaningful college experiences, including internships and studying abroad, may not matter as much as your major and what school you attend when it comes to job satisfaction and earnings, according to research by NYU's Steinhardt School of Culture, Education, and Human Development.
"Our study adds important nuance to our understanding of the influence specific college experiences have on economic and attitudinal job outcomes in the years following college graduation," said Gregory Wolniak, director of the Center for Research on Higher Education Outcomes at NYU Steinhardt ...
