Disney Research demonstrates markerless motion capture

(Press-News.org) PITTSBURGH -- Conventional motion capture for film and game production involves multiple cameras and actors festooned with markers. A new technique developed by Disney Research, Pittsburgh, has demonstrated how three-dimensional motion capture can be accomplished with a single camera and without aid of markers.

The technique, developed in collaboration with Brown University, not only captures the 3D poses of actors, as is done with traditional motion capture systems, but derives "biped controllers" — programs that incorporate the underlying physics of the motion. Bipedal controllers generate the poses by computing the forces acting on the body and integrating them over time.

Leonid Sigal, research scientist at Disney Research, Pittsburgh, said this understanding of the process that generates those poses will enable computer graphic artists to not only reproduce those motions in animations, but to change the environments in which the action takes place. The motion captured from an actor walking across a room, for instance, could be applied to an animated character walking down a slope, crossing a slippery surface or carrying a heavy object.

Though Sigal emphasized that the technique requires more refinement, recovering 3D motion capture from a single camera opens up a number of new possibilities for motion capture, including the ability to capture motions or expressions from old movies or existing video. It could aid in capturing motion from outdoor scenes or types of motion that might be difficult to achieve within a traditional motion capture studio. It would be particularly suitable for use in developing games and other applications that involve interaction between an animated character and a player or a changing environment, he added.

Outside of the realm of filmmaking and game creation, coaches might use video of athletes recorded during practice or during competition to analyze body mechanics.

In addition to Sigal, the research team included Jessica Hodgins, director of Disney Research, Pittsburgh; Marek Vondrak, a Disney Research intern and a PhD student in computer science at Brown, and Odest Jenkins, associate professor of computer science at Brown. The researchers will present their findings Aug. 6 at SIGGRAPH 2012, the International Conference on Computer Graphics and Interactive Techniques, at the Los Angeles Convention Center.

For more information and videos, please visit the project web site at http://disneyresearch.cms.newbold.net/research/projects/cv_videoBased3DMC_drp.htm

Solving the problem of markerless motion capture and bipedal controller estimation simultaneously is critical, said Vondrak, whose PhD thesis focuses on the role of physics-based models in video-based motion capture. When recovering poses, the estimated biped controllers help to determine which of the possible sequences of 3D poses would reasonably fit the video images. On the other hand, recovered controllers allow direct application and natural adaptation of the motions to characters in new and dynamically changing environments.

"We didn't want to break these problems up, but to do them simultaneously," Sigal said. "Given the current technology, I don't think you can do them separately."

The technique is computationally intensive and does not yet match the quality possible with traditional motion capture, Sigal said. But given the difficulty of markerless motion capture with a single camera, the preliminary results are very promising, he added. Moreover, many real-world applications of markerless motion capture likely will employ multiple cameras, which reduces the technical challenge. The Disney researchers also attempted to make the process totally automated; adding a human to the loop for tasks such as identifying joint locations could also improve results.### About Disney Research Disney Research (www.disneyresearch.com) is a network of research laboratories supporting The Walt Disney Company. Its purpose is to pursue scientific and technological innovation to advance the company's broad media and entertainment efforts. Disney Research is managed by an internal Disney Research Council co-chaired by Disney-Pixar's Ed Catmull and Walt Disney Imagineering's Bruce Vaughn, and including the directors of the individual labs. It has facilities in Los Angeles, San Francisco, Pittsburgh, Boston and Zurich. Research topics include computer graphics, video processing, computer vision, robotics, radio and antennas, wireless communications, human-computer interaction, displays, data mining, machine learning and behavioral sciences.