(Press-News.org) New York, NY--June 28, 2021--Predicting what someone is about to do next based on their body language comes naturally to humans but not so for computers. When we meet another person, they might greet us with a hello, handshake, or even a fist bump. We may not know which gesture will be used, but we can read the situation and respond appropriately.
In a new study, Columbia Engineering researchers unveil a computer vision technique for giving machines a more intuitive sense for what will happen next by leveraging higher-level associations between people, animals, and objects.
"Our algorithm is a step toward machines being able to make better predictions about human behavior, and thus better coordinate their actions with ours," said Carl Vondrick, assistant professor of computer science at Columbia, who directed the study, which was presented at the International Conference on Computer Vision and Pattern Recognition on June 24, 2021. "Our results open a number of possibilities for human-robot collaboration, autonomous vehicles, and assistive technology."
It's the most accurate method to date for predicting video action events up to several minutes in the future, the researchers say. After analyzing thousands of hours of movies, sports games, and shows like "The Office," the system learns to predict hundreds of activities, from handshaking to fist bumping. When it can't predict the specific action, it finds the higher-level concept that links them, in this case, the word "greeting."
Past attempts in predictive machine learning, including those by the team, have focused on predicting just one action at a time. The algorithms decide whether to classify the action as a hug, high five, handshake, or even a non-action like "ignore." But when the uncertainty is high, most machine learning models are unable to find commonalities between the possible options.
Columbia Engineering PhD students Didac Suris and Ruoshi Liu decided to look at the longer-range prediction problem from a different angle. "Not everything in the future is predictable," said Suris, co-lead author of the paper. "When a person cannot foresee exactly what will happen, they play it safe and predict at a higher level of abstraction. Our algorithm is the first to learn this capability to reason abstractly about future events."
Suris and Liu had to revisit questions in mathematics that date back to the ancient Greeks. In high school, students learn the familiar and intuitive rules of geometry -- that straight lines go straight, that parallel lines never cross. Most machine learning systems also obey these rules. But other geometries, however, have bizarre, counter-intuitive properties; straight lines bend and triangles bulge. Suris and Liu used these unusual geometries to build AI models that organize high-level concepts and predict human behavior in the future.
"Prediction is the basis of human intelligence," said Aude Oliva, senior research scientist at the Massachusetts Institute of Technology and co-director of the MIT-IBM Watson AI Lab, an expert in AI and human cognition who was not involved in the study. "Machines make mistakes that humans never would because they lack our ability to reason abstractly. This work is a pivotal step towards bridging this technological gap."
The mathematical framework developed by the researchers enables machines to organize events by how predictable they are in the future. For example, we know that swimming and running are both forms of exercising. The new technique learns how to categorize these activities on its own. The system is aware of uncertainty, providing more specific actions when there is certainty, and more generic predictions when there is not.
The technique could move computers closer to being able to size up a situation and make a nuanced decision, instead of a pre-programmed action, the researchers say. It's a critical step in building trust between humans and computers, said Liu, co-lead author of the paper. "Trust comes from the feeling that the robot really understands people," he explained. "If machines can understand and anticipate our behaviors, computers will be able to seamlessly assist people in daily activity."
While the new algorithm makes more accurate predictions on benchmark tasks than previous methods, the next steps are to verify that it works outside the lab, says Vondrick. If the system can work in diverse settings, there are many possibilities to deploy machines and robots that might improve our safety, health, and security, the researchers say. The group plans to continue improving the algorithm's performance with larger datasets and computers, and other forms of geometry.
"Human behavior is often surprising," Vondrick commented. "Our algorithms enable machines to better anticipate what they are going to do next."
INFORMATION:
About the Study
The study is titled "Learning the Predictability of the Future."
Authors are: Didac Suris, Ruoshi Liu, Carl Vondrick
Department of Computer Science, Columbia Engineering
The study was supported in part by National Science Foundation (NSF) National Robotics Initiative (NRI) Award #1925157, the DARPA MCS program under Federal Agreement No. N660011924032, the DARPA KAIROS program under PTE Federal Award No. FA8750-19-2-1004, and an Amazon Research Award.
The authors declare no financial or other conflicts of interest.
Publication Details
The study was presented at the Conference on Computer Vision and Pattern Recognition on June 24, 2021.
LINKS:
Paper: https://arxiv.org/pdf/2101.01600.pdf
Video demo:
https://drive.google.com/file/d/1nZQ3m0UovhpB0OvEMn2x8_OO9xDJ0DTN/view?usp=sharing
http://engineering.columbia.edu/
http://hyperfuture.cs.columbia.edu/
https://arxiv.org/pdf/2101.01600.pdf
http://cvpr2021.thecvf.com/
https://apnews.com/article/7ec97c70223e49b98fbc7df2ff824d59
http://www.cs.columbia.edu/~vondrick/
http://www.didacsuris.com/
https://ruoshiliu.github.io/
https://www.cs.columbia.edu
Columbia Engineering
Columbia Engineering, based in New York City, is one of the top engineering schools in the U.S. and one of the oldest in the nation. Also known as The Fu Foundation School of Engineering and Applied Science, the School expands knowledge and advances technology through the pioneering research of its more than 220 faculty, while educating undergraduate and graduate students in a collaborative environment to become leaders informed by a firm foundation in engineering. The School's faculty are at the center of the University's cross-disciplinary research, contributing to the Data Science Institute, Earth Institute, Zuckerman Mind Brain Behavior Institute, Precision Medicine Initiative, and the Columbia Nano Initiative. Guided by its strategic vision, "Columbia Engineering for Humanity," the School aims to translate ideas into innovations that foster a sustainable, healthy, secure, connected, and creative humanity.
Sixteen of the 146 health-related All-Party Parliamentary Groups (APPGs) in the Houses of Parliament (UK) received over a £1 million in payments from 35 pharmaceutical companies between 2012-2018 according to a new study.
The researchers behind the analysis from the University of Bath's Centre for the Analysis of Social Policy suggest their findings reveal a worrying lack of transparency over payments received and potential conflicts of interests towards public policy.
Through their research they extracted details from 6,624 entries about funding ...
Experts estimate more than 6 million Americans are living with Alzheimer's dementia. But a recent study, led by the University of Cincinnati, sheds new light on the disease and a highly debated new drug therapy.
The UC-led study, conducted in collaboration with the Karolinska Institute in Sweden, claims that the treatment of Alzheimer's disease might lie in normalizing the levels of a specific brain protein called amyloid-beta peptide. This protein is needed in its original, soluble form to keep the brain healthy, but sometimes it hardens into "brain stones" or clumps, called amyloid plaques.
The study, which appears in the journal EClinicalMedicine (published by the Lancet), comes on the heels of the FDA's conditional approval of a new medicine, aducanumab, that ...
CORVALLIS, Ore. - Oregon State University researchers have some good news for the well-meaning masses who place bird feeders in their yards: The small songbirds who visit the feeders seem unlikely to develop an unhealthy reliance on them.
"There's still much we don't know about how intentional feeding might induce changes in wild bird populations, but our study suggests that putting out food for small birds in winter will not lead to an increased dependence on human-provided food," said Jim Rivers, an animal ecologist with the OSU College of Forestry.
Findings from the research, which looked at black-capped chickadees outfitted with radio frequency identification tags, ...
New York, NY (June 28, 2021) - Patients with a type of blood cancer called multiple myeloma had a widely variable response to COVID-19 vaccines--in some cases, no detectable response--pointing to the need for antibody testing and precautions for these patients after vaccination, according to a study that will be published in Cancer Cell this week.
Mount Sinai researchers found that multiple myeloma patients mount variable and sometimes suboptimal responses after receiving the Pfizer-BioNTech or Moderna COVID-19 vaccines. Almost 16 percent of these patients developed no detectible antibodies after both vaccine doses. These findings may be relevant to other cancer patients undergoing treatment and to immunocompromised patients.
"This study ...
Did you know chocolate comes from cocoa beans grown in some of the most biodiverse tropical landscapes on the planet? The cocoa tree (Theobroma cacao) is farmed within biodiversity hotspots of West Africa, South America, and Southeast Asia. Cocoa was traditionally grown under a canopy of native trees that provided habitat for birds and other wildlife. However, pressure to increase cocoa production has pushed many farmers to clear forest and eliminate the shade trees on their farms.
An estimated 2-3 million hectares of tropical forest were converted to cocoa from 1988-2008 with severe consequences for biodiversity. Unsustainable cocoa monocultures (agricultural ...
Multi-business firms have flexibility advantages over single-business rivals because they have the option to redeploy resources across businesses. This flexibility, it has been assumed without empirical evidence, is purported to inspire quicker exits from markets.
A 2017 survey revealed that 70 percent of corporate executives expected to make at least one divestment in the subsequent two years, with the primary motive being strategic realignment of portfolios as non-core assets are shed. A large number of academic studies have established that parent firms tend ...
Imagine having feelings of distress and negativity at some point as you are going about your day. Then imagine feeling that way every day, for almost 21 days. Lesbian, gay, bisexual, transgender and queer youth don't have to imagine having negative feelings at some point throughout the day for an extended period of days. A new study from American University reveals just how pervasive emotional distress is related to stigma around sexual orientation.
"Although we know that LGBTQ youth experience bullying, discrimination, and microaggressions during adolescence, we don't know how prevalent these experiences ...
The electrochemical nitrogen reduction reaction (e-NRR) under ambient conditions is an emerging strategy to tackle the hydrogen- and energy-intensive processes entailed in industrial ammonia (NH3) synthesis via the traditional Haber-Bosch process. However, the e-NRR performance is currently impeded by the inherent inertness of N2 molecules, extremely slow kinetics, and overwhelming competition from the hydrogen evolution reaction (HER), all of which result in an unsatisfactory yield and ammonia selectivity (Faradaic efficiency, FE). To achieve a high-selectivity and high-performance NRR under ambient conditions, the rational design of efficient electrocatalysts is urgently required. Defect and interface engineering ...
A team of computer scientists at the USC Viterbi School of Engineering set out to develop new tools automate and organize social science data. What did they use as their data sets? Twitter posts from coastal capitals, New York City and Los Angeles.
The researchers found that they could identify similar tweets that do not have hashtags by using natural language processing and neural networks to create clusters of alike tweets. "It's using AI to create a map of similar tweets," says Mayank Kejriwal, a Research Assistant Professor in the USC Department of Industrial and Systems Engineering, and a Research Lead at the USC Information ...
Scientists at The University of Toledo discovered new movement in sperm that provides innovative avenues for diagnostics and therapeutic strategies for male infertility.
The research published in Nature Communications finds that the atypical centriole in the sperm neck acts as a transmission system that controls twitching in the head of the sperm, mechanically synchronizing the sperm tail movement to the new head movement.
The centriole has historically been considered a rigid structure that acts like a shock absorber.
"We think the atypical centriole in the sperm's neck is an evolutionary innovation whose function is to make your sperm move better," said Dr. Tomer Avidor-Reiss, professor of biological sciences in the UToledo College of Natural Sciences and Mathematics. "Reproductive ...
