(Press-News.org) ITHACA, N.Y. – Cornell University researchers have developed two technologies that track a person’s gaze and facial expressions through sonar-like sensing. The technology is small enough to fit on commercial smartglasses or virtual reality or augmented reality headsets, yet consumes significantly less power than similar tools using cameras.
Both use speakers and microphones mounted on an eyeglass frame to bounce inaudible soundwaves off the face and pick up reflected signals caused by face and eye movements. One device, GazeTrak, is the first eye-tracking system that relies on acoustic signals. The second, EyeEcho, is the first eyeglass-based system to continuously and accurately detect facial expressions and recreate them through an avatar in real time.
The devices can last for several hours on a smartglass battery and more than a day on a VR headset.
“It’s small, it’s cheap and super low-powered, so you can wear it on smartglasses everyday – it won’t kill your battery,” said Cheng Zhang, assistant professor of information science. Zhang directs the Smart Computer Interfaces for Future Interactions (SciFi) Lab that created the new devices.
“In a VR environment, you want to recreate detailed facial expressions and gaze movements so that you can have better interactions with other users,” said Ke Li, a doctoral student who led the GazeTrak and EyeEcho development.
For GazeTrak, researchers positioned one speaker and four microphones around the inside of each eye frame of a pair of glasses, to bounce and pick up soundwaves from the eyeball and the area around the eyes. The resulting sound signals are fed into a customized deep learning pipeline that uses artificial intelligence to continuously infer the direction of the person’s gaze.
For EyeEcho, one speaker and one microphone is located next to the glasses’ hinges, pointing down to catch skin movement as facial expressions change. The reflected signals are also interpreted using AI.
With this technology, users can have hands-free video calls through an avatar, even in a noisy café or on the street. While some smartglasses have the ability to recognize faces or distinguish between a few specific expressions, currently, none track expressions continuously like EyeEcho.
These two advances have applications beyond enhancing a person’s VR experience. GazeTrak could be used with screen readers to read out portions of text for people with low vision as they peruse a website.
GazeTrak and EyeEcho could also potentially help diagnose or monitor neurodegenerative diseases, like Alzheimer’s and Parkinsons. With these conditions, patients often have abnormal eye movements and less expressive faces, and this type of technology could track the progression of the disease from the comfort of a patient’s home.
Li will present GazeTrak at the Annual International Conference on Mobile Computing and Networking in the fall and EyeEcho at the Association of Computing Machinery CHI conference on Human Factors in Computing Systems in May.
For additional information, see this Cornell Chronicle story.
Media note: Pictures can be viewed and downloaded here: https://cornell.box.com/v/sonarsmartglasses.
-30-
END
AI-powered ‘sonar’ on smartglasses tracks gaze, facial expressions
2024-04-10
ELSE PRESS RELEASES FROM THIS DATE:
Newly found genetic variant defends against Alzheimer’s disease
2024-04-10
NEW YORK, NY (April 10, 2024)--Columbia researchers have discovered a genetic variant that reduces the odds of developing Alzheimer’s disease by up to 70% and may be protecting thousands of people in the United States from the disease.
The discovery of the protective variant, which appears to allow toxic forms of amyloid out of the brain and through the blood-brain barrier, supports emerging evidence that the brain’s blood vessels play a large role in Alzheimer's disease and could herald a new direction in therapeutic development.
“Alzheimer’s disease may get started with amyloid deposits in the brain, but ...
Serious flu damage prevented by compound that blocks unnecessary cell death
2024-04-10
As lung cells are killed by the influenza virus, they burst open, releasing molecular signals that trigger the immune cells that can combat the infection. This strategy can be an important red flag that something is wrong; however, if one cell death response, called necroptosis, continues unchecked, it can cause life-threatening injury to lung tissue. In a study published April 10 in the journal Nature, Tufts University School of Medicine scientists and collaborators present a newly developed compound ...
CHOP, Stanford researchers identify protein that controls CAR T cell longevity
2024-04-10
Philadelphia, April 10, 2024 – CAR T cell therapy has revolutionized the way certain types of cancer are treated, and the longer those CAR T cells live in a patient’s body, the more effectively they respond to cancer. Now, in a new study, researchers at Children’s Hospital of Philadelphia (CHOP) and Stanford Medicine have found that a protein called FOXO1 improves the survival and function of CAR T cells, which may lead to more effective CAR T cell therapies and could potentially expand its use in difficult-to-treat cancers. The findings were published online today by the journal Nature.
T ...
Race, ethnicity, and delayed time to COVID-19 testing among health care workers
2024-04-10
About The Study: In this cross-sectional study of health care personnel (HCP), compared with non-Hispanic white HCP and clinical HCP with graduate degrees, non-Hispanic Black HCP, non-Hispanic HCP of other races, and HCP of all other professional and education backgrounds were more likely to have delayed COVID-19 testing. These findings suggest that time to testing may serve as a valuable metric in evaluating sociodemographic disparities in the response to COVID-19 and future health mitigation strategies.
Authors: DaMarcus ...
Trends in deaths of despair by race and ethnicity
2024-04-10
About The Study: As of 2022, the midlife mortality rates from deaths of despair (deaths from suicide, drug overdose, and alcoholic liver disease) among Black individuals were higher than rates among white individuals, and rates among American Indian or Alaska Native individuals remained higher than rates in the other groups. Rising inequalities in deaths of despair among American Indian or Alaska Native and Black individuals were largely attributable to disproportionate early mortality from drug- and alcohol-related causes, which increased leading up to and during the COVID-19 pandemic.
Authors: Joseph ...
Revolutionary molecular device unleashes potential for targeted drug delivery and self-healing materials
2024-04-10
In a new breakthrough that could revolutionise medical and material engineering, scientists have developed a first-of-its-kind molecular device that controls the release of multiple small molecules using force.
The researchers from The University of Manchester describe a force-controlled release system that harnesses natural forces to trigger targeted release of molecules, which could significantly advance medical treatment and smart materials.
The discovery, published today in the journal Nature, uses a novel technique using a type of interlocked molecule known as rotaxane. Under the influence of mechanical force - such ...
The genesis of our cellular skeleton, image by image
2024-04-10
Cells contain various specialised structures - such as the nucleus, mitochondria or peroxisomes - known as “organelles’’. Tracing their genesis and determining their structure is fundamental to understanding cell function and the pathologies linked to their dysfunction. Scientists at the University of Geneva (UNIGE) have combined high resolution microscopy and kinematic reconstruction techniques to visualise, in motion, the genesis of the human centriole. This organelle, essential to the organisation of the cell skeleton, is associated - in case of dysfunction - with certain cancers, brain disorders or retinal diseases. This work, published in the journal Cell, elucidates ...
Quantum breakthrough when light makes materials magnetic
2024-04-10
The potential of quantum technology is huge but is today largely limited to the extremely cold environments of laboratories. Now, researchers at Stockholm University, at the Nordic Institute for Theoretical Physics and at the Ca’ Foscari University of Venice have succeeded in demonstrating for the very first time how laser light can induce quantum behavior at room temperature – and make non-magnetic materials magnetic. The breakthrough is expected to pave the way for faster and more energy-efficient computers, information transfer and data storage.
Within a few decades, the advancement of quantum technology ...
Living near green space associated with fewer emotional problems in preschool-age kids, NIH study finds
2024-04-10
Children who live in areas with natural spaces (e.g., forests, parks, backyards) from birth may experience fewer emotional issues between the ages of 2 and 5, according to a study funded by the NIH Environmental Influences on Child Health Outcomes (ECHO) program.
While research has suggested that time in nature is important for mental health, studies examining the effects on young children are limited. ECHO investigators addressed this research gap by analyzing information from parents about the behavior of their children from ages 2 to 11. They combined this data with the family’s ...
Researchers explore role of androgens in shaping sex differences
2024-04-10
Sex differences are widespread across human development, physiological processes, and diseases, making it important to characterize the impact of sex differences in these areas. Understanding the regulatory mechanisms associated with these differences, including the role of androgens, is also vital for clinical translation—especially for diseases more prevalent in one sex.
To answer these questions, a team led by Prof. GAO Dong and Prof. CHEN Luonan from the Center for Excellence in Molecular Cell Science, Shanghai Institute ...