(Press-News.org) In the high-stakes world of aviation, a pilot's ability to perform under stress can mean the difference between a safe flight and disaster. Comprehensive and precise training is crucial to equip pilots with the skills needed to handle these challenging situations.
Pilot trainers rely on augmented reality (AR) systems for teaching, by guiding pilots through various scenarios so they learn appropriate actions. But those systems work best when they are tailored to the mental states of the individual subject.
Enter HuBar, a novel visual analytics tool designed to summarize and compare task performance sessions in AR — such as AR-guided simulated flights — through the analysis of performer behavior and cognitive workload.
By providing deep insights into pilot behavior and mental states, HuBar enables researchers and trainers to identify patterns, pinpoint areas of difficulty, and optimize AR-assisted training programs for improved learning outcomes and real-world performance.
HuBar was developed by a research team from NYU Tandon School of Engineering that will present it at the 2024 IEEE Visualization and Visual Analytics Conference on October 17, 2024.
“While pilot training is one potential use case, HuBar isn't just for aviation,” explained Claudio Silva, NYU Tandon Institute Professor in the Computer Science and Engineering (CSE) Department, who led the research with collaboration from Northrop Grumman Corporation (NGC). “HuBar visualizes diverse data from AR-assisted tasks, and this comprehensive analysis leads to improved performance and learning outcomes across various complex scenarios.”
“HuBar could help improve training in surgery, military operations and industrial tasks,” said Silva, who is also the co-director of the Visualization and Data Analytics Research Center (VIDA) at NYU.
The team introduced HuBar in a paper that demonstrates its capabilities using aviation as a case study, analyzing data from multiple helicopter co-pilots in an AR flying simulation. The team also produced a video about the system.
Focusing on two pilot subjects, the system revealed striking differences: one subject maintained mostly optimal attention states with few errors, while the other experienced underload states and made frequent mistakes.
HuBar's detailed analysis, including video footage, showed the underperforming copilot often consulted a manual, indicating less task familiarity. Ultimately, HuBar can enable trainers to pinpoint specific areas where copilots struggle and understand why, providing insights to improve AR-assisted training programs.
What makes HuBar unique is its ability to analyze non-linear tasks where different step sequences can lead to success, while integrating and visualizing multiple streams of complex data simultaneously.
This includes brain activity (fNIRS), body movements (IMU), gaze tracking, task procedures, errors, and mental workload classifications. HuBar's comprehensive approach allows for a holistic analysis of performer behavior in AR-assisted tasks, enabling researchers and trainers to identify correlations between cognitive states, physical actions, and task performance across various task completion paths.
HuBar's interactive visualization system also facilitates comparison across different sessions and performers, making it possible to discern patterns and anomalies in complex, non-sequential procedures that might otherwise go unnoticed in traditional analysis methods.
"We can now see exactly when and why a person might become mentally overloaded or dangerously underloaded during a task," said Sonia Castelo, VIDA Research Engineer, Ph.D. student in VIDA, and the HuBar paper’s lead author. "This kind of detailed analysis has never been possible before across such a wide range of applications. It's like having X-ray vision into a person's mind and body during a task, delivering information to tailor AR assistance systems to meet the needs of an individual user.”
As AR systems – including headsets like Microsoft Hololens, Meta Quest and Apple Vision Pro – become more sophisticated and ubiquitous, tools like HuBar will be crucial for understanding how these technologies affect human performance and cognitive load.
"The next generation of AR training systems might adapt in real-time based on a user's mental state," said Joao Rulff, a Ph.D. student in VIDA who worked on the project. "HuBar is helping us understand exactly how that could work across diverse applications and complex task structures."
HuBar is part of the research Silva is pursuing under the Defense Advanced Research Projects Agency (DARPA) Perceptually-enabled Task Guidance (PTG) program. With the support of a $5 million DARPA contract, the NYU group aims to develop AI technologies to help people perform complex tasks while making these users more versatile by expanding their skillset — and more proficient by reducing their errors. The pilot data in this study came from NGC as part of the DARPA PTG
In addition to Silva, Castelo and Rulff, the paper’s authors are: Erin McGowan, PhD Researcher, VIDA; Guande Wu, Ph.D. student, VIDA; Iran R. Roman, Post-Doctoral Researcher, NYU Steinhardt; Roque López, Research Engineer, VIDA; Bea Steers, Research Engineer, NYU Steinhardt; Qi Sun, Assistant Professor of CSE, NYU; Juan Bello, Professor, NYU Tandon and NYU Steinhardt; Bradley Feest, Lead Data Scientist, Northrop Grumman Corporation; Michael Middleton, Applied AI Software Engineer and Researcher, Northrop Grumman Corporation, and PhD student, NYU Tandon; Ryan McKendrick, Applied Cognitive Scientist, Northrop Grumman Corporation.
END
New tool helps analyze pilot performance and mental workload in augmented reality
2024-10-17
ELSE PRESS RELEASES FROM THIS DATE:
Researchers advance ideas on abiotic organic synthesis
2024-10-17
Recently, a Chinese research team reported the discovery of abiotic organic compounds in the oceanic crust of the Southwest Indian Ridge and proposed a molecular mechanism for organic condensation.
This breakthrough followed the team’s discovery of nanoscale abiotic organic matter in mantle rocks from the Yap Trench in 2021. It represents a significant advance in research on deep-sea carbon cycling and the origin of life, shedding light on key pathways for abiotic organic synthesis in nature.
The scientists published their findings in the Proceedings ...
Visible light energy yields two-for-one deal when added to CO2 recycling process
2024-10-17
CHAMPAIGN, Ill. — By combining visible light and electrochemistry, researchers have enhanced the conversion of carbon dioxide into valuable products and stumbled upon a surprising discovery. The team found that visible light significantly improved an important chemical attribute called selectivity, opening new avenues not only for CO2 conversion but also for many other chemical reactions used in catalysis research and chemical manufacturing.
One way that chemists recycle CO2 into valuable products is through a process called electrochemical reduction, where a stream of CO2 gas moves through an electrolysis cell that breaks the CO2 and water ...
Search continues for a treatment that reduces long COVID symptoms
2024-10-17
The first randomized, placebo-controlled clinical trial evaluating the impact of therapeutic monoclonal antibodies on the risk of developing long COVID was led by researchers at Weill Cornell Medicine. The study, published Sept. 1 in eClinicalMedicine, details an international, multicenter phase 2/3 clinical trial that found a combination of amubarvimab and romlusevimab did not reduce the occurrence of long COVID symptoms.
Lasting weeks or even months after SARS-CoV-2 infection, long COVID is characterized by a variety of symptoms that affect each person differently. These include extreme fatigue, difficulty concentrating, ...
The Journal of Medical Internet Research theme issue call for papers: The emergence of medical futures studies
2024-10-17
(Toronto, October 17, 2024) JMIR Publications invites submissions to a new theme issue titled “The Emergence of Medical Futures Studies” in the Journal of Medical Internet Research, the leading peer-reviewed journal for digital medicine and health and health care in the Internet age, indexed in PubMed, PMC, MEDLINE, SCIE, Scopus and DOAJ.
The rapid progress of technology in health care, combined with significant cultural shifts toward digital health and the urgent need for preparedness, highlighted by events such as the COVID-19 pandemic and climate change, have brought about an increased emphasis ...
License to chill: Bond shows ‘regressive nostalgia’ can freeze a brand's future
2024-10-17
Super-spy James Bond is a prime example of ‘regressive nostalgia’ highlighting how certain consumer groups cling to idealised past versions of brands and resist attempts to move with the times, a new study reveals.
Researchers examined the James Bond movie franchise - a cultural icon for over 70 years - and discovered that some ‘super-consumers’ react negatively to modern portrayals of the fictional British secret agent that reflect contemporary societal values.
Whilst loyal to the brand, these consumers prefer traditional, more exclusionary, versions of Bond which most closely follow author Ian Fleming’s original 1950s and 1960s vision – ...
Researchers from Brazil and Italy search industrial waste for new Alzheimer’s drugs
2024-10-17
A self-proclaimed Brazil-Italy collaboration enthusiast, researcher Laura Bolognesi created the B2AlzD2 Joint Lab at the Department of Pharmacy and Biotechnology of the Università di Bologna (UNIBO), the first Brazil-Bologna joint laboratory dedicated to the development of new drugs for the treatment of Alzheimer’s disease. The partners include scientists from four Brazilian universities: the Federal University of Rio de Janeiro (UFRJ), the University of Brasília (UnB), the University of São Paulo (USP Ribeirão Preto) and the Federal University of Minas Gerais (UFMG).
One of the laboratory’s ongoing ...
BU, Boston Medical Center researchers join forces with GSK to fight lung diseases
2024-10-17
(Boston)—Researchers from the Center for Regenerative Medicine (CReM) at Boston University and Boston Medical Center (BMC) have announced a new collaboration with the global biopharma company GSK to advance innovative research focused on developing cutting-edge models to study and treat lung diseases like pulmonary fibrosis.
Pulmonary fibrosis, including its most common form, idiopathic pulmonary fibrosis (IPF), is a progressive and life-threatening condition that results in scarring of the lungs, making it increasingly difficult for patients to breathe. ...
Bacteria thrive by playing nice before going their own way
2024-10-17
Biofilms — slimy communities of bacteria — grow on all sorts of surfaces: from glaciers and hot springs to plant roots, your bathtub and fridge, wounds, and medical devices such as catheters. Most biofilms are composed of multiple bacterial species, but how these species manage to live together is unclear.
A new study by Dartmouth scientists in Current Biology uses experiments and modeling to delve into how three species of biofilm bacteria coexist — and when they move out on their own. One species, Pseudomonas aeruginosa, a verstaile pathogen known to be antibiotic resistant, dominated ...
Identifying the genes that viruses ‘steal’ from ocean microbes
2024-10-17
COLUMBUS, Ohio – The microbes that cycle nutrients in the ocean don’t do the work on their own – the viruses that infect them also influence the process. It’s a vital job for the rest of the planet, enabling oceans to absorb half of the human-generated carbon in the atmosphere and produce half of the oxygen we breathe.
A new study gets scientists closer to more fully understanding where viruses fit into the global ocean picture of cycling nutrients such as nitrogen, phosphorous and, of particular interest, carbon. The research broadly expands on a 20-year-old finding that genes can be exchanged between viruses and the photosynthetic ...
CDC/PEPFAR awards Georgetown $27.5 million to address HIV/AIDS in Haiti
2024-10-17
WASHINGTON (Oct. 17, 2024) -- The U.S. Centers for Disease Control and Prevention (CDC) and the President’s Emergency Plan for AIDS Relief (PEPFAR), have awarded $27.5 million to the Center for Global Health Practice and Impact (CGHPI) at Georgetown University Medical Center to expand its ongoing work in Haiti to address HIV/AIDS.
For the 150,000 people in Haiti living with HIV, losing access to basic life-saving therapy can lead to unnecessary suffering, risk of transmission to others, and ...