PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

Technique enables AI on edge devices to keep learning over time

With the PockEngine training method, machine-learning models can efficiently and continuously learn from user data on edge devices like smartphones

Technique enables AI on edge devices to keep learning over time
2023-11-16
(Press-News.org)

Personalized deep-learning models can enable artificial intelligence chatbots that adapt to understand a user’s accent or smart keyboards that continuously update to better predict the next word based on someone’s typing history. This customization requires constant fine-tuning of a machine-learning model with new data. 

Because smartphones and other edge devices lack the memory and computational power necessary for this fine-tuning process, user data are typically uploaded to cloud servers where the model is updated. But data transmission uses a great deal of energy, and sending sensitive user data to a cloud server poses a security risk.   

Researchers from MIT, the MIT-IBM Watson AI Lab, and elsewhere developed a technique that enables deep-learning models to efficiently adapt to new sensor data directly on an edge device.

Their on-device training method, called PockEngine, determines which parts of a huge machine-learning model need to be updated to improve accuracy, and only stores and computes with those specific pieces. It performs the bulk of these computations while the model is being prepared, before runtime, which minimizes computational overhead and boosts the speed of the fine-tuning process.     

When compared to other methods, PockEngine significantly sped up on-device training, performing up to 15times faster on some hardware platforms. Moreover, PockEngine didn’t cause models to have any dip in accuracy. The researchers also found that their fine-tuning method enabled a popular AI chatbot to answer complex questions more accurately.

“On-device fine-tuning can enable better privacy, lower costs, customization ability, and also lifelong learning, but it is not easy. Everything has to happen with a limited number of resources. We want to be able to run not only inference but also training on an edge device. With PockEngine, now we can,” says Song Han, an associate professor in the Department of Electrical Engineering and Computer Science (EECS), a member of the MIT-IBM Watson AI Lab, a distinguished scientist at NVIDIA, and senior author of a paper describing PockEngine.

Han is joined on the paper by lead author Ligeng Zhu, an EECS graduate student, as well as others at MIT, the MIT-IBM Watson AI Lab, and the University of California San Diego. The paper was recently presented at the IEEE/ACM International Symposium on Microarchitecture.

Layer by layer

Deep-learning models are based on neural networks, which comprise many interconnected layers of nodes, or “neurons,” that process data to make a prediction. When the model is run, a process called inference, a data input (such as an image) is passed from layer to layer until the prediction (perhaps the image label) is output at the end. During inference, each layer no longer needs to be stored after it processes the input. 

But during training and fine-tuning, the model undergoes a process known as backpropagation. In backpropagation, the output is compared to the correct answer, and then the model is run in reverse. Each layer is updated as the model’s output gets closer to the correct answer. 

Because each layer may need to be updated, the entire model and intermediate results must be stored, making fine-tuning more memory demanding than inference

However, not all layers in the neural network are important for improving accuracy. And even for layers that are important, the entire layer may not need to be updated. Those layers, and pieces of layers, don’t need to be stored. Furthermore, one may not need to go all the way back to the first layer to improve accuracy — the process could be stopped somewhere in the middle.

PockEngine takes advantage of these factors to speed up the fine-tuning process and cut down on the amount of computation and memory required.

The system first fine-tunes each layer, one at a time, on a certain task and measures the accuracy improvement after each individual layer. In this way, PockEngine identifies the contribution of each layer, as well as trade-offs between accuracy and fine-tuning cost, and automatically determines the percentage of each layer that needs to be fine-tuned.

“This method matches the accuracy very well compared to full back propagation on different tasks and different neural networks,” Han adds.

A pared-down model

Conventionally, the backpropagation graph is generated during runtime, which involves a great deal of computation. Instead, PockEngine does this during compile time, while the model is being prepared for deployment. 

PockEngine deletes bits of code to remove unnecessary layers or pieces of layers, creating a pared-down graph of the model to be used during runtime. It then performs other optimizations on this graph to further improve efficiency.

Since all this only needs to be done once, it saves on computational overhead for runtime.

“It is like before setting out on a hiking trip. At home, you would do careful planning — which trails are you going to go on, which trails are you going to ignore. So then at execution time, when you are actually hiking, you already have a very careful plan to follow,” Han explains.

When they applied PockEngine to deep-learning models on different edge devices, including Apple M1 Chips and the digital signal processors common in many smartphones and Raspberry Pi computers, it performed on-device training up to 15 times faster, without any drop in accuracy. PockEngine also significantly slashed the amount of memory required for fine-tuning.

The team also applied the technique to the large language model Llama-V2. With large language models, the fine-tuning process involves providing many examples, and it’s crucial for the model to learn how to interact with users, Han says. The process is also important for models tasked with solving complex problems or reasoning about solutions.

For instance, Llama-V2 models that were fine-tuned using PockEngine answered the question “What was Michael Jackson’s last album?” correctly, while models that weren’t fine-tuned failed. PockEngine cut the time it took for each iteration of the fine-tuning process from about seven seconds to less than one second on a NVIDIA Jetson Orin, an edge GPU platform.

In the future, the researchers want to use PockEngine to fine-tune even larger models designed to process text and images together. 

This work was supported, in part, by the MIT-IBM Watson AI Lab, the MIT AI Hardware Program, the MIT-Amazon Science Hub, the National Science Foundation (NSF), and the Qualcomm Innovation Fellowship.

###

Written by Adam Zewe, MIT News

Paper: “PockEngine: Sparse and Efficient Fine-tuning in a Pocket”

https://arxiv.org/pdf/2310.17752.pdf

END


[Attachments] See images for this press release:
Technique enables AI on edge devices to keep learning over time Technique enables AI on edge devices to keep learning over time 2

ELSE PRESS RELEASES FROM THIS DATE:

Department of Chemical Engineering receives $3.5 million award to study impact of adolescent exposure to opioids

Department of Chemical Engineering receives $3.5 million award to study impact of adolescent exposure to opioids
2023-11-16
Opioid addiction is a pressing public health crisis with far-reaching implications. More than 100,000 deaths a year have been linked to drug overdoses since 2020. The Centers for Disease Control and Prevention reports that more people died from drug overdoses in 2021 than from firearm and motor vehicle deaths combined. Three-quarters of these overdose deaths were attributable to opioids. A five-year, $3.5 million grant from the National Institutes of Health’s National Institute on Drug Abuse will fund the Virginia Tech Department of Chemical Engineering’s pioneering research to understand how adolescent ...

Terrorism rather than pandemics more concerning for those with those with authoritarian views, analysis shows

2023-11-16
Those with authoritarian political views are more likely to be concerned about terrorism and border control than a future new health pandemic, new research shows. During the pandemic, rather than a desire for a stronger government with the ability to impose measures to address the pandemic and its consequences, people with authoritarian views rejected this and embraced individual autonomy. Researchers analysed public perceptions of security threats in 2012 and in 2020. They believe COVID-19 belongs to a distinct category of threats of which those with authoritarian views are less ...

University of Miami receives $1.8 million NOAA grant to study South Florida’s coastal ecosystems

University of Miami receives $1.8 million NOAA grant to study South Florida’s coastal ecosystems
2023-11-16
The University of Miami Rosenstiel School of Marine, Atmospheric, and Earth Science has been awarded a nearly $1.8 million grant from the National Oceanic and Atmospheric Administration (NOAA) as part of an anticipated four-year, $4.2 million project to support research on the impacts to South Florida’s coastal ecosystems from a multitude of climate change stressors.  The newly funded project, co-led by the University of Miami Rosenstiel School and NOAA’s Atlantic Oceanographic and Meteorological Laboratory (AOML) will focus on climate impacts to South Florida’s coastal and marine ecosystems, including the Florida Keys National Marine Sanctuary ...

USF researchers help reduce lead levels in Madagascar drinking water

USF researchers help reduce lead levels in Madagascar drinking water
2023-11-16
TAMPA, Fla. (Nov. 16, 2023) -- A team of engineers and public health experts from the University of South Florida is helping Toamasina, Madagascar, residents reduce their exposure to lead – a major global environmental pollutant that causes more than 1 million premature deaths each year. By combining efforts to replace water pumps and educate city technicians, USF researchers helped decrease the blood lead levels of 87 percent of the children tested during their study. “They were taking old car batteries and melting them down to make check ...

UofL law professor developing generative AI toolkit to aid legal writing instruction

UofL law professor developing generative AI toolkit to aid legal writing instruction
2023-11-16
LOUISVILLE, Ky. – While many are wary of artificial intelligence and its feared effect of supplanting the human creation of content, one University of Louisville professor is leading an effort to help her colleagues use it in the classroom. Susan Tanner, assistant professor of law at UofL’s Brandeis Law School, has won a teaching grant from the Association of Legal Writing Directors to develop a toolkit that law professors anywhere can use to incorporate generative artificial intelligence (genAI) into their legal writing curricula. GenAI is technology that can create text, images, videos and other media in response to prompts inputted by a user – otherwise known ...

Novel predictor of prediabetes in Latino youth identified in new USC study

2023-11-16
A team of researchers from the Keck School of Medicine of USC have identified two metabolites, substances produced by the body during metabolism, that may help predict which young Latino people are most likely to develop prediabetes, a precursor to developing type 2 diabetes. The study, funded by the National Institutes of Health and published in Diabetes Care, is the first large-scale study to look at metabolites as possible predictors of prediabetes or type 2 diabetes in young Latino people. The researchers found that when they added these two metabolites to current prediction models, they could more accurately ...

More than 1 in 10 pediatric ambulance runs are for mental health emergencies

2023-11-16
A new study offers a novel look at the scope of the youth mental health crisis across the United States – in 2019-2020, more than 1 in 10 kids who were brought to the hospital by ambulance had a behavioral health emergency. Out of these behavioral health emergencies, 85 percent were in 12-17-year-olds. Findings were published in the journal Academic Emergency Medicine. “Our study found that pediatric behavioral health emergencies requiring an ambulance were much too frequent,” said senior author Jennifer Hoffmann, MD, MS, emergency ...

Sunny Jardine appointed new Editor-in-Chief of Marine Resource Economics

2023-11-16
Marine Resource Economics (MRE) is proud to announce the appointment of Sunny Jardine as the journal’s new Editor in Chief, effective January 8, 2024. Jardine is an associate professor and the Rae S. and Bell M. Shimada Endowed Faculty Fellow in Memory of Warren S. Wooster in the School of Marine and Environmental Affairs at the University of Washington.  Professor Jardine has supported MRE as a long-time associate editor.  In that capacity, she handled papers across a range of marine and resource economics applications.  She brings expertise in commercial fisheries management, conservation planning, the ...

Mayo Clinic and Columbia University receive $10.6 million grant from NCI to advance glioblastoma research with mathematical oncology

2023-11-16
Mayo Clinic Comprehensive Cancer Center and Columbia University received a five-year, $10.6 million U54 center grant from the National Cancer Institute (NCI) to further study combining the molecular analysis of glioblastoma with MRI. Glioblastoma is a fast-growing and aggressive brain tumor that begins as a growth of cells in the brain or spinal cord. As it grows, it can invade and destroy healthy tissue. There is no cure, but treatments may slow the cancer's growth and reduce symptoms. Glioblastoma is a diverse cancer, which means ...

A new ultrasound patch can measure how full your bladder is

A new ultrasound patch can measure how full your bladder is
2023-11-16
CAMBRIDGE, MA -- MIT researchers have designed a wearable ultrasound monitor, in the form of a patch, that can image organs within the body without the need for an ultrasound operator or application of gel. In a new study, the researchers showed that their patch can accurately image the bladder and determine how full it is. This could help patients with bladder or kidney disorders more easily track whether these organs are functioning properly, the researchers say. This approach could also be adapted to monitor other organs within the body by changing the location of the ultrasound array and tuning the frequency ...

LAST 30 PRESS RELEASES:

Federal Trade Commission actions on prescription drugs, 2000-2022

Fluoride exposure during pregnancy linked to increased risk of childhood neurobehavioral problems, study finds

The Ukraine war caused migrating eagles to deviate from their usual flight plan

Endangered migrating eagles impacted by Ukraine war

Study explores association between fluoride exposure in pregnancy and neurobehavioral issues in young children

Using magnetic resonance spectroscopy to design safer, higher-performance lithium batteries

Should your exercise goals be in minutes or steps? Study suggests they are equally beneficial

Racial and ethnic inequities in cancer care continuity during the COVID-19 pandemic among those with SARS-CoV-2

Effect of sleep restriction on adolescent cognition by adiposity

Webb Telescope offers first glimpse of an exoplanet’s interior

Alkyl-aromatic hybrid micelles formed from emergent umbrella-shaped molecules

First study from the African Ancestry Neuroscience Research Initiative identifies key genes in the brain that account for higher rates of some brain disorders in Black Americans

NIH awards Coast-to-Coast Consortium $5.6 million for All of Us Research Program

Ben-Gurion University scientist hunts for drug candidate to treat brain tumors

New Health Blueprint maps healthier future for rural, underserved Southwest Virginia

Survival benefit associated with participation in clinical trials of anticancer drugs

Expanding on the fundamental principles of liquid movement

Chemical Insights Research Institute partners with Duke University and the East-West Center to examine dust and ash from devastating Hawai’ian wildfires

NCCN publishes new resource for patients with intestinal cancer type most have never heard of before diagnosis

Subduction zone splay faults compound hazards of great earthquakes

Record low Antarctic sea ice ‘extremely unlikely’ without climate change

After hundreds of years, study confirms Bermuda now home to cownose rays

Scientists uncover promising treatment target for resistant brain cancer

Revolutionizing cancer treatment by intracellular protein delivery using hybrid nanotubes

Chemist Julian West makes C&EN magazine’s ‘Talented 12’ list

Robot-phobia could exasperate hotel, restaurant labor shortage

Study offers new detail on how COVID-19 affects the lungs

Body’s ‘message in a bottle’ delivers targeted cancer treatment

1 in 4 parents say their teen consumes caffeine daily or nearly every day

What makes some brown algae shimmer and others not?

[Press-News.org] Technique enables AI on edge devices to keep learning over time
With the PockEngine training method, machine-learning models can efficiently and continuously learn from user data on edge devices like smartphones