PRESS-NEWS.org - Press Release Distribution
PRESS RELEASES DISTRIBUTION

Tokyo Tech, Tohoku University, Fujitsu, and RIKEN start collaboration to develop distributed training of Large Language Models

Tokyo Tech, Tohoku University, Fujitsu, and RIKEN start collaboration to develop distributed training of Large Language Models
2023-05-22
(Press-News.org) Tokyo Institute of Technology (Tokyo Tech), Tohoku University, Fujitsu Limited, and RIKEN today announced that they will embark on the research and development of a distributed training of Large Language Models (LLM) [1]on supercomputer Fugaku in May 2023, within the scope of the initiatives for use of Fugaku defined by Japanese policy.

LLMs are AI models for deep learning that serve as the core of generative AI including ChatGPT[2]. The four organizations aim to improve the environment for creating LLMs that can be widely used by academia and companies, contribute to improving the research capabilities of AI in Japan, and increase the value of utilizing Fugaku in both academic and industrial fields by disclosing the results of this R&D in the future.

Background While many anticipate that LLMs and generative AI will play a fundamental role in as the research and development of technologies for security, the economy, and society overall, the advancement and refinement of these models will require high-performance computing resources that can efficiently process large amounts of data.

Tokyo Tech, Tohoku University, Fujitsu, and RIKEN are undertaking an initiative to this end that will focus on research and development toward distributed training of LLMs.

Implementation period From May 24, 2023 to March 31, 2024 *Period of the initiative for use Fugaku for Japanese policies

Roles of each organization and company The technology used in this initiative will allow the organizations to efficiently perform large-scale language model training on the large-scale parallel computing environment of the supercomputer Fugaku. The roles of each organization and company are as follows:

Tokyo Institute of Technology: Oversight of overall processes, parallelization and acceleration of LLMs Tohoku University: Collection of learning data, selection of models Fujitsu: Acceleration of LLMs RIKEN: Distributed parallelization and accelerating communication of LLMs, acceleration of LLMs Future plans To support Japanese researchers and engineers to develop LLMs in the future, the four organizations plan to publish the research results obtained through the scope of the initiatives for use of Fugaku defined by Japanese policy on GitHub[3] and Hugging Face[4] in fiscal 2024. It is also anticipated that many researchers and engineers will participate in the improvement of the basic model and new applied research to create efficient methods that lead to the next generation of innovative research and business results.

The four organizations will additionally consider collaborations with Nagoya University, which develops data generation and learning methods for multimodal applications in industrial fields such as manufacturing, and CyberAgent, Inc., which provides data and technology for building LLMs.

Comments Comment from Toshio Endo, Professor, Global Scientific Information and Computing Center, Tokyo Institute of Technology: "The collaboration will integrate parallelization and acceleration of large-scale language models using the supercomputer "Fugaku" by Tokyo Tech and RIKEN, Fujitsu's development of high-performance computing infrastructure software for Fugaku and performance tuning of AI models, and Tohoku University's natural language processing technology. In collaboration with Fujitsu, we will also utilize the small research lab we established under the name of "Fujitsu Collaborative Research Center for Next Generation Computing Infrastructure" in 202X. We look forward to working together with our colleagues to contribute to the improvement of Japan's AI research capabilities, taking advantage of the large-scale distributed deep learning capabilities offered by "Fugaku"."

Comment from Kentaro Inui, Professor, Graduate School of Information Sciences, Tohoku University: "We aim to build a large-scale language model that is open-source, available for commercial use, and primarily based on Japanese data, with transparency in its training data. By enabling traceability of the learning data, we anticipate that this will facilitate research robust enough to scientifically verify issues related to the black box problem, bias, misinformation, and so-called "hallucination" phenomena common to AI. Leveraging the insights gained from deep learning from Japanese natural language processing developed at Tohoku University, we will construct large-scale models. We look forward to contributing to the enhancement of AI research capabilities in our country and beyond, sharing the results of the research we obtain through the initiative for researchers and developers."

Comment from Seishi Okamoto, EVP, Head of Fujitsu Research, Fujitsu Limited: "We are excited for the chance to leverage the powerful, parallel computing resources of the supercomputer Fugaku to supercharge research into AI and advance research and development of LLMS. Going forward, we aim to incorporate the fruits of this research into Fujitsu's new AI Platform, codenamed "Kozuchi," to deliver paradigm-shifting applications that contribute to the realization of a sustainable society."

Comment from Satoshi Matsuoka, Director, RIKEN Center for Computational Science: "The A64FX[5] CPU is equipped with an AI acceleration function known as SVE.
Software development and optimization are essential to maximize its capabilities and to utilize it for AI applications, however. We feel that this joint research will play an important role in bringing together experts of LLMs and computer science in Japan, including RIKEN R-CCS researchers and engineers, to advance techniques for building LLMs on the supercomputer "Fugaku". Together with our collaborators, we contribute to the realization of Society 5.0."

Project name Distributed Training of Large Language Models on Fugaku (Project Number: hp230254)

[Terms]

[1] Large-scale language models : Neural networks with hundreds of millions to billions of parameters that have been pre-learned using large amounts of data. Recently, GPT in language processing and ViT in image processing are known as representative large-scale learning models.

[2] ChatGPT : A large-scale language model for natural language processing developed by OpenAI that supports tasks such as interactive systems and automatic sentence generation with high accuracy.

[3] GitHub : A platform used to publish open-source software around the world. GitHub

[4] Hugging Face : A platform used to publish AI datasets around the world. Hugging Face

[5] A64FX : An ARM-based CPU developed by Fujitsu installed in supercomputer Fugaku.

 

###

About Tokyo Institute of Technology

Tokyo Tech stands at the forefront of research and higher education as the leading university for science and technology in Japan. Tokyo Tech researchers excel in fields ranging from materials science to biology, computer science, and physics. Founded in 1881, Tokyo Tech hosts over 10,000 undergraduate and graduate students per year, who develop into scientific leaders and some of the most sought-after engineers in industry. Embodying the Japanese philosophy of “monotsukuri,” meaning “technical ingenuity and innovation,” the Tokyo Tech community strives to contribute to society through high-impact research. https://www.titech.ac.jp/english/

 

END

[Attachments] See images for this press release:
Tokyo Tech, Tohoku University, Fujitsu, and RIKEN start collaboration to develop distributed training of Large Language Models Tokyo Tech, Tohoku University, Fujitsu, and RIKEN start collaboration to develop distributed training of Large Language Models 2

ELSE PRESS RELEASES FROM THIS DATE:

Nitrate: healthy heart or cancer risk? Meet nutrition’s Jekyll and Hyde

2023-05-22
Despite our understanding of nutrition expanding remarkably in recent times, few aspects of our diet continue to confuse and divide the experts like nitrate.   For a long time nitrate has been viewed warily, with previous research showing it could potentially be linked to causing cancer.   However, subsequent research has revealed dietary nitrate also has various cardiovascular health benefits, which could help reduce the risk of related conditions such as heart disease, dementia and diabetes.   So, how can one dietary compound have such contrasting potential risks and benefits?   Edith Cowan University’s (ECU) Nutrition and Health ...

Can charismatic robots help teams be more creative?

2023-05-22
Increasingly, social robots are being used for support in educational contexts. But does the sound of a social robot affect how well they perform, especially when dealing with teams of humans? Teamwork is a key factor in human creativity, boosting collaboration and new ideas. Danish scientists set out to understand whether robots using a voice designed to sound charismatic would be more successful as team creativity facilitators. “We had a robot instruct teams of students in a creativity task. The robot ...

Stop eradication of small mammals to protect vital ecosystems, say scientists

Stop eradication of small mammals to protect vital ecosystems, say scientists
2023-05-22
A new article published in the Journal of Animal Ecology suggests that current measures to protect grasslands in the Qinghai-Tibetan Plateau are damaging the ecosystem and should be stopped. The existing policy, introduced in 2000, calls for the eradication of small burrowing mammals. These include the mountain-dwelling herbivores, the plateau pika, and another small rodent, the zokor. Both are keystone species and are known as ecosystem engineers due to their modification of and impact on the environment. The ...

Study linking mucus plugs and COPD mortality could help save lives

2023-05-21
A retrospective analysis of patient data from the COPDGene study suggests that targeting mucus plugs could help prevent deaths from chronic obstructive pulmonary disease—the fourth leading cause of death in the United States Many patients with chronic obstructive pulmonary disease (COPD) experience airway-clogging mucus plugs, an accumulation of mucus in the lungs that can affect quality of life and lung functioning. A new study led by researchers from Brigham and Women’s Hospital, a founding ...

Sacubitril/valsartan shows benefit in heart failure with ejection fraction above 40%

2023-05-21
Prague, Czechia – 21 May 2023:  Sacubitril/valsartan leads to greater reduction in plasma NT-proBNP levels compared to valsartan alone after stabilisation for worsening heart failure in patients with an ejection fraction (EF) above 40%, according to late breaking science presented today at Heart Failure 2023, a scientific congress of the European Society of Cardiology (ESC),1 and published in the Journal of the American College of Cardiology. Principal investigator Dr. Robert Mentz of Duke University Medical Center, Durham, US said: “These data add to the evidence supporting a potential treatment benefit of sacubitril/valsartan ...

New device gently moves esophagus, making heart ablations safer, study found

New device gently moves esophagus, making heart ablations safer, study found
2023-05-21
A new device invented with the help of an electrophysiologist at The Ohio State University Wexner Medical Center makes a heart procedure safer for patients suffering from atrial fibrillation (AFib), a common irregular heart rhythm. AFib affects millions of people worldwide and greatly increases their risk of stroke and heart failure. To treat AFib, doctors use cardiac ablation to help restore the heart’s rhythm. Heat or cold energy delivered through a catheter destroys the heart tissue causing rapid and irregular heartbeats. ...

Society for Cardiovascular Angiography & Interventions names George D. Dangas, MD, MSCAI, President for 2023-24

2023-05-21
PHOENIX (May 20, 2023) – George D. Dangas, MD, PhD, MSCAI, Professor of Medicine (Cardiology and Surgery), and Director of Cardiovascular Innovation at the Zena and Michael A. Weiner Cardiovascular Institute at the Icahn School of Medicine of Mount Sinai in New York City, and Chief of Cardiology at Mount Sinai Queens assumed the office of president of the Society for Cardiovascular Angiography & Interventions (SCAI) today during the closing ceremonies at the SCAI 2023 Scientific Sessions in Phoenix, AZ. Dr. Dangas is an authoritative voice in the performance of nonsurgical cardiovascular and valve interventions using both established ...

SCAI announces 2023-24 SCAI-WIN CHIP Fellowship Recipient

2023-05-21
PHOENIX (May 20, 2023) – Njambi Mathenge, MD, MPH, an interventional cardiology fellow at the Massachusetts General Hospital has been selected as the recipient of the SCAI-Women in Innovations (SCAI-WIN) CHIP Fellowship, the Society for Cardiovascular Angiography and Interventions announced today. The $115,000 fellowship opportunity was made possible by support from Abiomed (founding supporter), Boston Scientific, Medtronic, and Shockwave Medical, Inc., and is offered to interventional cardiology (IC) fellows or practicing interventional cardiologists interested in ...

SCAI announces new award recognizing the contributions of early career interventional cardiologists

2023-05-21
PHOENIX (May 20, 2023) – Today during the SCAI 2023 Scientific Sessions, thirty early-career interventional cardiologists were the inaugural recipients of a newly created award highlighting excellence in interventional cardiology. 30 in Their 30’s is a new recognition program created by the Society for Cardiovascular Angiography & Interventions (SCAI) for early-career members for their proven leadership and demonstration of SCAI’s core values. 30 in Their 30’s recognizes the best and brightest young professionals in interventional cardiology. This award ...

Society for Cardiovascular Angiography and Interventions bestows highest designation ranking to leading interventional cardiologists

2023-05-21
PHOENIX (May 20, 2023) – Today, the Society for Cardiovascular Angiography & Interventions (SCAI) announced its 2023 Master Interventionalists of SCAI (MSCAI) designation recipients during the SCAI 2023 Scientific Sessions in Phoenix. The MSCAI designation is awarded to individuals who have demonstrated excellence in invasive/interventional cardiology over the course of their career and for their commitment to the highest levels of clinical care, innovation, publication, and teaching. This year’s MSCAI designations were awarded to the following group of outstanding interventionalists: David ...

LAST 30 PRESS RELEASES:

Call for papers: 14th Asia-Pacific Conference on Transportation and the Environment (APTE 2025)

A novel disturbance rejection optimal guidance method for enhancing precision landing performance of reusable rockets

New scan method unveils lung function secrets

Searching for hidden medieval stories from the island of the Sagas

Breakthrough study reveals bumetanide treatment restores early social communication in fragile X syndrome mouse model

Neuroscience leader reveals oxytocin's crucial role beyond the 'love hormone' label

Twelve questions to ask your doctor for better brain health in the new year

Microelectronics Science Research Centers to lead charge on next-generation designs and prototypes

Study identifies genetic cause for yellow nail syndrome

New drug to prevent migraine may start working right away

Good news for people with MS: COVID-19 infection not tied to worsening symptoms

Department of Energy announces $179 million for Microelectronics Science Research Centers

Human-related activities continue to threaten global climate and productivity

Public shows greater acceptance of RSV vaccine as vaccine hesitancy appears to have plateaued

Unraveling the power and influence of language

Gene editing tool reduces Alzheimer’s plaque precursor in mice

TNF inhibitors prevent complications in kids with Crohn's disease, recommended as first-line therapies

Twisted Edison: Bright, elliptically polarized incandescent light

Structural cell protein also directly regulates gene transcription

Breaking boundaries: Researchers isolate quantum coherence in classical light systems

Brain map clarifies neuronal connectivity behind motor function

Researchers find compromised indoor air in homes following Marshall Fire

Months after Colorado's Marshall Fire, residents of surviving homes reported health symptoms, poor air quality

Identification of chemical constituents and blood-absorbed components of Shenqi Fuzheng extract based on UPLC-triple-TOF/MS technology

'Glass fences' hinder Japanese female faculty in international research, study finds

Vector winds forecast by numerical weather prediction models still in need of optimization

New research identifies key cellular mechanism driving Alzheimer’s disease

Trends in buprenorphine dispensing among adolescents and young adults in the US

Emergency department physicians vary widely in their likelihood of hospitalizing a patient, even within the same facility

Firearm and motor vehicle pediatric deaths— intersections of age, sex, race, and ethnicity

[Press-News.org] Tokyo Tech, Tohoku University, Fujitsu, and RIKEN start collaboration to develop distributed training of Large Language Models