Tokyo Tech, Tohoku University, Fujitsu and RIKEN launch collaboration to develop large-scale language models: Fujitsu Global

Tokyo Institute of Technology, Tohoku University, Fujitsu Ltd., RIKEN

Tokyo, May 22, 2023

Tokyo Institute of Technology (Tokyo Tech), Tohoku University, Fujitsu Co., Ltd., and RIKEN have announced that they will begin research and development of distributed training for large language models (LLM). ⁽¹⁾ On the supercomputer Fugaku in May 2023 within the scope of the Fugaku Utilization Initiative defined by Japanese policy.

LLMs are deep learning AI models that serve as a core component of generative AI, including ChatGipt. ⁽²⁾. The four organizations will contribute to improving the environment for creating LLMs that can be widely used by academics and companies, improve AI research capabilities in Japan, and share the results to increase the value of using Fugaku in academic and industrial fields. This R&D forward.

Background

While many envision LLMS and generative AI technologies to play a fundamental role in research and development for security, the economy, and society, the development and improvement of these models requires high-performance computing resources that can efficiently process high-performance computing. Data size.

Tokyo Tech, Tohoku University, Fujitsu, and RIKEN are developing LLMS courses that focus on research and development for this purpose.

Application period

May 24, 2023 to March 31, 2024 *Starting period for using fugaku for Japanese policies

The roles of each organization and company

The technology used in this initiative enables the organizations to efficiently perform large-scale language model training on the supercomputer Fugaku’s massively parallel computing environment. The roles of each organization and company are as follows.

Tokyo Institute of Technology: General Process Control, LLMS Parallelism and Acceleration

Tohoku University: Learning data collection, model selection

Fujitsu: LMM

RIKEN: Distributed Parallel and Linkage Acceleration of LLMS, LLMS Acceleration

Future plans

To support Japanese researchers and engineers in developing future LLMs, the four organizations plan to publish research results from initiatives designed to use Fugaku as defined by GitHub policy in Japan. ⁽³⁾ and huggable face ⁽⁴⁾ In fiscal 2024. In addition, many researchers and engineers are expected to participate in the development of basic models and new applied research, creating efficient methods for further innovative research and business results.

The four firms are from Nagoya University, which develops data generation and learning methods for multimodal applications in industrial fields such as manufacturing, and CyberAgent Inc., which provides data and technology to build LLMs.

Comment from Toshio Endo, Professor, Global Scientific Information and Computing Center, Tokyo Institute of Technology:

“The collaboration combines parallelization and acceleration of large-scale language models using Tokyo Tech and RIKEN’s supercomputer “Fugaku”, Fujitsu’s high-performance computing infrastructure software for Fugaku and performance tuning of AI models, and Tohoku University’s natural language. Technology. In collaboration with Fujitsu in 202X We will use a small research laboratory that we have established under the name of “Fujitsu Collaborative Research Center for Next-Generation Computing Infrastructure”. We look forward to working with our colleagues to contribute to the improvement of Japan’s AI research capabilities by leveraging the deep learning capabilities of “Fugaku”.

Kentaro Inui, a professor at Tohoku University’s Graduate School of Information Science, commented.

“Our goal is to build an open-source, commercially available, and primarily Japanese data-based, comprehensive language model with transparency on the training data. By enabling learning data discovery, this avoids black-box problems, biases, misinformation, and “cheats” common with AI. We expect that it will facilitate robust research to scientifically validate the so-called issues. We will build large-scale models using the insights we gained from the deep learning of Japanese natural language processing at Tohoku University. We look forward to contributing to the development of AI research capacity in our country and beyond, with the results of our research inspired by researchers and developers. By sharing.

Comment from Seishi Okamoto, EVP, Head of Fujitsu Research, Fujitsu Ltd.:

“We are delighted to have the opportunity to leverage the supercomputer Fugaku’s powerful parallel computing resources to advance AI research and expand the research and development of LLMS. Going forward, we aim to incorporate the fruits of this research into Fujitsu’s new AI platform, dubbed “Kozuchi,” to deliver paradigm-shifting applications that contribute to the realization of a sustainable society.

Comment from Satoshi Matsuoka, director of the RIKEN Computational Science Center:

“A64FX ⁽⁵⁾ The CPU has an AI acceleration function known as SVE.

Software development and optimization are essential to maximize its potential and use for AI applications. We feel that this joint research will play an important role in bringing together LLMs and computer science experts in Japan, RIKEN R-CCS researchers and engineers, to advance the techniques of building LMS on the supercomputer “Fugaku”. Together with our partners, we contribute to making Society 5.0 a reality.

Project name

Distributed Training of Large Language Models on Fugaku (Project Number: hp230254)

---

Press the contacts

---

All company or product names mentioned herein are trademarks or trademarks of their respective owners. The information presented in this press release is correct at the time of publication and is subject to change without prior notice.

day – May 22, 2023

City: Tokyo, Japan.

Company: Tokyo Institute of Technology, Tohoku University, Fujitsu Ltd., RIKEN