Megatron GPT Bootcamp
Title: Megatron GPT Bootcamp
SNIC Project: Berzelius-2021-53
Project Type: LiU Berzelius
Principal Investigator: Johan Raber <raber@nsc.liu.se>
Affiliation: Linköpings universitet
Duration: 2021-10-01 – 2021-11-01
Classification: 10201
Homepage: https://www.gpuhackathons.org/event/enccs-nsc-megatron-bootcamp
Keywords:

Abstract

This is a course project for the Megatron Bootcamp course held between Oct 25-27 in a collaboration between NSC, NVIDIA and ENCCS. The course sessions are of the half-day type, held before lunch. NVIDIA Megatron-LM is an open-source framework for training very large language models. The course has a cap of 40 participants, most seats of which are expected to be filled. Software used in the course will be PyTorch container images prepared in advance as well as the NVIDIA Nsight Systems profiling tool (available on the system) to visualise profile traces in a ThinLinc environment.