Description of the ML/DL ecosystem
The active implementation of the neural network approach, methods and algorithms of machine learning and deep learning (ML/DL) for solving a wide range of problems is defined by many factors. The development of computing architectures, especially while using DL methods for training convolutional neural networks, the development of libraries, in which various algorithms are implemented, and frameworks, which allow building different models of neural networks can be referred to the main factors. To provide all the possibilities both for developing mathematical models and algorithms and carrying out resource-intensive computations including graphics accelerators, which significantly reduce the calculation time, an ecosystem for tasks of ML/DL and data analysis has been created and is actively developing for HybriLIT platform users.
Video “Introduction into ML/DL/HPC Ecosystem”
Oksana Streltsova, Deputy Head of the Group, LIT JINR (in Russian)
Video by A.S. Vorontsov
The created ecosystem has three components (Fig.1):
- the first component is designed for the development of models and algorithms on the JupyterHub – basis, i.e. a multi-user platform for working with Jupyter Notebook (known as IPython with the possibility to work in a web browser) – https://jhub.jinr.ru;
- the second component is aimed at carrying out resource-intensive, massive parallel tasks, for example, for neural network training using NVIDIA graphics accelerators – https://jhub2.jinr.ru;
- the third component JLabHPC is designed for calculations on the compute nodes of the HybriLIT platform, application development and scientific visualization – https://jlabhpc.jinr.ru.
The virtual machine (VM) parameters for the first and third components and the servers with graphics processors NVIDIA Volta are presented in Fig.1.
The most frequently used libraries and frameworks installed on the components for solving tasks of ML/DL and data analysis are given below.
Work within the ML/DL/HPC ecosystem
To get started you need to:
- Log in with your HybriLIT account in GitLab:
- Enter the components (the autorization is done via GitLab):
(without graphics accelerators)
|Component for carrying out resource-intensive calculations
(with graphics accelerators NVIDIA)
|Component for HPC on the HybriLIT platform nodes and data analysis
(JupyterHub and SLURM)
After the authorization the Jupyter Notebook interactive environment opens:
Home directories are available for users, the file systems NFS, ZFS, EOS are reinstalled.
How to create a directory or a file is shown on the screenshot above.