Installed Software – Platform"HybriLIT"

System software (Linux, SLURM, Modules)

Intel Software Development Tools

System software (Linux, SLURM, Modules)

Scientific Linux

7.9 (Nitrogen)
SLURM
Modules
gcc

gcc/v4.9.3-1, gcc/v5.3.0-1, gcc/v6.2.0-2, gcc/v7.2.0-1, gcc/v8.2.0-1, gcc/v8.3.0, gcc/v9.1.0-1, gcc/v10.2.0, gcc/v11.2.0

Parallel computing software

* To view the current software version, type the following command in terminal: module avail and the name of the module, for example:

module avail CUDA

1	module avail CUDA

CUDA

CUDA library allows using resources of GPU NVIDIA.
Versions: v8.0-1, v9.2, v10.0-1, v10.1-1, v11.4
To set environmental variable, please use the following command:, for example:, for CUDA version 10.0-1:

module add CUDA/10.0-1

1

module add CUDA/10.0-1

Website: http://www.nvidia.ru/object/cuda-parallel-computing-ru.html
intel

Intel C++ Compiler, also known as icc or icl, is a group of C and C++ compilers from Intel.
Versions: v2018.1.163-9, v2019.3.199, v2021.1, intel-qs/v20-07-14, intel-qs/v21-01-14v
To set environmental variable, please use the following command (according to each version):

module add intel/v2021.1

1

module add intel/v2021.1

Website: https://software.intel.com/en-us/compilers
Средства Intel для разработки программного обеспечения
OpenMPI

Open source implementation of MPI-2 technology
Versions: v1.8.8-1, v2.1.2-2, v3.1.2, v3.1.3, v3.1.3_psm2, v4.1.1_gcc1120
To set environmental variable, please use the following command (according to each version):

module add openmpi/v3.1.3

1

module add openmpi/v3.1.3

Website: https://www.open-mpi.org/

Application Packages

* To view the current software version, type the following command in terminal: module avail and the name of the module, for example:

module avail CMake

1	module avail CMake

CMake is a cross-platform free and open-source software application for managing the build process of software using a compiler-independent method.
Versions: v3.16.5, v3.19.1, v3.22.0
To set environmental variable, please use the following command (according to each version):

module add CMake/v3.16.5

1	module add CMake/v3.16.5

Website: https://cmake.org/

FairSoft

The FairSoft framework is an object oriented simulation, reconstruction and data analysis framework.
Versions: may16p1_gcc485, oct17p1, may18p1, june19p1_gcc485, june19p2_gcc485, apr21patches_gcc1120
To set environmental variable, please use the following command (according to each version):

module add FairSoft/apr21patches_gcc1120

1	module add FairSoft/apr21patches_gcc1120

Website: https://fairroot.gsi.de/

FairRoot

The FairRoot framework is an object oriented simulation, reconstruction and data analysis framework.
Versions: v16.06_gcc485, oct17p1, v18.0.4, v18.2.0_gcc485, v18.2.1_gcc485, v18.4.2_gcc1120
To set environmental variable, please use the following command (according to each version):

module add FairRoot/oct17p1

1	module add FairRoot/oct17p1

Website: https://fairroot.gsi.de/

FLAIR

FLAIR – is an advanced user friendly interface for FLUKA to facilitate the editing of FLUKA input files, execution of the code and visualization of the output files.
Version: v2.3.0
To set environmental variable, please use the following command:

module add FLAIR v2.3.0

1	module add FLAIR v2.3.0

Website: http://www.fluka.org/flair/

FLUKA

FLUKA – is a fully integrated Monte Carlo simulation package for the interaction and transport of particles and nuclei in matter. FLUKA has many applications in particle physics, high energy experimental physics and engineering, shielding, detector and telescope design, cosmic ray studies, dosimetry, medical physics, radiobiology.
Versions: v2011.2x-8, v2020.0.3, v2021.2.2
To set environmental variable, please use the following command (according to each version):

module add FLUKA/v2011.2x-8

1	module add FLUKA/v2011.2x-8

Website: http://www.fluka.org/fluka.php

GEANT4

GEANT – is a platform for “the simulation of the passage of particles through matter” using Monte Carlo methods.
Version: v4.10.07.p01_gcc910
To set environmental variable, please use the following command:

module add GEANT4/v4.10.07.p01_gcc910

1	module add GEANT4/v4.10.07.p01_gcc910

Website: http://www.geant4.org/geant4/

GROMACS

GROMACS is a molecular dynamics package mainly designed for simulations of proteins, lipids, and nucleic acids.
Versions: v2019.3, v5.1.3_gcc485_cuda80
To set environmental variable, please use the following command (according to each version):

module add GROMACS/v2019.3

1	module add GROMACS/v2019.3

Website: http://www.gromacs.org/

java

java is a compiler for the programming language Java.

Version: v8u181
To set environmental variable, please use the following command:

module add java v8u181

1

module add java v8u181
Version: v8u91-1
To set environmental variable, please use the following command:

module add java v8u91-1

1

module add java v8u91-1

LAMMPS

LAMMPS – is a molecular dynamics program from Sandia National Laboratories. LAMMPS makes use of Message Passing Interface (MPI) for parallel communication and is free and open-source software, distributed under the terms of the GNU General Public License.
Version: v12.12.18
To set environmental variable, please use the following command:

module add LAMMPS v12.12.18

1	module add LAMMPS v12.12.18

Website: https://lammps.sandia.gov/

PandaRoot

PandaRoot – a framework for offline simulation framework for The PANDA experiment at FAIR in Darmstadt based upon the ROOT and Virtual Monte-Carlo.Versions: dec17p2b, may19
To set environmental variable, please use the following command (according to each version):

module add PandaRoot/dec17p2b

1	module add PandaRoot/dec17p2b

Website: https://panda.gsi.de/

Python

Python – an interpreted, high-level, general-purpose programming language. Python has a design philosophy that emphasizes code readability, notably using significant whitespace. It provides constructs that enable clear programming on both small and large scales.
Versions: 2.7.10-3, 3.6.5
To set environmental variable, please use the following command (according to each version):

module add Python/v2.7.10-3

1	module add Python/v2.7.10-3

Website: https://www.python.org/

REDUCE

REDUCE is a system for doing scalar, vector and matrix algebra by computer, which also supports arbitrary precision numerical approximation and interfaces to gnuplot to provide graphics.Version: svn-4830
Version: svn-4830
To set environmental variable, please use the following command:

module add reduce-algebra/svn-4830

1	module add reduce-algebra/svn-4830

Website: https://reduce-algebra.sourceforge.io

ROOT

ROOT is a modular scientific software toolkit. It provides all the functionalities needed to deal with big data processing, statistical analysis, visualisation and storage.
Version: 6-18-00
To set environmental variable, please use the following command:

module add ROOT/v6-18-00

1	module add ROOT/v6-18-00

Website: https://root.cern.ch/

Licensed software packages

* To view the current software version, type the following command in terminal: module avail and the name of the module, for example:

module avail COMSOL

1	module avail COMSOL

Comsol Multiphysics

Comsol Multiphysics is a cross-platform finite element analysis, solver and multiphysics simulation software. It allows conventional physics-based user interfaces and coupled systems of partial differential equations (PDEs). COMSOL provides an IDE and unified workflow for electrical, mechanical, fluid, and chemical applications.
Version: 6.0, 6.2
Available licenses: 4 graphic and 1 for each module
To set environmental variable, please use the following command:

module add COMSOL/v6.0

1

module add COMSOL/v6.0

Website: https://www.comsol.ru/comsol-multiphysics
Maple

Maple is a symbolic and numeric computing environment, and is also a multi-paradigm programming language. Developed by Maplesoft, Maple also covers other aspects of technical computing, including visualization, data analysis, matrix computation, and connectivity.
Version: 2020.2
Available licenses: 5
To set environmental variable, please use the following command:

module add Maple/v2020.2

1

module add Maple/v2020.2

Website: https://www.maplesoft.com/products/maple/
Wolfram Mathematica

Wolfram Mathematica is a modern technical computing system spanning most areas of technical computing — including neural networks, machine learning, image processing, geometry, data science, visualizations, and others. The system is used in many technical, scientific, engineering, mathematical, and computing fields.
Version: 11.2-1
Available licenses: 5
To set environmental variable, please use the following command:

module add Mathematica/v11.2-1

1

module add Mathematica/v11.2-1

Website: https://reference.wolfram.com/language/
Matlab

Matlab (matrix laboratory) is a multi-paradigm numerical computing environment and proprietary programming language developed by MathWorks. MATLAB allows matrix manipulations, plotting of functions and data, implementation of algorithms, creation of user interfaces, and interfacing with programs written in other languages, including C, C++, C#, Java, Fortran and Python.
Version: R2020b, R2022b
Available licenses: 6
To set environmental variable, please use the following command:

module add MATLAB/R2020b

1

module add MATLAB/R2020b

Website: https://www.mathworks.com/help/matlab/

Libraries

* To view the current software version, type the following command in terminal: module avail and the name of the module, for example:

module avail CLN

1	module avail CLN

CLN

CLN is a library for efficient computations with all kinds of numbers in arbitrary precision.
Version: 1.3.4-1
To set environmental variable, please use the following command:

module add CLN/v1.3.4-1

1	module add CLN/v1.3.4-1

Website: https://www.ginac.de/CLN/

FFTW

The Fastest Fourier Transform in the West (FFTW) is a software library for computing discrete Fourier transforms (DFTs).
Versions: v3.3.7-2, v3.3.7-5
To set environmental variable, please use the following command (according to each version):

module add fftw/v3.3.7-2

1	module add fftw/v3.3.7-2

Website: http://www.fftw.org/

Ginac

Ginac – is C++ libarary, is a free computer algebra system released under the GNU General Public License. The name is a recursive acronym for “GiNaC is Not a CAS” (Computer Algebra System).
Version: v1.7.3-1
To set environmental variable, please use the following command:

module add Ginac/v1.7.3-1

1	module add Ginac/v1.7.3-1

Website: https://www.ginac.de/

GSL v1.16

The GNU Scientific Library (or GSL) is a software library for numerical computations in applied mathematics and science. The GSL is written in C; wrappers are available for other programming languages.

Versions: v1.16-1, v2.6
To set environmental variable, please use the following command:

module add gsl/v2.6

1	module add gsl/v2.6

Website: https://www.gnu.org/software/gsl/

PostgreSQL

PostgreSQL is an object-relational database management system (ORDBMS) with an emphasis on extensibility and standards compliance.
Version: 12.1
To set environmental variable, please use the following command:

module add PostgreSQL/v12.1_gcc910

1	module add PostgreSQL/v12.1_gcc910

Website: https://postgresql.org/

Protocol Buffers

Protocol Buffers is a method of serializing structured data developed by Google.
Version: 3.11.3
To set environmental variable, please use the following command:

module add protobuf/v3.11.3

1	module add protobuf/v3.11.3

Website: https://developers.google.com/protocol-buffers/

Quantum ESPRESSO

Quantum opEn-Source Package for Research in Electronic Structure, Simulation, and Optimization is an integrated suite of Open-Source computer codes for electronic-structure calculations and materials modeling at the nanoscale. It is based on density-functional theory, plane waves, and pseudopotentials.
Version: 6.4.1, 6.6_oneapi
To set environmental variable, please use the following command:

module add quantum-espresso/v6.4.1

1	module add quantum-espresso/v6.4.1

Website: http://www.quantum-espresso.org/

TRNG

Tina’s Random Number Generator Library is a state of the art C++ pseudo-random number generator library for sequential and parallel Monte Carlo simulations.
Version: 4.24
To set environmental variable, please use the following command:

module add trng/v4.24_gcc102

1	module add trng/v4.24_gcc102

Website: https://www.numbercrunch.de/trng/

zlib

Free cross-platform data compression library.
Version: 1.2.11
To set environmental variable, please use the following command:

module add zlib/v1.2.11-1

1	module add zlib/v1.2.11-1

Website: http://www.zlib.net/

Intel Software Development Tools

Aplication Performance Snapshot

Intel Trace Analyzer and Collector

Open Visual Inference & Neural Network Optimization

Aplication Performance Snapshot

Aplication Performance Snapshot is a tool for application profiling. This application is included to the Intel Parallel Studio installed on the HybriLIT cluster.
Aplication Performance Snapshot allows analyze such parameters as memory usage, work time of each process, operating delay, etc.

In order tp profile an application, it’s necessary to:

Load a module that provides support of envarinmental variables for work with Intel Parallel Studio:

$ module add intel/v2019.0.018

1	$ module add intel/v2019.0.018

Compile the application by means of loading a library:

For MPI programs

$ mpiicc name_app.cpp

1	$ mpiicc name_app.cpp

for OpenMP programs

$ icc –qopenmp name_app.cpp

1	$ icc –qopenmp name_app.cpp

$ icc name_app.cpp

1	$ icc name_app.cpp

for a sequential program where name_app.cpp – is the name of the compiled file.

Run the executable file (by default a.out) with the key – aps, or using a script-file:

For MPI programs

#!/bin/sh
#SBATCH -p cpu
#SBATCH -n 4
#SBATCH -t 600
mpirun aps ./a.out

#!/bin/sh

#SBATCH -p cpu

#SBATCH -n 4

#SBATCH -t 600

mpirun aps ./a.out

for OpenMP programs

#!/bin/sh
#SBATCH -p cpu
#SBATCH -n 4
#SBATCH -t 600
export OMP_NUM_THREADS=5
export OMP_PLACES=cores
aps ./a.out

#!/bin/sh

#SBATCH -p cpu

#SBATCH -n 4

#SBATCH -t 600

export OMP_NUM_THREADS=5

export OMP_PLACES=cores

aps ./a.out

for a sequential program

#!/bin/sh
#SBATCH -p cpu
#SBATCH -n 1
#SBATCH -t 600
aps ./a.out

#!/bin/sh

#SBATCH -p cpu

#SBATCH -n 1

#SBATCH -t 600

aps ./a.out

All received data will be placed in the folder aps_result_[launch date]. In order to display these data, generate a report in HTML (pic 1, pic 2) using the following command:

$ aps -R aps_result_ [дата запуска]

1	$ aps -R aps_result_ [дата запуска]

Pic.1 . Screenshot of an output-file of Aplication Perfomance Snapshot for MPI application (click the picture to see it at the report page)

Pic. 2. Screenshot of an output-file of Aplication Perfomance Snapshot for OpenMP application (click the picture to see it at the report page)

Intel Trace Analyzer and Collector

Intel Trace Analyzer and Collector is aimed at tracing MPI processes..

Main tasks:

Behavior vizualization of a parallel application;
Estimation of statistic profile and load balancing;
Performance analysis of programs and code blocks;
Detailing of model of exchange and performance data;
Detection of a hotspot;
Decrease of execution time and increase of application efficiency.

To launch an application use parameter –trace. Please see an example below:

#!/bin/sh
#SBATCH -p cpu
#SBATCH -n 4
#SBATCH -t 600
mpirun -trace ./a.out

#!/bin/sh

#SBATCH -p cpu

#SBATCH -n 4

#SBATCH -t 600

mpirun -trace ./a.out

To view results, please use ITAC GUI by means of the command:

$ traceanalyzer

1	$ traceanalyzer

As a result, an application winsow will be opened and you will need to upload a file (a.out.stf by default). An examples is shown on Pic. 3.

Distribution of computational load on MPI processes and interprocess communication (Event Timeline scale) are shown at the top of Pic.3.; numeric values are shown at the bottom of Pic.3.

Pic. 3. Examples of communication in an MPI application.

By means of Message Profile, it is now possible to estimate which processes comunicate and which communications are the most time consuming (Pic. 4.).

Pic. 4. Interprocess communication in an MPI application.

In Collective operations page, it is possible to prepare a table that reflects total span time for collective operations in MPI application (Pic.5.).

Рис. 5. Таблица временных затрат на коллективные операции в MPI-приложении.

Conclusion
Application Performance Snapshot — is aimed at fast estimation of efficiency; it doesn’t require additional charges; provides profiling up to 32000 MPI processes; allows get fasr estimation of MPI and OpenMP disbalance; and provides total estimation of performance (GFLOPS).
Intel Trace Analyzer and Collector — allows carrying out detailed analysis of MPI applications, detect communication patterns, and locate specific bottle neck of programs.

Open Visual Inference & Neural Network Optimization

OpenVINO (Open Visual Inference и Neural Network Optimization) is a free set of tools that simplifies the optimization of the deep learning model from the infrastructure and deployment using the output mechanism on Intel hardware.

The Movidius Neural Compute Stick (NCS Movidius)is a miniature accelerator for solutions related to artificial intelligence, neural networks and deep learning. The accelerator allows implementing elements of artificial intelligence on various platforms and devices. The given device is not designed for training neural network models.

The major advantage of the product is its performance, minimum size and almost zero number of dependencies. The product is ideal for implementing applications that use Deep Learning and Computer Vision to solve problems. Its disadvantage is that support for training neural networks is not included in the product.

This manual describes the steps to install and configure the tool OpenVINO Tools for Windows and Linux, as well as the work with the model optimizer.

For output to Inference Engine, read the documentation specifically for your case: Inference Engine Developer Guide

Quantum computing tools

Intel-QS

Intel Quantum Simulator (Intel-QS), also known as qHiPSTER (The Quantum High Performance Software Testing Environment), is a simulator of quantum circuits optimized to take maximum advantage of multi-core and multi-nodes architectures. It is based on a complete representation of the qubit state, but avoids the explicit representation of gates and other quantum operations in terms of matrices. Intel-QS uses the MPI (message-passing-interface) protocol to handle communication between the distributed resources used to store and manipulate quantum states.

To set environmental variable, please use the following command:

<strong>module add intel-qs/v20-07-14</strong>

1	<strong>module add intel-qs/v20-07-14</strong>

<strong>module add intel-qs/v21-01-14</strong>

1	<strong>module add intel-qs/v21-01-14</strong>

Website: https://github.com/iqusoft/intel-qs

QuEST

The Quantum Exact Simulation Toolkit is a high-performance simulator of quantum circuits, state-vectors and density matrices. QuEST uses multithreading, GPU acceleration and distributed computing systems.

Website: https://quest.qtechtheory.org/

Scientific Linux

SLURM

Modules

gcc

CUDA

intel

OpenMPI

Comsol Multiphysics

Maple

Wolfram Mathematica

Matlab

Intel-QS

QuEST