Jean Zay: Softwares for artificial intelligence

The artificial intelligence softwares are installed in conda virtual environments.

These environments can be supplemented upon request to the IDRIS support team (assist@idris.fr).

Since support for Python version 2.7 ended on 01/01/2020, we are only installing new software versions for Python 3 since.

The conda environments which we install are built around four principal libraries:

• TensorFlow
• PyTorch
• Caffe
• MXNet

These environments are accessible via the module command. To display the list of available AI modules:

$ module avail tensorflow pytorch caffe mxnet

To activate your chosen environment, it is necessary to load the corresponding module. For example, to activate the conda environment containing TensorFlow version 2.5.0, you should load:

$ module load tensorflow-gpu/py3/2.5.0

Once the environment is activated, you can list all the Python packages installed in this environment by using the following commands:

$ pip list
$ conda list

To list the versions installed on Jean Zay for a given environment (TensorFlow, PyTorch, caffe or MXNet), you must use the module avail command. For example, regarding TensorFlow:

$ module avail tensorflow

The names of the products appear in the form: <software>-gpu/<python version>/<product version>. Except for MXNet which have been installed after the end of life of Python 2.
For example, TensorFlow version 2.2.0 for Python 3 will be named tensorflow-gpu/py3/2.2.0.

When you load one of these products via the module command, you also load several dependencies: OpenMPI, CUDA, NCCL, cuDNN,… The versions of these dependencies are indicated at the loading of the product. For example :

$ module load tensorflow-gpu/py3/2.2.0
Loading tensorflow-gpu/py3/2.2.0
  Loading requirement: cuda/10.1.2 nccl/2.5.6-2-cuda cudnn/7.6.5.32-cuda-10.1 intel-compilers/19.0.4 openmpi/4.0.2-cuda

You are also loading the corresponding Python version (py3 or py2).

Note: New versions can be installed upon request to the IDRIS support team (assist@idris.fr).

• The module purge, module switchand module unload commands do not function with conda. To deactivate a conda environment and activate another one by loading another module, you need to run the following commands:

  conda deactivate
  module purge
  module load <new_environment>

• To use environments compatible with the A100 GPUs, you need to load the following module beforehand:

  module load arch/a100

  For more informations: Modules compatible with ''gpu_p5'' partition.

• To use environments compatible with the H100 GPUs, you need to load the following module beforehand:

  module load arch/h100

  For more informations: Modules compatible with ''gpu_p6'' partition.
  Important: the H100 software ecosystem is recent and does not support old versions of libraries. We do not plan to install TensorFlow versions lower than 2.17 or PyTorch versions lower than 2.3.1 on this partition.

• A conda environment dedicated to the domaine of atomistic simulation is available via the following command:

  $ module load atomistic_simulation/py3

  You can find more information on the page of associated documentation.