Python personal environment

IDRIS offers many modules dedicated to AI. The modules are designed for the most efficient use of the Jean Zay intrastructure and the installed libraries (such as cuDNN, Intel MKL, NCCL).

If a module corresponds to your needs but lacks libraries which are indispensable for your codes, you have two solutions:

• Make a request to the user support team ([email protected]) to install the library and specify the concerned library and module. This will also potentially benefit other Jean Zay users.
• Add the missing libraries locally via the pip command, by loading one of the proposed modules as an extension to the environment. The advantages and disadvantages of this method are described below.

If what you need is really very different from what is offered by IDRIS, you can create your own conda environment with exactly what is necessary for you. This gives you total control but there are some disadvantages:

• An environment takes up space (particularly in terms of inodes) and could saturate your allocated space. See the following section for tactics to limit this problem.
• The installed codes are not optimised for Jean Zay and are not necessarily going to find core libraries such as cuDNN (even by loading the cudnn/7.6.5.32-cuda-10.2).
• The user support team will have more difficulty assisting you in case your code shows abnormal conduct.

Some useful commands:

• To know which modules are available: module avail, for example module avail python
• To load a module: module load <module_name>
• To obtain a complete list of associated modules: module list (after a module is loaded)
• To “unload” all the modules: module purge (in case of anomaly, it could be preferable to begin a new session)

The storage directory for packages PyPI is situated in $HOME/.local by default. Therefore, you could rapidly saturate the disk quota of your $HOME if you keep this default behaviour. To avoid this, we recommend that you create a target directory in your $WORK and create a symbolic link in the $HOME which points to the target directory:

• If the $HOME/.local file already exists, you must first move it to the $WORK:

$ mv $HOME/.local $WORK
$ ln -s $WORK/.local $HOME

• If the $HOME/.local file does not yet exist, you must first create a directory in the $WORK:

$ mkdir $WORK/.local
$ ln -s $WORK/.local $HOME

Likewise, via conda, libraries are installed in $HOME/.conda; it is strongly encouraged, therefore, to proceed in the same way.

• If the $HOME/.conda file already exists, you must first move it to the $WORK:

$ mv $HOME/.conda $WORK
$ ln -s $WORK/.conda $HOME

• If the $HOME/.conda file does not yet exist, you must first create a directory in the $WORK:

$ mkdir $WORK/.conda
$ ln -s $WORK/.conda $HOME

To know the usage of the $WORK:

• idr_quota_user --space work gives the usage percentage in GB and inodes but does not provide real-time information (updated every morning).
• For more detail, enter du --inodes --max-depth=1 $WORK or du -h --max-depth=1 $WORK.

Advantages and disadvantages of this method:

• This method is suited for adding libraries to an existing module and avoids needing to install everything.
• In return, however, libraries installed via pip will be visible to every module which uses the same version of Python (installation is done in .local/lib/pythonx.y).
• It is necessary, therefore, to be vigilant about possible incompatibilities if you launch calculations with multiple environments (for example, pytorch-gpu/py3/1.5.0 and pytorch-gpu/py3/1.7.1 are based on the same version of Python (3.7) but not the same version of numpy).
• Local installation of a library takes precedence over the version from a module but it is necessary to stay vigilant about the dependencies.

To locally install packages from the PyPI storage, use the --user and --no-cache-dir options:

$ pip install --user --no-cache-dir <package>

$ pip uninstall <package>

Other useful commands (for the options, see official documentation):

• pip cache purge (useful if you have forgotten the --no-cache-dir option)
• pip list to see all the accessible packages (from the module or from a local installation)

You do not need to install Miniconda to create your own environment! You just need to load a Miniforge module, for example by launching:

 module load miniforge/24.9.0

To list all the versions of Miniforge available:

module avail miniforge

The useful commands are:

• conda deactivate to exit from the environment loaded with module load
• conda create -n <name> python=x.y to create an environment called “name” including the Python version x.y (the Python version of the module is maintained by default otherwise)
• conda activate <name> to activate the environmenet
• conda env remove --name <name> to correctly delete the environment
• conda clean -a to remove unused packages and the cache, you will avoid saturating your disk spaces this way

Note about the Anaconda paid license: since summer 2024, the use of Anaconda repositories has been subject to a fee. The behavior of any existing conda environment on Jean Zay has been modified, and all installations carried out by conda target the conda-forge open repository in priority. We strongly encourage you to work with Miniforge rather than Anaconda to avoid any request to paid Ananconda repository.

• The variables $PATH, $PYTHONPATH and $CONDA_PREFIX are configured correctly; it should not be necessary to modify them (which could change the directory precedence order, or result in paths to incompatible packages). However, if a pip installation creates an executable (this is the case in Horovod and Deepspeed), it will be located in ~/.local/bin. To access this executable, it will be necessary to edit the PATH as follows:

   export PATH=~/.local/bin:$PATH

• If you use multiple Jean Zay modules alternatively (for example, Tensorflow and Pytorch) and want to complete each one of them with different personnal “pip” installations, it is possible to redefine the location of locally installed packages by defining the PYTHONUSERBASE variable. For example, to place your personnal installations related to the tensorflow-gpu-2.4.0 environment in a directory called $WORK/.local_tf240, you can run the following command:

  export PYTHONUSERBASE=$WORK/.local_tf240

  This must be done before the packages installation and before launching the computations.

If ever you need to compile AI codes using the GPUs (CUDA, cuDNN), intel MKL, etc., do not hesitate to contact the user support team ([email protected]).