IDRIS

Institute for Development and Resources in Intensive Scientific Computing

Navigation du site

My projects

On the Jean Zay machine, each scientific project is identified by a Unix group. IDRIS has developed the commands idrproj (see idrproj -h) and idrenv (see idrenv -h) which allow you to list your projects and/or select a project.

The idrproj command

By default, the idrproj command displays the different Unix groups which reference your projects. Example for a login attached to two projects, grp1 and grp2:

$ idrproj
 
Available projects:
-------------------
  grp1 (123456) 
  grp2 (167890) [default][active]

Note that, for each project, the command indicates:

The Unix group of reference (here “grp1” and “grp2”).
The IDRIS project number (here “123456” and “167890”) which is different from the DARI project number.
Which project is active (indicated by [active]): the project on which you are currently working.
The default project (indicated by [default]): the project which is automatically selected at each connection on the machine and, therefore, active by default.

Change the default project

To change the default project ([default]), use the -d option of the idrproj command. With the preceding example, this is as follows:

$ idrproj -d grp1
Password:
Default project changed to grp1

Note that changing the default project requires the user password.

Then, verification of the default project:

$ idrproj
 
Available projects:
-------------------
  grp1 (123456) [default]
  grp2 (167890) [active]

Important: This does not change the active project (here “gpr2”) of the current session so you are still working on the same project! However, the change in default project remains in effect for future connections on the machine. Consequently, the new default project (here “gpr1”) will be the active project for all the new connections. Thus, after reconnection, the display of the idrproj command for this example will resemble the following:

$ idrproj
 
Available projects:
-------------------
  grp1 (123456)  [default] [active]
  grp2 (167890)

The changing of the default project is particularly useful for switching to another project if your current project doesn't have any computing hours left (hours allocation exhausted or the project not renewed) or if you wish to compute or work on another project for a while.

Note that it may happen that you no longer have a default project (and therefore no longer an active project) if the one you had has been deleted because it has not been renewed. The IDRPROJ, WORK, STORE and SCRATCH variables will then no longer be defined. You just need to follow the change default project procedure above to restore a correct compute environment.

Note that you can also temporarily activate a project (i.e. for only one work session) as indicated in the section below, “Changing the active project”.

The idrenv command

By default, the IDRIS command idrenv simply displays all the environment variables (the WORK, SCRATCH and STORE common spaces, IDRPROJ) related to the default project which is automatically selected and activated at each connection on the machine (in this example, “grp1”) as well as all the environment variables indicating the ad hoc directories of your other projects (in this example, “grp2”) :

$ idrenv
export ALL_CCFRSCRATCH=/...scratchpath.../grp1/commun; 
export ALL_CCFRSTORE=/...storepath.../grp1/commun; 
export ALL_CCFRWORK=/...workpath.../grp1/commun; 
export CCFRHOME=/...homepath.../gengrp/login; 
export CCFRSCRATCH=/...scratchpath.../grp1/login; 
export CCFRSTORE=/...storepath.../grp1/login; 
export CCFRWORK=/...workpath.../grp1/login; 
export HOME=/...homepath.../gengrp/login; 
export IDRPROJ=grp1; 
export grp1_ALL_CCFRSCRATCH=/...scratchpath.../grp1/commun; 
export grp1_ALL_CCFRSTORE=/...storepath.../grp1/commun; 
export grp1_ALL_CCFRWORK=/...workpath.../grp1/commun; 
export grp1_CCFRSCRATCH=/...scratchpath.../grp1/login; 
export grp1_CCFRSTORE=/...storepath.../grp1/login; 
export grp1_CCFRWORK=/...workpath.../grp1/login; 
export grp2_ALL_CCFRSCRATCH=/...scratchpath.../grp2/commun; 
export grp2_ALL_CCFRSTORE=/...storepath.../grp2/commun; 
export grp2_ALL_CCFRWORK=/...workpath.../grp2/commun; 
export grp2_CCFRSCRATCH=/...scratchpath.../grp2/login; 
export grp2_CCFRSTORE=/...storepath.../grp2/login; 
export grp2_CCFRWORK=/...workpath.../grp2/login; 
export SCRATCH=/...scratchpath.../grp1/login; 
export STORE=/...storepath.../grp1/login;  
export WORK=/...workpath.../grp1/login;

Comments:

The IDRPROJ environment variable references the active project on which you are currently working (echo $IDRPROJ).
Variables whose names contain the “CCFR” character string are defined in order to have naming coherency between the different national computing centres.
If the variable names begin by “grp1” or “grp2”, you may access the data (directories) of the corresponding projects regardless of which project you are working on at that time..

Changing the active project

To change the active project (indicated by [active]), you should use the -d option of the IDRIS command idrenv and pass both in argument of the Unix command eval. Thus, using the first example, to activate the “grp1” project:

$ eval $(idrenv -d grp1)

Then, verification of the active project:

$ idrproj
 
Available projects:
-------------------
  grp1 (123456) [active]
  grp2 (167890) [default]

Caution: Changing the active project (here “gpr1”) is only effective for the work session in which the command is executed.

Comments:

This does not change the default project (here “gpr2”) which remains the active project for future connections on the machine. Therefore, after reconnection, the idrproj command display for this example will resemble as follows :
```
$ idrproj
 
Available projects:
-------------------
  grp1 (123456)
  grp2 (167890)  [default] [active]
```
As the default project is not changed when changing the active project, the display of the idrenv command (without options) will not correspond to that of the active project. To obtain the correct display, you must then use the command idrenv -d $IDRPROJ.

Good practices for multi-projects

The first thing you should always do is verify which project you are currently working on via one of the following commands:
- The content of the environment variable IDRPROJ:
```
$ echo $IDRPROJ
grp2
```
  Comment: The IDRPROJ environment variable references the active project to which you are currently attached.
- The IDRIS command idrproj :
```
$ idrproj
 
Available projects:
-------------------
  grp1 (123456) 
  grp2 (167890) [default] [active]
```
  Comment: One of your projects is automatically selected by default ([default]).

If needed, following this verification, you may switch to another of your projects on which you wish to work:
- For a change of active project valid only for the current working session:
```
$ eval $(idrenv -d grp1)
```
- For a change of default project which will only be active for the following connections, therefore involving a disconnection / reconnection:
```
$ idrproj -d grp1
```
Verify that the switch has been made via one of the following commands:
- The content of the IDRPROJ environment variable:
```
$ echo $IDRPROJ
grp1
```
- The IDRIS command idrproj:
```
$ idrproj
 
Available projects:
-------------------
  grp1 (123456) [active]
  grp2 (167890) [default]
```

In this way, you can be assured of creating your files in the WORK, SCRATCH and STORE common directories which are attached to the project you are currently working on. You can then work (create/edit files, compile, …) with a perfectly defined environment.

However, to submit SLURM jobs, or for interactive executions, you must imperatively specify the hours allocation on which the consumed computing hours should be counted. Without this information, the consumed hours could be counted for a different project than the one on which you are actually working.

It is compulsory, therefore, that you add one of the following:

Either the SLURM directive #SBATCH --account=... in the header of the submission script,
Or the --account=... option to the sbatch, salloc or srun commands,

With taking care to indicate the right project (my_project) and the right type of hours (cpu or v100) as accounting value:

Thus, for cpu hours:

# As argument for the sbatch, salloc and srun commands:
$ sbatch ... --account=my_project@cpu job_cpu.slurm
 
# Or via the SLURM directive in a batch job:
#SBATCH --account=my_project@cpu

And for v100 gpu hours:

# As argument for the sbatch, salloc and srun commands:
$ sbatch ... --account=my_project@v100 job_gpu.slurm
 
# Or via the SLURM directive in a batch job:
#SBATCH --account=my_project@v100

Comment: The value of my_project corresponds to the value of IDRPROJ environment variable which references the active project on which you are currently attached: echo $IDRPROJ.

Institute for Development and Resources in Intensive Scientific Computing

Navigation du site

Resource Management

For Users

News

Table des matières

Jean Zay: Guide for the multi-project user