Jean Zay : Memory measurement and the idrmem tools

On Jean Zay, the memory limit applies to the physical memory actually used by your program. The idrmem tools are put at your disposition to allow you to know the maximum consumption of physical memory during your program executions.

Two idrmem tools exist: One is destined for MPI codes (the MPI idrmem library) and the other for OpenMP or sequential codes (the idrmem command). These return the memory usage information at the end of the program execution.

The idrmem tools are available with the module command:

$ module avail idrmem
 
------- /gpfslocalsup/pub/modules-idris/modulefiles/linux-rhel7-x86_64 -------
idrmem/1.4        idrmem/1.4-mpi

The MPI idrmem library allows the displaying of the maximum consumptions of virtual and physical memory used by an MPI code throughout its execution. Note that the display is shown at the moment the program executes one of the following instructions: MPI_Finalize or MPI_Abort.

It is necessary to load the corresponding idrem module file to compile your MPI program:

$ module load idrmem/1.4-mpi
$ mpiifort main.f90 -o main -lidrmem

However, no change is necessary in the submission script (no module to load and no command to execute).

At the end of the execution, you will obtain the following information on the standard output: the elapsed time, the maximum consumption of virtual memory (VMSizeMax), the maximum consumption of physical memory (RSSMax) and the maximum consumption of the stack (StackMax), as well as the ranks of the MPI processes on which the maximums were reached.

IdrisMemMPI v1.4 : elapsed = 0.32 s, VMSizeMax = 497 mB on rank 1, RSSMax = 28 mB on rank 0, StackMax = 408 kB on rank 1

If you wish the current consumption at a precise moment to be displayed during the execution of an MPI program, you simply insert a call to the idr_print_mem routine provided with the MPI idrmem library:

  CALL idr_print_mem()

• In MPMD mode, all of the codes must be compiled with the MPI idrmem library.
• If for any reason the program does not execute the MPI_Finalize instruction or the MPI_Abort instruction, no information is returned.

In this case, the idrmem command allows displaying the maximum consumptions of virtual and physical memory. To use this with a sequential or OpenMP code, you just need to load the corresponding module and the idrmem command to execute your program:

$ module load idrmem/1.4
$ idrmem ./executable_seq

Note that the idrmem command can also be used with MPI codes even though, of course, the MPI idrmem library is preferable. In this case you should do as follows:

$ module load idrmem/1.4
$ srun idrmem ./executable_mpi

The idrmem command displays in the standard output: the elapsed time, the maximum consumption of virtual memory (VMSizeMax), the maximum consumption of physical memory (RSSMax) and the maximum consumption of the stack (StackMax):

IdrisMem v1.4 : elapsed = 24.61 s, VMSizeMax = 1257 MB, RSSMax = 791 MB, StackMax = 92 kB

To effectuate and display measurements during the execution of OpenMP or sequential codes, we suggest that you insert the procstat procedure into your code (see below).

• Does not work with an alias command.
• Does not give good results if the command is a script or an executable file which launches other executable files.
• If the execution time (#SBATCH –time=HH:MM:SS) is reached, there will not be a display.
• Does not stop immediately when there is a call to MPI_Abort. This is the reason it is preferable to use the MPI idrmem library version for MPI codes.

If you wish to be able to display the maximum memory consumption at any moment during the execution, here is a Fortran procedure which you can add to your code:

subroutine_procstat

subroutine procstat()
  character(len=128) :: key, mot1, mot2
  integer :: ios, iter, My_unit
  ! Ouverture du fichier
  OPEN ( NEWUNIT = My_unit ,    FILE     = '/proc/self/status', &
         FORM   = 'formatted' , ACCESS   = 'sequential',        &
         ACTION = 'read' ,      POSITION = 'rewind' ,&          
         IOSTAT = ios )                                    
  IF ( ios /= 0 ) stop " Probleme a l'ouverture "
 
  ! Lecture du nom de l'executable (pour verification)
  READ (UNIT =My_unit , FMT=*, IOSTAT = ios ) key, mot1
  DO WHILE (trim(key) /= "VmPeak:")
    READ (UNIT =My_unit , FMT=*, IOSTAT = ios ) key, mot1,mot2
  END DO
  print *, "procstat : VMSizeMax = ", trim(mot1), trim(mot2)
  DO WHILE (trim(key) /= "VmHWM:")
    READ (UNIT =My_unit , FMT=*, IOSTAT = ios ) key, mot1, mot2
  END DO
  print *, "procstat : RSSMax = ", trim(mot1), trim(mot2)
  DO WHILE (trim(key) /= "VmStk:")
    READ (UNIT =My_unit , FMT=*, IOSTAT = ios ) key, mot1, mot2
  END DO
  print *, "procstat : StackMax = ", trim(mot1), trim(mot2)
  CLOSE ( UNIT =My_unit )
end subroutine procstat

Subsequently, a call procstat() in your code will provoke the display in the standard output of the maximum consumption of the virtual memory, physical memory and the stack since the launching of the program to the moment of your call.