Benchmark for Gromacs 6vnx case

The system consists of a protein in a box of water:

• Number of atoms:
  • Total = 932,310
  • Solvent = 887,139
  • Protein = 45,156
  • Counter ions = 15
• Box:
  • cubic 21.14 Å

The simulation is run with Gromacs 2020.1 and has the following characteristics:

• Type of simulation: molecular dynamics
• Ensemble: NVT
• Number of steps: 500,000
• Timestep : 2 fs
• Long range electrostatic : Particle Mesh Ewald
• PME cutoff: 1.2 Å
• van der Waals cutoff: 1.1 Å
• cutoff-scheme: Verlet
• Temperature coupling: V-rescale
• Constraints: LINCS on H-bonds

The benchmark is performed using JUBE.

Gromacs has a rather large number of degrees of freedom when benchmarking. So it is plausible that a different fine-tuned configuration might give a better result. We tried to be as general as possible.

A first benchmark was performed on a small number of steps (50,000), then the best configurations obtained were re-run for 500,000 time steps to reduce the impact of the first slower steps.

The default number of PME/PP tasks is kept as default. Please note that no large imbalance occurred during the benchmarks. This is something you might want to check when doing your tests.

Here is an example of the mdrun command for the thread-MPI version of Gromacs:

gmx mdrun -ntmpi 40  -ntomp 1 \ 
          -dlb yes -update auto -bonded auto \
          -nb auto -pme auto -pmefft auto \
          -deffnm 6vxx_nvt -v \
          -nsteps 500000  -resetstep 300000

The GPU benchmark was done to check the best results for:

• 1 quarter of a node (1 GPU, 10 physical cores)
• 1 half node (2 GPUs, 20 physical cores)
• 1 node (4 GPUs, 40 physical cores)

The idea was to offload the maximum computation possible. So the following choices have been made:

• Offload PME: This means that the number of PME tasks has to be set to 1 (-npme 1) since PME on GPU does not support domain decomposition.
• For the thread-MPI version, the newest developments available since 2020 version are offloaded to GPU. Please read this page before doing a production run
  • Update of neighbors list
  • Bonded interactions

The thread-MPI version is limited to running on 1 node, meaning a maximum of 4 GPUs. This means that a large imbalance might occur. But this is still the best choice for performance.

Here is an example of the mdrun command for GPU thread-MPI version of Gromacs:

export GMX_GPU_PME_PP_COMMS=true
export GMX_FORCE_UPDATE_DEFAULT_GPU=1
export GMX_GPU_DD_COMMS=true

gmx mdrun -ntmpi 5 -npme 1 -ntomp 4 \
          -dlb yes -update gpu -bonded gpu \
          -nb gpu -pme gpu -pmefft gpu \
          -deffnm 6vxx_nvt -v \
          -nsteps 500000  -resetstep 300000

The performance is measured on the 200,000 last steps.

A few indicators such as the imbalance and the PME/F ratio are indicated.

The multithread column indicates if the hyperthreading was used during the computation.

Table of speedup:

When the performance is -1.0, it means that the run did not finish. Most of the time this is due to a segmentation fault in the program.

Some indicators such as the imbalance and the PME/F ratio are shown.

The multithread column indicates if hyperthreading was used during the computation.