Dear Paolo,
Thank you for your prompt response. Your suggestion was very helpful.
I have reviewed the numbers and discovered that regardless of the value set in
--gres=gpu:X, the number of ndev remains consistently at 1. Our HPC
documentation indicates that --gres=gpu:X is the correct method to set GPUs,
with each node having 4 GPUs. Here is the output when I set --gres=gpu:4:
GPU acceleration is ACTIVE.
GPU-aware MPI enabled
nproc (MPI process): 4
ndev (GPU per Node): 1
nnode (Nodes): 1
Message from routine print_cuda_info:
High GPU oversubscription detected. Are you sure this is what you want?
I monitored GPU core usage every 10 seconds, it appears that all 4 GPU cores
are activated when setting --gres=gpu:4:
utilization.gpu [%], utilization.memory [%]
96 %, 37 %
95 %, 76 %
95 %, 50 %
95 %, 76 %
time = 70 s
For reference, here is my sbatch submission script:
-----------------------------------------------------------------------
#!/bin/bash -x
#SBATCH --gres=gpu:4 --partition=dc-gpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --time=00:00:20
export OMP_NUM_THREADS=1
module load NVHPC/23.7-CUDA-12
module load CUDA/12
module load OpenMPI/4.1.5
module load mpi-settings/CUDA
module load imkl/2023.2.0
monitor_gpu_usage() {
while true; do
nvidia-smi --query-gpu=utilization.gpu,utilization.memory --format=csv
>> gpu_usage_$SLURM_JOB_ID.csv
sleep 10
done
}
monitor_gpu_usage &
srun -n 4 pw.x -nk 4 -nd 1 -nb 1 -nt 1 < inp_pwscf > out_pwscf
-------------------------------------------------------------------------
Could you please provide guidance on resolving the oversubscription issue?
Thank you very much in advance.
Kind regards,
Yin-Ying Ting
On 29.11.23 15:53, Paolo Giannozzi wrote:
On 11/27/23 11:32, Yin-Ying Ting wrote:
Based on the *environment.f90* file, this message is triggered when /nproc > ndev
* nnode * 2/. If I understand correctly, I have nproc (Number of parallel
processe)=4, ndev(Number of GPU Devices per Node) =4 and nnode (Number of Nodes)=1.
This condition seems to be false (4 > 8). Despite this, the message still appears.
All 4 GPUs were active during the run.
funny. Even funnier, the number of GPUs actually used does not seem to be
written anywhere on output.
Add a line printing nproc, ndev, nnode just before the warning is issued,
recompile and re-run. One (at least) of those numbers is not what you expect.
Computers are not among the most reliable machines, but they should be able to
find out who is larger between 4 and 8
Paolo
--
Forschungszentrum Jülich GmbH
Institute of Energy and Climate Research
Theory and Computation of Energy Materials (IEK-13)
E-mail: y.t...@fz-juelich.de<mailto:y.t...@fz-juelich.de>
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Jülich GmbH
52425 Jülich
Sitz der Gesellschaft: Jülich
Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Stefan Müller
Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende),
Karsten Beneke (stellv. Vorsitzender), Dr. Ir. Pieter Jansens
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
_______________________________________________
The Quantum ESPRESSO community stands by the Ukrainian
people and expresses its concerns about the devastating
effects that the Russian military offensive has on their
country and on the free and peaceful scientific, cultural,
and economic cooperation amongst peoples
_______________________________________________
Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
users mailing list users@lists.quantum-espresso.org
https://lists.quantum-espresso.org/mailman/listinfo/users
