Hi Bjørn-Helge,
hi everone,
ok, I see. I also just re-read the documentation to find this in the
description of the "CPUs" option:
"This can be useful when you want to schedule only the cores on a
hyper-threaded node. If CPUs is omitted, its default will be set equal
to the product of Boards, Sockets, CoresPerSocket, and ThreadsPerCore."
Do you happen to know if there is a difference between setting CPUs
explicitely like you do it and not setting it but using "ThreadsPerCore=1"?
My guess is that there is no difference and in both cases only the
physical cores are "handed out to jobs". But maybe I am wrong?
Regards,
Hermann
On 3/25/22 8:49 AM, Bjørn-Helge Mevik wrote:
For what it's worth, we have a similar setup, with one crucial
difference: we are handing out physical cores to jobs, not hyperthreads,
and we are *not* seeing this behaviour:
$ srun --cpus-per-task=1 -t 10 --mem-per-cpu=1g -A nn9999k -q devel echo foo
srun: job 5371678 queued and waiting for resources
srun: job 5371678 has been allocated resources
foo
$ srun --cpus-per-task=3 -t 10 --mem-per-cpu=1g -A nn9999k -q devel echo foo
srun: job 5371680 queued and waiting for resources
srun: job 5371680 has been allocated resources
foo
We have
SelectType=select/cons_tres
SelectTypeParameters=CR_CPU_Memory
and node definitions like
NodeName=DEFAULT CPUs=40 Sockets=2 CoresPerSocket=20 ThreadsPerCore=2
RealMemory=182784 Gres=localscratch:330G Weight=1000
(so we set CPUs to the number of *physical cores*, not *hyperthreads*).
