Re: [slurm-users] unable to run on all the logical cores

[David Bellot]

Indeed, it makes sense now. However, if I launch many R processes using the
"parallel" package, I can easily have all the "logical" cores running. In
the background, if I'm correct ,R will "fork" and not create a thread. So
we have independent processes. On a 20 cores CPU for example, I have 40
"logical" cores and all the cores are running, according to htop.

With Slurm, I can't reproduce the same behavior even if I use the
SelectTypeParameters=CR_CPU.

So, is there a config to tune, an option to use in "sbatch" to achieve the
same result, or should I rather launch 20 jobs per node and have each job
split in two internally (using "parallel" or "future" for example)?

On Thu, Oct 8, 2020 at 6:32 PM William Brown 
wrote:

> R is single threaded.
>
> On Thu, 8 Oct 2020, 07:44 Diego Zuccato,  wrote:
>
>> Il 08/10/20 08:19, David Bellot ha scritto:
>>
>> > good spot. At least, scontrol show job is now saying that each job only
>> > requires one "CPU", so it seems all the cores are treated the same way
>> now.
>> > Though I still have the problem of not using more than half the cores.
>> > So I suppose it might be due to the way I submit (batchtools in this
>> > case) the jobs.
>> Maybe R is generating single-threaded code? In that case, only a single
>> process can run on a given core at a time (processes does not share
>> memory map, threads do, and on Intel CPUs there's a single MMU per core,
>> not one per thread as in some AMDs).
>>
>> --
>> Diego Zuccato
>> DIFA - Dip. di Fisica e Astronomia
>> Servizi Informatici
>> Alma Mater Studiorum - Università di Bologna
>> V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>> tel.: +39 051 20 95786
>>
>>

-- 

David Bellot
Head of Quantitative Research

A. Suite B, Level 3A, 43-45 East Esplanade, Manly, NSW 2095
E. david.bel...@lifetrading.com.au
P. (+61) 0405 263012

Re: [slurm-users] unable to run on all the logical cores

[Chris Samuel]

On 10/7/20 10:13 pm, David Bellot wrote:

NodeName=foobar01 CPUs=80 Boards=1 SocketsPerBoard=2 CoresPerSocket=20 
ThreadsPerCore=2 RealMemory=257243 State=UNKNOWN


With this configuration Slurm is allocation a single physical core (with 
2 thread units) per task. So you are using all (physical) cores.


However, if what you want is to have 1 process per thread unit (not 
necessarily a good idea, depending on how your code works) then I think 
you'd need to adjust your config to lie to Slurm and tell it it's got 40 
cores per socket and 1 thread per core instead.


All the best,
Chris
--
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA

Re: [slurm-users] unable to run on all the logical cores

[William Brown]

Using Parallel or similar will be the easiest and most efficient, but does 
require that you have control of the R code.  There are different approaches 
for work within a single node or across many nodes, with the latter having a 
set-up and tear-down cost so the workload within the loop must be worthwhile.

 

I think if you configure Slurm to misrepresent the hardware you might find odd 
things happen elsewhere.

 

We always disable hyper-threading on the compute nodes, at the recommendation 
of the company that installed it, who reported that it ran faster that way.  
Multi-threading has a cost and is not ideal for compute workloads.  It is 
better for things like web servers where some tasks are waiting IO or user 
input.  That said, it likely depends on the CPU and what is true for Intel 
might not be true for AMD.

 

 

 

From: slurm-users  On Behalf Of David 
Bellot
Sent: 12 October 2020 01:14
To: Slurm User Community List 
Subject: Re: [slurm-users] unable to run on all the logical cores

 

Indeed, it makes sense now. However, if I launch many R processes using the 
"parallel" package, I can easily have all the "logical" cores running. In the 
background, if I'm correct ,R will "fork" and not create a thread. So we have 
independent processes. On a 20 cores CPU for example, I have 40 "logical" cores 
and all the cores are running, according to htop.

 

With Slurm, I can't reproduce the same behavior even if I use the 
SelectTypeParameters=CR_CPU.

 

So, is there a config to tune, an option to use in "sbatch" to achieve the same 
result, or should I rather launch 20 jobs per node and have each job split in 
two internally (using "parallel" or "future" for example)?

 

On Thu, Oct 8, 2020 at 6:32 PM William Brown mailto:will...@signalbox.org.uk> > wrote:

R is single threaded.

 

On Thu, 8 Oct 2020, 07:44 Diego Zuccato, mailto:diego.zucc...@unibo.it> > wrote:

Il 08/10/20 08:19, David Bellot ha scritto:

> good spot. At least, scontrol show job is now saying that each job only
> requires one "CPU", so it seems all the cores are treated the same way now.
> Though I still have the problem of not using more than half the cores.
> So I suppose it might be due to the way I submit (batchtools in this
> case) the jobs.
Maybe R is generating single-threaded code? In that case, only a single
process can run on a given core at a time (processes does not share
memory map, threads do, and on Intel CPUs there's a single MMU per core,
not one per thread as in some AMDs).

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786




 

-- 



  




David Bellot



Head of Quantitative Research




 




A. Suite B, Level 3A, 43-45 East Esplanade, Manly, NSW 2095



E.   david.bel...@lifetrading.com.au





P. (+61) 0405 263012