Thanks for the info!
Didn't know Slurm supported to set a soft limit. Did have a look in the
code and found function "memcg_initialize" in file
"plugins/task/cgroup/task_cgroup_memory.c".
It looks like in order to set both hard and soft limits you need to do:
ConstrainRAMSpace=yes
AllowedRAMSpace=200
This should allow jobs to use 200% the requested memory. So if a job
request --mem=10G then it will be able to use 10G RAM but will not be
killed unless it uses 20G (if there are free RAM available), but it will
also consume all swap first.
After some testing, this works just fine if I have ConstrainSwapSpace=no
but if I change that to yes I get strange behaviours. At least the
output from sacct is not what I expect it to be. I'm running the exact
same test program (just trying to get as much memory as possible) but in
the second run I have ConstrainSwapSpace=yes. Why is it not recording
the correct(?) amount of memory used?
ConstrainRAMSpace=yes
AllowedRAMSpace=150
JobID State ExitCode MaxRSS
MaxVMSize AllocTRES
- -- -- --
911124 OUT_OF_ME+ 0:125 billing=1,cpu=1,mem=2G,node=1
911124.extern COMPLETED 0:0 1104K 171896K
billing=1,cpu=1,mem=2G,node=1
911124.0 OUT_OF_ME+ 0:125 3141964K 14340268K
cpu=1,mem=2G,node=1
ConstrainRAMSpace=yes
AllowedRAMSpace=150
ConstrainSwapSpace=yes
AllowedSwapSpace=200
JobID State ExitCode MaxRSS
MaxVMSize AllocTRES
- -- -- --
911081 OUT_OF_ME+ 0:125 billing=1,cpu=1,mem=2G,node=1
911081.extern COMPLETED 0:0 1096K 171896K
billing=1,cpu=1,mem=2G,node=1
911081.0 OUT_OF_ME+ 0:125 1216K 240648K
cpu=1,mem=2G,node=1
Memory is complicated stuff...
Regards,
Alexander
On 03/05/2019 08:20, Janne Blomqvist wrote:
On 02/05/2019 17.53, Alexander Åhman wrote:
Hi,
Is it possible to configure slurm/cgroups in such way that jobs that are
using more memory than they asked for are not killed if there still are
free memory available on the compute node? When free memory gets low
these jobs can be killed as usual.
Today when a job has exceeded its limits it is killed immediately. Since
the applications only requires maximum memory for a short period of time
we can often not run as many concurrent jobs as we want.
Maybe I can rephrase the question a bit: How can you configure memory
limits for a job when the job only needs maximum memory during a short
time? Example: Job1 needs 80G RAM but only during 15% of the execution
time, during the remaining 85% it only needs 30G.
I guess the obvious thing is to use "CR_Core" instead of
"CR_Core_Memory" we use today. But we have to constrain memory in some
way because the nodes are also running daemons for the distributed file
system and that must not be affected by running jobs.
Any ideas?
Regards,
Alexander
It's possible by setting the cgroup soft and hard limits appropriately.
See https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt for
details.
Unfortunately the way the slurm cgroup.conf configuration maps to the
actual cgroups is not exactly intuitive (one can hope that when
eventually slurm starts to support cgroupsv2 this would be fixed...).
For example, if you want to set the cgroup soft limit to the amount of
memory that the job requests with --mem= or --mem-per-core= options, and
then set the hard limit (when the job will be killed) to 5x the soft limit:
ConstrainRAMSpace=no
ConstrainSwapSpace=yes
AllowedSwapSpace=400
(Note that it's not possible to separately set the maximum swap usage.
If you instead limit only the memory and not mem+swap, it will limit
memory but swap usage will be unlimited.)
As for you second part of the question, no, it's not possible to have
varying limits during a job.
