Re: [slurm-users] How to limit # of execution slots for a given node
David, There are several possible answers depending on what you hope to accomplish. What exactly is the issue that you're trying to solve? Do you mean that you have users who need, say, 8 GB of RAM per core but you only have 4 GB of RAM per core on the system and you want a way to account for that? Or is it something else? Ryan On 1/6/22 14:39, David Henkemeyer wrote: All, When my team used PBS, we had several nodes that had a TON of CPUs, so many, in fact, that we ended up setting np to a smaller value, in order to not starve the system of memory. What is the best way to do this with Slurm? I tried modifying # of CPUs in the slurm.conf file, but I noticed that Slurm enforces that "CPUs" is equal to Boards * SocketsPerBoard * CoresPerSocket * ThreadsPerCore. This left me with having to "fool" Slurm into thinking there were either fewer ThreadsPerCore, fewer CoresPerSocket, or fewer SocketsPerBoard. This is a less than ideal solution, it seems to me. At least, it left me feeling like there has to be a better way. Thanks! David
Re: [slurm-users] How to limit # of execution slots for a given node
Also I recommend setting: *CoreSpecCount* Number of cores reserved for system use. These cores will not be available for allocation to user jobs. Depending upon the *TaskPluginParam* option of *SlurmdOffSpec*, Slurm daemons (i.e. slurmd and slurmstepd) may either be confined to these resources (the default) or prevented from using these resources. Isolation of the Slurm daemons from user jobs may improve application performance. If this option and *CpuSpecList* are both designated for a node, an error is generated. For information on the algorithm used by Slurm to select the cores refer to the core specialization documentation ( https://slurm.schedmd.com/core_spec.html ). and *MemSpecLimit* Amount of memory, in megabytes, reserved for system use and not available for user allocations. If the task/cgroup plugin is configured and that plugin constrains memory allocations (i.e. *TaskPlugin=task/cgroup* in slurm.conf, plus *ConstrainRAMSpace=yes* in cgroup.conf), then Slurm compute node daemons (slurmd plus slurmstepd) will be allocated the specified memory limit. Note that having the Memory set in *SelectTypeParameters* as any of the options that has it as a consumable resource is needed for this option to work. The daemons will not be killed if they exhaust the memory allocation (ie. the Out-Of-Memory Killer is disabled for the daemon's memory cgroup). If the task/cgroup plugin is not configured, the specified memory will only be unavailable for user allocations. These will restrict specific memory and cores for system use. This is probably the best way to go rather than spoofing your config. -Paul Edmon- On 1/7/2022 2:36 AM, Rémi Palancher wrote: Le jeudi 6 janvier 2022 à 22:39, David Henkemeyer a écrit : All, When my team used PBS, we had several nodes that had a TON of CPUs, so many, in fact, that we ended up setting np to a smaller value, in order to not starve the system of memory. What is the best way to do this with Slurm? I tried modifying # of CPUs in the slurm.conf file, but I noticed that Slurm enforces that "CPUs" is equal to Boards * SocketsPerBoard * CoresPerSocket * ThreadsPerCore. This left me with having to "fool" Slurm into thinking there were either fewer ThreadsPerCore, fewer CoresPerSocket, or fewer SocketsPerBoard. This is a less than ideal solution, it seems to me. At least, it left me feeling like there has to be a better way. I'm not sure you can lie to Slurm about the real number of CPUs on the nodes. If you want to prevent Slurm from allocating more than n CPUs below the total number of CPUs of these nodes, I guess one solution is to use MaxCPUsPerNode=n at the partition level. You can also mask "system" CPUs with CpuSpecList at node level. The later is better if you need fine grain control over the exact list of reserved CPUs regarding NUMA topology or whatever. -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io
Re: [slurm-users] How to limit # of execution slots for a given node
You can actually spoof the number of cores and RAM on a node by using the config_override option. I've used that before for testing purposes. Mind you core binding and other features like that will not work if you start spoofing the number of cores and ram, so use with caution. -Paul Edmon- On 1/7/2022 2:36 AM, Rémi Palancher wrote: Le jeudi 6 janvier 2022 à 22:39, David Henkemeyer a écrit : All, When my team used PBS, we had several nodes that had a TON of CPUs, so many, in fact, that we ended up setting np to a smaller value, in order to not starve the system of memory. What is the best way to do this with Slurm? I tried modifying # of CPUs in the slurm.conf file, but I noticed that Slurm enforces that "CPUs" is equal to Boards * SocketsPerBoard * CoresPerSocket * ThreadsPerCore. This left me with having to "fool" Slurm into thinking there were either fewer ThreadsPerCore, fewer CoresPerSocket, or fewer SocketsPerBoard. This is a less than ideal solution, it seems to me. At least, it left me feeling like there has to be a better way. I'm not sure you can lie to Slurm about the real number of CPUs on the nodes. If you want to prevent Slurm from allocating more than n CPUs below the total number of CPUs of these nodes, I guess one solution is to use MaxCPUsPerNode=n at the partition level. You can also mask "system" CPUs with CpuSpecList at node level. The later is better if you need fine grain control over the exact list of reserved CPUs regarding NUMA topology or whatever. -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io
Re: [slurm-users] How to limit # of execution slots for a given node
Le jeudi 6 janvier 2022 à 22:39, David Henkemeyer a écrit : > All, > > When my team used PBS, we had several nodes that had a TON of CPUs, so many, > in fact, that we ended up setting np to a smaller value, in order to not > starve the system of memory. > > What is the best way to do this with Slurm? I tried modifying # of CPUs in > the slurm.conf file, but I noticed that Slurm enforces that "CPUs" is equal > to Boards * SocketsPerBoard * CoresPerSocket * ThreadsPerCore. This left me > with having to "fool" Slurm into thinking there were either fewer > ThreadsPerCore, fewer CoresPerSocket, or fewer SocketsPerBoard. This is a > less than ideal solution, it seems to me. At least, it left me feeling like > there has to be a better way. I'm not sure you can lie to Slurm about the real number of CPUs on the nodes. If you want to prevent Slurm from allocating more than n CPUs below the total number of CPUs of these nodes, I guess one solution is to use MaxCPUsPerNode=n at the partition level. You can also mask "system" CPUs with CpuSpecList at node level. The later is better if you need fine grain control over the exact list of reserved CPUs regarding NUMA topology or whatever. -- Rémi Palancher Rackslab: Open Source Solutions for HPC Operations https://rackslab.io
Re: [slurm-users] How to limit # of execution slots for a given node
Hi David, On 1/6/22 22:39, David Henkemeyer wrote: When my team used PBS, we had several nodes that had a TON of CPUs, so many, in fact, that we ended up setting np to a smaller value, in order to not starve the system of memory. What is the best way to do this with Slurm? I tried modifying # of CPUs in the slurm.conf file, but I noticed that Slurm enforces that "CPUs" is equal to Boards * SocketsPerBoard * CoresPerSocket * ThreadsPerCore. This left me with having to "fool" Slurm into thinking there were either fewer ThreadsPerCore, fewer CoresPerSocket, or fewer SocketsPerBoard. This is a less than ideal solution, it seems to me. At least, it left me feeling like there has to be a better way. If your goal is to limit the amount of RAM memory per job, then kernel Cgroups is probably the answer. I've collected some information in my Wiki page: https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#cgroup-configuration If some users need more RAM than available for 1 core, they have to submit jobs for a larger number of cores to get it. This makes a lot of sense, IMHO. SchedMD is working on the use of Cgroups v2, see the talk "Slurm 21.08 and Beyond" by Tim Wickberg, SchedMD, https://slurm.schedmd.com/publications.html You could probably "fool" Slurm as you describe it, but that shouldn't be necessary. I hope this helps. /Ole
[slurm-users] How to limit # of execution slots for a given node
All, When my team used PBS, we had several nodes that had a TON of CPUs, so many, in fact, that we ended up setting np to a smaller value, in order to not starve the system of memory. What is the best way to do this with Slurm? I tried modifying # of CPUs in the slurm.conf file, but I noticed that Slurm enforces that "CPUs" is equal to Boards * SocketsPerBoard * CoresPerSocket * ThreadsPerCore. This left me with having to "fool" Slurm into thinking there were either fewer ThreadsPerCore, fewer CoresPerSocket, or fewer SocketsPerBoard. This is a less than ideal solution, it seems to me. At least, it left me feeling like there has to be a better way. Thanks! David
