Re: [slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

2019-09-03 Thread Chris Samuel
On Monday, 2 September 2019 11:02:57 AM PDT Ole Holm Nielsen wrote: > We have some users requesting that a certain minimum size of the > *Available* (i.e., free) TmpFS disk space should be present on nodes > before a job should be considered by the scheduler for a set of nodes. At Swinburne I did

[slurm-users] Can Slurm schedule around resources shared between nodes?

2019-09-03 Thread Conrad Herrmann
I know that Slurm can schedule jobs to efficiently use resources that are available on a single node by sharing those resources between multiple jobs. Examples include CPU codes, memory, and GPUs. I'm trying to figure out if Slurm can schedule jobs around limited resources that are shared by mult

Re: [slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

2019-09-03 Thread Juergen Salk
* Ole Holm Nielsen [190903 11:14]: > How do you dynamically update your gres=localtmp resource according to the > current disk free space? I mean, there is already a TmpFS disk space size > defined in slurm.conf, so how does your gres=localtmp differ from TmpFS? Dear Ole, I think (but please c

Re: [slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

2019-09-03 Thread Bjørn-Helge Mevik
Juergen Salk writes: > We are also going to implement disk quotas for the amount of local > scratch space that has been allocated for the job by means of generic > resources (e.g. `--gres=scratch:100´ for 100GB). This is especially > important when several users share a node. Indeed. > This lea

Re: [slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

2019-09-03 Thread Bjørn-Helge Mevik
Ole Holm Nielsen writes: > I figured that other sites need the free disk space feature as well > :-) :) > How do you dynamically update your gres=localtmp resource according to > the current disk free space? I mean, there is already a TmpFS disk > space size defined in slurm.conf, so how does

Re: [slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

2019-09-03 Thread Juergen Salk
Dear Bjørn-Helge, this is unfortunately no answer to the question but I'd be glad to hear some more thoughts on that, too. We are also going to implement disk quotas for the amount of local scratch space that has been allocated for the job by means of generic resources (e.g. `--gres=scratch:100´

Re: [slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

2019-09-03 Thread Ole Holm Nielsen
Hi Bjørn-Helge, I figured that other sites need the free disk space feature as well :-) How do you dynamically update your gres=localtmp resource according to the current disk free space? I mean, there is already a TmpFS disk space size defined in slurm.conf, so how does your gres=localtmp di

Re: [slurm-users] How can jobs request a minimum available (free) TmpFS disk space?

2019-09-03 Thread Bjørn-Helge Mevik
We are facing more or less the same problem. We have historically defined a Gres "localtmp" with the number of GB initially available on local disk, and then jobs ask for --gres=localtmp:50 or similar. That prevents slurm from allocating jobs on the cluster if they ask for more disk than is curre