[slurm-users] Re: [External] Is there a way to list allocated/unallocated resources defined in a QoS?
Hi Alistair,
I was holding off replying in the hope someone would have a good answer. In
lieu of that, here’s my partial answer:
When I looked at trying to report per-user and per-group qos values a few
months I discovered that SLURM reports the information via this command:
scontrol -o show assoc_mgr flags=qos
I haven’t found any documentation explaining the format of that output. It
seems to be parsable, but I’m not sure if the format will change in later
version of SLURM. I’m using perl regexp’s for the reporting I’m doing, but
here’s a grep-based example to extract per-group limits of cpus which works on
my setup:
scontrol -o show assoc_mgr flags=qos|grep QOS=dept1|grep -o 'GrpTRES=[^
]*'|grep -o 'cpu=[0-9]*'
That information is available to all SLURM users. But given the different
contexts a qos can be used in, I’m not sure how you might be able to limit
reporting only to users who are permitted to use a specific qos.
And for completeness, here’s a similar method for extracting per-user gpu
limits:
scontrol -o show assoc_mgr flags=qos|grep QOS=myqosname|grep -o
'myusernamel([0-9]*)={[^}]*}'|grep -o 'MaxTRESPU=[^ ]*'|grep -o
'gres/gpu=[0-9]*([0-9]*)'
Regards,
Mike
From: Alastair Neil via slurm-users
Sent: Tuesday, February 6, 2024 11:30 PM
To: slurm-us...@schedmd.com
Subject: [External] [slurm-users] Is there a way to list allocated/unallocated
resources defined in a QoS?
This email originated outside the University. Check before clicking links or
attachments.
Slurm version 23.02.07
If I have a QoS defined that has a set number of say GPU devices set in the
GrpTRES. Is there an easy way to generate a list of how much of the defined
quota is allocated or conversely un-allocated?
e.g.:
Name|Priority|GraceTime|Preempt|PreemptExemptTime|PreemptMode|Flags|UsageThres|UsageFactor|GrpTRES|GrpTRESMins|GrpTRESRunMins|GrpJobs|GrpSubmit|GrpWall|MaxTRES|MaxTRESPerNode|MaxTRESMins|MaxWall|MaxTRESPU|MaxJobsPU|MaxSubmitPU|MaxTRESPA|MaxJobsPA|MaxSubmitPA|MinTRES|
normal|0|00:00:00|||cluster|||1.00|||cpu=3000,gres/gpu=20|||
dept1|1|00:00:00|||cluster|||1.00|cpu=256,gres/gpu:1g.10gb=16,gres/gpu:2g.20gb=8,gres/gpu:3g.40gb=8,gres/gpu:a100.80gb=8|
dept2|1|00:00:00|||cluster|||1.00|cpu=256,gres/gpu:1g.10gb=0,gres/gpu:2g.20gb=0,gres/gpu:3g.40gb=0,gres/gpu:a100.80gb=16|
So dept1 and dept2 qos are set on the same partition. How can a user with
access to one or other see if there are available resources in the partition?
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
Re: [slurm-users] [External] Re: Troubleshooting job stuck in Pending state
Hi Davide, The jobs do eventually run, but can take several minutes or sometimes several hours to switch to a running state even when there’s plenty of resources free immediately. With Grid Engine it was possible to turn on scheduling diagnostics and get a summary of the scheduler’s decisions on a pending job by running “qstat -j jobid”. But there doesn’t seem to be any functional equivalent with SLURM? Regards, Mike From: slurm-users On Behalf Of Davide DelVento Sent: Monday, December 11, 2023 4:23 PM To: Slurm User Community List Subject: [External] Re: [slurm-users] Troubleshooting job stuck in Pending state This email originated outside the University. Check before clicking links or attachments. By getting "stuck" do you mean the job stays PENDING forever or does eventually run? I've seen the latter (and I agree with you that I wish Slurm will log things like "I looked at this job and I am not starting it yet because") but not the former On Fri, Dec 8, 2023 at 9:00 AM Pacey, Mike mailto:m.pa...@lancaster.ac.uk>> wrote: Hi folks, I’m looking for some advice on how to troubleshoot jobs we occasionally see on our cluster that are stuck in a pending state despite sufficient matching resources being free. In the case I’m trying to troubleshoot the Reason field lists (Priority) but to find any way to get the scheduler to tell me what exactly is the priority job blocking. * I tried setting the scheduler log level to debug3 for 5 minutes at one point, but my logfile ballooned from 0.5G to 1.5G and didn’t offer any useful info for this case. * I’ve tried ‘scontrol schedloglevel 1’ but it returns the error: ‘slurm_set_schedlog_level error: Requested operation is presently disabled’ I’m aware that the backfill scheduler will occasionally hold on to free resources in order to schedule a larger job with higher priority, but in this case I can’t find any pending job that might fit the bill. And to possibly complicate matters, this is on a large partition that has no maximum time limit and most pending jobs have no time limits either. (We use backfill/fairshare as we have smaller partitions of rarer resources that benefit from it, plus we’re aiming to use fairshare even on the no-time-limits partitions to help balance out usage). Hoping someone can provide pointers. Regards, Mike
[slurm-users] Troubleshooting job stuck in Pending state
Hi folks, I'm looking for some advice on how to troubleshoot jobs we occasionally see on our cluster that are stuck in a pending state despite sufficient matching resources being free. In the case I'm trying to troubleshoot the Reason field lists (Priority) but to find any way to get the scheduler to tell me what exactly is the priority job blocking. * I tried setting the scheduler log level to debug3 for 5 minutes at one point, but my logfile ballooned from 0.5G to 1.5G and didn't offer any useful info for this case. * I've tried 'scontrol schedloglevel 1' but it returns the error: 'slurm_set_schedlog_level error: Requested operation is presently disabled' I'm aware that the backfill scheduler will occasionally hold on to free resources in order to schedule a larger job with higher priority, but in this case I can't find any pending job that might fit the bill. And to possibly complicate matters, this is on a large partition that has no maximum time limit and most pending jobs have no time limits either. (We use backfill/fairshare as we have smaller partitions of rarer resources that benefit from it, plus we're aiming to use fairshare even on the no-time-limits partitions to help balance out usage). Hoping someone can provide pointers. Regards, Mike
Re: [slurm-users] [External] Re: Granular or dynamic control of partitions?
Hi Feng, Thanks - that's what I was looking for, though for my version of SLURM (23.02.0) it looks like the syntax is "scontrol update partition=mypart". Good to know that SLURM can cope with on-the-fly changes without affecting jobs. With the "live" config now being different from the static I guess best practice is to ensure slurm.conf's partition definitions also need to be edited? Regards, Mike -Original Message- From: slurm-users On Behalf Of Feng Zhang Sent: Friday, August 4, 2023 7:36 PM To: Slurm User Community List Subject: [External] Re: [slurm-users] Granular or dynamic control of partitions? This email originated outside the University. Check before clicking links or attachments. You can try command as: scontrol update partition mypart Nodes=node[1-90],ab,ac #exclude the one you want to remove "Changing the Nodes in a partition has no effect upon jobs that have already begun execution." Best, Feng On Fri, Aug 4, 2023 at 10:47 AM Pacey, Mike wrote: > > Hi folks, > > > > We’re currently moving our cluster from Grid Engine to SLURM, and I’m having > trouble finding the best way to perform a specific bit of partition > maintenance. I’m not sure if I’m simply missing something in the manual or if > I need to be thinking in a more SLURM-centric way. My basic question: is it > possible to ‘disable’ specific partition/node combinations rather than whole > nodes or whole partitions? Here’s an example of the sort of thing I’m looking > to do: > > > > I have node ‘node1’ with two partitions ‘x’ and ‘y’. I’d like to remove > partition ‘y’, but there are currently user jobs in that partition on that > node. With Grid Engine, I could disable specific queue instances (ie, I could > just run “qmod -d y@node1’ to disable queue/partition y on node1 and wait for > the jobs to complete and then remove the partition. That would be the least > disruptive option because: > > Queue/partition ‘y’ on other nodes would be unaffected User jobs for > queue/partition ‘x’ would still be able to launch on node1 the whole > time > > > > I can’t seem to find a functional equivalent of this in SLURM: > > I can set the whole node to Drain > I can set the whole partition to Inactive > > > > Is there some way to ‘disable’ partition y just on node1? > > > > Regards, > > Mike
[slurm-users] Granular or dynamic control of partitions?
Hi folks, We're currently moving our cluster from Grid Engine to SLURM, and I'm having trouble finding the best way to perform a specific bit of partition maintenance. I'm not sure if I'm simply missing something in the manual or if I need to be thinking in a more SLURM-centric way. My basic question: is it possible to 'disable' specific partition/node combinations rather than whole nodes or whole partitions? Here's an example of the sort of thing I'm looking to do: I have node 'node1' with two partitions 'x' and 'y'. I'd like to remove partition 'y', but there are currently user jobs in that partition on that node. With Grid Engine, I could disable specific queue instances (ie, I could just run "qmod -d y@node1' to disable queue/partition y on node1 and wait for the jobs to complete and then remove the partition. That would be the least disruptive option because: * Queue/partition 'y' on other nodes would be unaffected * User jobs for queue/partition 'x' would still be able to launch on node1 the whole time I can't seem to find a functional equivalent of this in SLURM: * I can set the whole node to Drain * I can set the whole partition to Inactive Is there some way to 'disable' partition y just on node1? Regards, Mike
Re: [slurm-users] [External] Re: Per-user TRES summary?
Hi Ole, On my system that doesn't show me any MaxTRESPU info, which is how I've implemented user limits. E.g.: % showuserlimits -q normal scontrol -o show assoc_mgr users=pacey account=local qos=normal flags=QOS Slurm share information: AccountUser RawShares NormSharesRawUsage EffectvUsage FairShare -- -- --- --- - -- local pacey 1 0 0.00 0.00 However, using the oneliner (-o) version of the scontrol output from below it was only a few minutes work to write a small script to extract the info I needed. Regards, Mike -Original Message- From: slurm-users On Behalf Of Ole Holm Nielsen Sent: 29 November 2022 15:25 To: Slurm User Community List Subject: Re: [slurm-users] [External] Re: Per-user TRES summary? Hi Mike, That sounds great! It seems to me that "showuserlimits -q " would also print the QOS information, but maybe this is not what you are after? Have you tried this -q option, or should the script perhaps be generalized to cover your needs? /Ole On 29-11-2022 14:39, Pacey, Mike wrote: > Hi Ole (and Jeffrey), > > Thanks for the pointer - those are some very useful scripts. I couldn't get > showslurmlimits or showslurmjobs to get quite what I was after (it wasn't > showing me memory usage). However, it pointed me in the right direction - the > scontrol command. I can run the following: > > scontrol show assoc_mgr flags=qos > > and part of the output reads: > > User Limits >[myuid] > MaxJobsPU=N(2) MaxJobsAccruePU=N(0) MaxSubmitJobsPU=N(2) > > MaxTRESPU=cpu=80(2),mem=327680(1000),energy=N(0),node=N(1),billing=N(2 > ),fs/disk=N(0),vmem=N(0),pages=N(0) > > Which is exactly what I'm looking for. The values outside the brackets are > the qos limit, and the values within are the current usage. > > Regards, > Mike > > -Original Message- > From: slurm-users On Behalf Of > Ole Holm Nielsen > Sent: 28 November 2022 18:58 > To: slurm-users@lists.schedmd.com > Subject: [External] Re: [slurm-users] Per-user TRES summary? > > This email originated outside the University. Check before clicking links or > attachments. > > Hi Mike, > > Would the "showuserlimits" tool give you the desired information? > Check out > https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith > ub.com%2FOleHolmNielsen%2FSlurm_tools%2Ftree%2Fmaster%2Fshowuserlimits > data=05%7C01%7Cpacey%40live.lancs.ac.uk%7C72d39cac43214bffb36008d > ad22110fa%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C638053336698158 > 268%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi > I6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=iuhXkuGAu2oOKLqLI62 > Og5JwgLJ218ZBJuznIXWz1UA%3Dreserved=0 > > /Ole > > > On 28-11-2022 16:16, Pacey, Mike wrote: >> Does anyone have suggestions as to how to produce a summary of a >> user's TRES resources for running jobs? I'd like to able to see how >> each user is fairing against their qos resource limits. (I'm looking >> for something functionally equivalent to Grid Engine's qquota >> command). The info must be in the scheduler somewhere in order for it >> to enforce qos TRES limits, but as a SLURM novice I've not found any way to >> do this. >> >> To summarise TRES qos limits I can do this: >> >> % sacctmgr list qos format=Name,MaxTRESPerUser%50 >> >> Name MaxTRESPU >> >> -- -- >> >> normalcpu=80,mem=320G >> >> But to work out what a user is currently using in currently running >> jobs, the nearest I can work out is: >> >> % sacct -X -s R --units=G -o User,ReqTRES%50 >> >>UserReqTRES >> >> - -- >> >> pacey billing=1,cpu=1,mem=0.49G,node=1 >> >> pacey billing=1,cpu=1,mem=0.49G,node=1 >> >> With a little scripting I can sum those up, but there might be a >> neater way to do this? >
Re: [slurm-users] [External] Re: Per-user TRES summary?
Hi Ole (and Jeffrey), Thanks for the pointer - those are some very useful scripts. I couldn't get showslurmlimits or showslurmjobs to get quite what I was after (it wasn't showing me memory usage). However, it pointed me in the right direction - the scontrol command. I can run the following: scontrol show assoc_mgr flags=qos and part of the output reads: User Limits [myuid] MaxJobsPU=N(2) MaxJobsAccruePU=N(0) MaxSubmitJobsPU=N(2) MaxTRESPU=cpu=80(2),mem=327680(1000),energy=N(0),node=N(1),billing=N(2),fs/disk=N(0),vmem=N(0),pages=N(0) Which is exactly what I'm looking for. The values outside the brackets are the qos limit, and the values within are the current usage. Regards, Mike -Original Message- From: slurm-users On Behalf Of Ole Holm Nielsen Sent: 28 November 2022 18:58 To: slurm-users@lists.schedmd.com Subject: [External] Re: [slurm-users] Per-user TRES summary? This email originated outside the University. Check before clicking links or attachments. Hi Mike, Would the "showuserlimits" tool give you the desired information? Check out https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOleHolmNielsen%2FSlurm_tools%2Ftree%2Fmaster%2Fshowuserlimitsdata=05%7C01%7Cpacey%40live.lancs.ac.uk%7Cbea74c16c0b34468c68908dad174dacc%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C638052597059366026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=9pKyND95SW9Z1E%2BjGPsGKUKwTZIqj3juwWGQ4d5AWRw%3Dreserved=0 /Ole On 28-11-2022 16:16, Pacey, Mike wrote: > Does anyone have suggestions as to how to produce a summary of a > user's TRES resources for running jobs? I'd like to able to see how > each user is fairing against their qos resource limits. (I'm looking > for something functionally equivalent to Grid Engine's qquota > command). The info must be in the scheduler somewhere in order for it > to enforce qos TRES limits, but as a SLURM novice I've not found any way to > do this. > > To summarise TRES qos limits I can do this: > > % sacctmgr list qos format=Name,MaxTRESPerUser%50 > >Name MaxTRESPU > > -- -- > > normalcpu=80,mem=320G > > But to work out what a user is currently using in currently running > jobs, the nearest I can work out is: > > % sacct -X -s R --units=G -o User,ReqTRES%50 > > UserReqTRES > > - -- > > pacey billing=1,cpu=1,mem=0.49G,node=1 > > pacey billing=1,cpu=1,mem=0.49G,node=1 > > With a little scripting I can sum those up, but there might be a > neater way to do this?
[slurm-users] Per-user TRES summary?
Hi folks, Does anyone have suggestions as to how to produce a summary of a user's TRES resources for running jobs? I'd like to able to see how each user is fairing against their qos resource limits. (I'm looking for something functionally equivalent to Grid Engine's qquota command). The info must be in the scheduler somewhere in order for it to enforce qos TRES limits, but as a SLURM novice I've not found any way to do this. To summarise TRES qos limits I can do this: % sacctmgr list qos format=Name,MaxTRESPerUser%50 Name MaxTRESPU -- -- normalcpu=80,mem=320G But to work out what a user is currently using in currently running jobs, the nearest I can work out is: % sacct -X -s R --units=G -o User,ReqTRES%50 UserReqTRES - -- pacey billing=1,cpu=1,mem=0.49G,node=1 pacey billing=1,cpu=1,mem=0.49G,node=1 With a little scripting I can sum those up, but there might be a neater way to do this? Regards, Mike
