[slurm-users] How to check the bench mark capacity of the SLURM setup
Dear All, Good morning We have setup of slurm setup for a HPC setup of 4 node We want to do a stress test , guidnace requested for getting a code which can test the functionality of the SLURM efficiency. If there is such a program, like to try out Guidance requestedThanks Joseph john
Re: [slurm-users] [External] Re: Troubleshooting job stuck in Pending state
Presumably what's in the squeue Reason column isn't rnough? It's not particularly informative, although it does distinguish "Resources" from "Priority", for example, and it'll also list various partition limits, e.g.
Re: [slurm-users] [External] Re: Troubleshooting job stuck in Pending state
I am not a Slurm expert by any stretch of the imagination, so my answer is not authoritative. That said, I am not aware of any functional equivalent for Slurm, and I would love to learn that I am mistaken! On Tue, Dec 12, 2023 at 1:39 AM Pacey, Mike wrote: > Hi Davide, > > > > The jobs do eventually run, but can take several minutes or sometimes > several hours to switch to a running state even when there’s plenty of > resources free immediately. > > > > With Grid Engine it was possible to turn on scheduling diagnostics and get > a summary of the scheduler’s decisions on a pending job by running “qstat > -j jobid”. But there doesn’t seem to be any functional equivalent with > SLURM? > > > > Regards, > > Mike > > > > > > *From:* slurm-users *On Behalf Of > *Davide DelVento > *Sent:* Monday, December 11, 2023 4:23 PM > *To:* Slurm User Community List > *Subject:* [External] Re: [slurm-users] Troubleshooting job stuck in > Pending state > > > > *This email originated outside the University. Check before clicking links > or attachments.* > > By getting "stuck" do you mean the job stays PENDING forever or does > eventually run? I've seen the latter (and I agree with you that I wish > Slurm will log things like "I looked at this job and I am not starting it > yet because") but not the former > > > > On Fri, Dec 8, 2023 at 9:00 AM Pacey, Mike > wrote: > > Hi folks, > > > > I’m looking for some advice on how to troubleshoot jobs we occasionally > see on our cluster that are stuck in a pending state despite sufficient > matching resources being free. In the case I’m trying to troubleshoot the > Reason field lists (Priority) but to find any way to get the scheduler to > tell me what exactly is the priority job blocking. > > > >- I tried setting the scheduler log level to debug3 for 5 minutes at >one point, but my logfile ballooned from 0.5G to 1.5G and didn’t offer any >useful info for this case. >- I’ve tried ‘scontrol schedloglevel 1’ but it returns the error: >‘slurm_set_schedlog_level error: Requested operation is presently disabled’ > > > > I’m aware that the backfill scheduler will occasionally hold on to free > resources in order to schedule a larger job with higher priority, but in > this case I can’t find any pending job that might fit the bill. > > > > And to possibly complicate matters, this is on a large partition that has > no maximum time limit and most pending jobs have no time limits either. (We > use backfill/fairshare as we have smaller partitions of rarer resources > that benefit from it, plus we’re aiming to use fairshare even on the > no-time-limits partitions to help balance out usage). > > > > Hoping someone can provide pointers. > > > > Regards, > > Mike > >
[slurm-users] Slurm doesn't allocate job on available MIGs
Hello, we have a problem on a DGX where the 4 A100s are split into different MIGs (Multi-Instance GPUs). We use slurm to allocate jobs on partitions grouping MIGs according to their size: - prod10 for 10 x 1g10gb - prod20 for 4 x 2g20gb - prod40 for 1 x 3g40gb - prod80 for 1 x A100g80gb The problem encountered is, for example: 1. a first job runs on prod40 2. a second job is pending for its place on prod40, since no more 3g40gb MIGs are available (reason: ressources) 3. a third job is waiting to run on prod10, even though all 10 1g10gb MIGs are available (reason: Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions) We don't understand why slurm doesn't allocate 1g.10gb MIGs for the 3rd job, which we don't think should have to wait. In the event that there is no second job waiting, the jobs can use prod10 without waiting. The slurm.conf and gres.conf files are available as attachments. Can anyone help us solve the problem? Have a nice day, Tristan Gillard dgx_gres.conf Description: dgx_gres.conf dgx_slurm.conf Description: dgx_slurm.conf
Re: [slurm-users] [External] Re: Troubleshooting job stuck in Pending state
Hi Davide, The jobs do eventually run, but can take several minutes or sometimes several hours to switch to a running state even when there’s plenty of resources free immediately. With Grid Engine it was possible to turn on scheduling diagnostics and get a summary of the scheduler’s decisions on a pending job by running “qstat -j jobid”. But there doesn’t seem to be any functional equivalent with SLURM? Regards, Mike From: slurm-users On Behalf Of Davide DelVento Sent: Monday, December 11, 2023 4:23 PM To: Slurm User Community List Subject: [External] Re: [slurm-users] Troubleshooting job stuck in Pending state This email originated outside the University. Check before clicking links or attachments. By getting "stuck" do you mean the job stays PENDING forever or does eventually run? I've seen the latter (and I agree with you that I wish Slurm will log things like "I looked at this job and I am not starting it yet because") but not the former On Fri, Dec 8, 2023 at 9:00 AM Pacey, Mike mailto:m.pa...@lancaster.ac.uk>> wrote: Hi folks, I’m looking for some advice on how to troubleshoot jobs we occasionally see on our cluster that are stuck in a pending state despite sufficient matching resources being free. In the case I’m trying to troubleshoot the Reason field lists (Priority) but to find any way to get the scheduler to tell me what exactly is the priority job blocking. * I tried setting the scheduler log level to debug3 for 5 minutes at one point, but my logfile ballooned from 0.5G to 1.5G and didn’t offer any useful info for this case. * I’ve tried ‘scontrol schedloglevel 1’ but it returns the error: ‘slurm_set_schedlog_level error: Requested operation is presently disabled’ I’m aware that the backfill scheduler will occasionally hold on to free resources in order to schedule a larger job with higher priority, but in this case I can’t find any pending job that might fit the bill. And to possibly complicate matters, this is on a large partition that has no maximum time limit and most pending jobs have no time limits either. (We use backfill/fairshare as we have smaller partitions of rarer resources that benefit from it, plus we’re aiming to use fairshare even on the no-time-limits partitions to help balance out usage). Hoping someone can provide pointers. Regards, Mike
