[slurm-users] How to check if there's a reservation

2018-05-10 Thread Prentice Bisbal
Dear Slurm Users, We've started using maintenance reservations. As you would expect, this caused some confusion for users who were wondering why their jobs were queuing up and not running. Some of my users provide a public service of sorts that automatically submits jobs to our cluster. They w

Re: [slurm-users] How to check if there's a reservation

2018-05-11 Thread Bill Wichser
The problem is that reservations can be in there yet have no effect on the submitted job if they would run before the reservation takes place. One can pull the starting time simply using something like this scontrol show res -o | awk '{print $2}' with output StartTime=2018-06-12T06:00:00 Star

Re: [slurm-users] How to check if there's a reservation

2018-05-11 Thread Paul Edmon
In the past we used the LUA job submit plugin to block jobs that would intersect maintenance reservations.  I would look at that. -Paul Edmon- On 05/11/2018 08:19 AM, Bill Wichser wrote: The problem is that reservations can be in there yet have no effect on the submitted job if they would run

Re: [slurm-users] How to check if there's a reservation

2018-05-11 Thread Douglas Jacobsen
A feature that many slurm users might like is sbatch --time-min. Using both --time-min and --time a user can specify the range of acceptable wall times limits. This can make it much easier to keep jobs running right up to the maintenance reservation. e.g.: sbatch --time-min=30:00 --time=48:00:

Re: [slurm-users] How to check if there's a reservation

2018-05-11 Thread Chris Samuel
Hey Prentice, On Friday, 11 May 2018 6:23:06 AM AEST Prentice Bisbal wrote: > They would like to have their submission framework automatically > detect if there's a reservation that may interfere with their jobs, and > act accordingly. As an additional data point there is also srun's "--test-onl

Re: [slurm-users] How to check if there's a reservation

2018-06-13 Thread Prentice Bisbal
Just to revisit this, for jobs that are queued, but prevented from running, will have a more useful reason in 18.08, which will address one of my issues with reservation collisions. https://bugs.schedmd.com/show_bug.cgi?id=5138 https://bugs.schedmd.com/show_bug.cgi?id=4987 Prentice Bisbal Lea

Re: [slurm-users] How to check if there's a reservation

2018-06-15 Thread Ryan Novosielski
That’s great news — this is is a vFAQ at our site. > On Jun 13, 2018, at 1:37 PM, Prentice Bisbal wrote: > > Just to revisit this, for jobs that are queued, but prevented from running, > will have a more useful reason in 18.08, which will address one of my issues > with reservation collisions.

Re: [slurm-users] How to check if there's a reservation

2018-06-15 Thread Prentice Bisbal
I agree. I brought it up with SchedMD after I spent almost an entire day trying to figure out why jobs were queued up but not running. I figured the reason column would say "reservation" if that was the issue. Instead, it provided some completely useless message, making me think the problem was