Re: [slurm-users] Creating priority quotas

[Manuel Rodríguez Pascual]

In case any of you is interested, my team integrated DMTCP checkpoint
library with Slurm, thus allowing this preemption to be done without
loosing any computation (and some other fancy stuff).  An important use
case for us is, in fact, these VIP queues :)

Here is a link to the documentation, including tests, configuration and
howtos: https://github.com/ciemat-tic/codec/wiki/Slurm-DMTCP

cheers,

Manuel


2018-01-25 17:18 GMT+01:00 Brian Novogradac <brian.novogra...@utoronto.ca>:

> Thank you for your input!
>
>
> Would any of you have an example on how you setup the conf file for the
> queue using the QOS method?  I'm reading the QOS docs as I type.
>
>
> Brian
> --
> *From:* slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of
> Loris Bennett <loris.benn...@fu-berlin.de>
> *Sent:* Thursday, January 25, 2018 11:09 AM
> *To:* Slurm User Community List
> *Subject:* Re: [slurm-users] Creating priority quotas
>
> Hi Brian,
>
> QOS is probably the right way to go.  You can set up a QOS 'vip', which
> can preempt other QOS.  We don't preempt, but use multifactor priority
> with various QOS with different priority values.  In that case, VIP jobs
> won't start immediately, but just get pushed to the front of the queue.
> On the other hand, no CPU-time is lost due to low-status jobs being
> terminated early due to preemption (although if the low-status jobs are
> able to do some form of checkpointing that will be less of an issue).
> Depending on how pushy your VIPs are, they might go for a preemptionless
> solution, too.
>
> Cheers,
>
> Loris
>
> John Hearns <hear...@googlemail.com> writes:
>
> > Brian, not my area of expertise. Do you want 'premption' - ie the VIP
> user runs something and other jobs are pre-empted?
> > https://slurm.schedmd.com/preempt.html
> Slurm Workload Manager <https://slurm.schedmd.com/preempt.html>
> slurm.schedmd.com
> Preemption. Slurm supports job preemption, the act of stopping one or more
> "low-priority" jobs to let a "high-priority" job run. Job preemption is
> implemented as a ...
>
>
> >
> > On 25 January 2018 at 16:27, Brian Novogradac <
> brian.novogra...@utoronto.ca> wrote:
> >
> >  I'm new to Slurm, and looking for some assistance.
> >
> >  I need to create various queues. The one i am having issues with is a
> "VIP" queue
> >
> >  I want to create a queue for a specific node that overides all jobs on
> that node when a "VIP" uses the queue.
> >
> >  We are using SSSD for our authentication system to the login node.
> >
> >  I'm looking at the QOS docs and am stumped.
> >
> >  Any help or direction much appreciated.
> >
> >  Brian Novogradac
> >
> >
> --
> Dr. Loris Bennett (Mr.)
> ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de
>
>

Re: [slurm-users] Creating priority quotas

[Brian Novogradac]

Thank you for your input!


Would any of you have an example on how you setup the conf file for the queue 
using the QOS method?  I'm reading the QOS docs as I type.


Brian


From: slurm-users <slurm-users-boun...@lists.schedmd.com> on behalf of Loris 
Bennett <loris.benn...@fu-berlin.de>
Sent: Thursday, January 25, 2018 11:09 AM
To: Slurm User Community List
Subject: Re: [slurm-users] Creating priority quotas

Hi Brian,

QOS is probably the right way to go.  You can set up a QOS 'vip', which
can preempt other QOS.  We don't preempt, but use multifactor priority
with various QOS with different priority values.  In that case, VIP jobs
won't start immediately, but just get pushed to the front of the queue.
On the other hand, no CPU-time is lost due to low-status jobs being
terminated early due to preemption (although if the low-status jobs are
able to do some form of checkpointing that will be less of an issue).
Depending on how pushy your VIPs are, they might go for a preemptionless
solution, too.

Cheers,

Loris

John Hearns <hear...@googlemail.com> writes:

> Brian, not my area of expertise. Do you want 'premption' - ie the VIP user 
> runs something and other jobs are pre-empted?
> https://slurm.schedmd.com/preempt.html
Slurm Workload Manager<https://slurm.schedmd.com/preempt.html>
slurm.schedmd.com
Preemption. Slurm supports job preemption, the act of stopping one or more 
"low-priority" jobs to let a "high-priority" job run. Job preemption is 
implemented as a ...



>
> On 25 January 2018 at 16:27, Brian Novogradac <brian.novogra...@utoronto.ca> 
> wrote:
>
>  I'm new to Slurm, and looking for some assistance.
>
>  I need to create various queues. The one i am having issues with is a "VIP" 
> queue
>
>  I want to create a queue for a specific node that overides all jobs on that 
> node when a "VIP" uses the queue.
>
>  We are using SSSD for our authentication system to the login node.
>
>  I'm looking at the QOS docs and am stumped.
>
>  Any help or direction much appreciated.
>
>  Brian Novogradac
>
>
--
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de

Re: [slurm-users] Creating priority quotas

[Jeff White]

If it is preemption here's how we do it:

PreemptMode=REQUEUE
PreemptType=preempt/partition_prio

PartitionName=plebeians  Nodes=cn[1-35]  Priority=1000 Default=YES
PartitionName=vips  Nodes=cn[1-35]  Priority=5000  Default=NO


Jeff White
HPC Systems Engineer - ITS
Question about or help with Kamiak? Please submit a Service Request 
.

On 01/25/2018 07:34 AM, John Hearns wrote:
Brian, not my area of expertise.  Do you want 'premption' - ie the VIP 
user runs something and other jobs are pre-empted?
https://slurm.schedmd.com/preempt.html 



On 25 January 2018 at 16:27, Brian Novogradac 
> 
wrote:


I'm new to Slurm, and looking for some assistance.


I need to create various queues.  The one i am having issues with
is a "VIP" queue


I want to create a queue for a specific node that overides all
jobs on that node when a "VIP" uses the queue.

We are using SSSD for our authentication system to the login node.


I'm looking at the QOS docs and am stumped.


Any help or direction much appreciated.



Brian Novogradac

Re: [slurm-users] Creating priority quotas

[Loris Bennett]

Hi Brian,

QOS is probably the right way to go.  You can set up a QOS 'vip', which
can preempt other QOS.  We don't preempt, but use multifactor priority
with various QOS with different priority values.  In that case, VIP jobs
won't start immediately, but just get pushed to the front of the queue.
On the other hand, no CPU-time is lost due to low-status jobs being
terminated early due to preemption (although if the low-status jobs are
able to do some form of checkpointing that will be less of an issue).
Depending on how pushy your VIPs are, they might go for a preemptionless
solution, too.

Cheers,

Loris

John Hearns  writes:

> Brian, not my area of expertise. Do you want 'premption' - ie the VIP user 
> runs something and other jobs are pre-empted?
> https://slurm.schedmd.com/preempt.html
>
> On 25 January 2018 at 16:27, Brian Novogradac  
> wrote:
>
>  I'm new to Slurm, and looking for some assistance.
>
>  I need to create various queues. The one i am having issues with is a "VIP" 
> queue
>
>  I want to create a queue for a specific node that overides all jobs on that 
> node when a "VIP" uses the queue.
>
>  We are using SSSD for our authentication system to the login node.
>
>  I'm looking at the QOS docs and am stumped.
>
>  Any help or direction much appreciated.
>
>  Brian Novogradac
>
>
-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de

Re: [slurm-users] Creating priority quotas

[John Hearns]

Brian, not my area of expertise.  Do you want 'premption'  - ie the VIP
user runs something and other jobs are pre-empted?
https://slurm.schedmd.com/preempt.html

On 25 January 2018 at 16:27, Brian Novogradac 
wrote:

> I'm new to Slurm, and looking for some assistance.
>
>
> I need to create various queues.  The one i am having issues with is a
> "VIP" queue
>
>
> I want to create a queue for a specific node that overides all jobs on
> that node when a "VIP" uses the queue.
>
> We are using SSSD for our authentication system to the login node.
>
>
> I'm looking at the QOS docs and am stumped.
>
>
> Any help or direction much appreciated.
>
>
>
> Brian Novogradac
>
>