Re: [slurm-users] remote license

[Brian Andrus]

Davide,

No worries. I like to see folks engage SchedMD, especially if they are 
seeing value in all the work that has gone into creating the software 
that runs the majority of fastest clusters on the planet.


To your question: yes, you can update slurm.conf and do 'scontrol 
reconfigure' to change the number of local licenses available. You could 
do that fairly regularly. I am not sure what it would do if you lowered 
the number to something smaller than the total in use. It should allow 
jobs to continue and drain down.


It is a much simpler way to achieve your goal, however, it affects the 
system more to run 'scontrol reconfigure' than it does to use the remote 
license method.
For a large scale system, that impact is very noticeable. For a small 
cluster, it is negligible. Thus, it may well be the best method for you 
and your environment. If you find yourself working on a large cluster 
sometime in your career, I would not recommend using it there.


Brian Andrus

On 9/16/2022 3:06 PM, Davide DelVento wrote:

Hi Brian,

 From your response, I speculate that my wording sounded harsh or
unrespectful. That was not my intention and therefore I sincerely
apologize for it.

In fact my perplexity is certainly due to my ignorance (as it must be
very clear by the number and "quality" of queries that I am posting on
this mailing list). It seemed to me that what is currently available
is a special edge case, whereas the simpler one is not covered, so I
was (perhaps still am) convinced that I must have misunderstood how
things work. Perhaps my "am I missing something?" sounded rhetorical
rather than sincere which was the spirit I wrote it with. Sorry about
that.

I find it a nice coincidence that you suggested paying SchedMD for
this, because after the clarifications which I am trying to get in
this thread, I thought to ask my management to do exactly that!!!

For the local license, are you suggesting to programmatically change
slurm.conf and reconfigure e.g. in a cron?

Thanks a lot for your help and have a great weekend

Davide






On Fri, Sep 16, 2022 at 11:52 AM Brian Andrus  wrote:

Davide,

I'll not engage on this. If you want a feature, pay SchedMD for support
and they will prioritize it and work on it.  You are already using a
very impressive bit of software for free.

As far as local license updates, yes, you can do the local license and
reconfigure regularly. Feel free to do that. It is not something that
scales well, but it looks like you have a rather beginner cluster that
would never be impacted by such choices.

Brian Andrus


On 9/16/2022 10:00 AM, Davide DelVento wrote:

Thanks Brian.

I am still perplexed. What is a database to install, administer,
patch, update, could break, be down, etc buying us? I see limited use
cases, e.g. a license server which does not provide the license
count/use in a parsable way, and that someone wants to use with
multiple SLURM installations (if it's on a single one, the local
license is perfect). Wouldn't it much, much easier for everybody if
one could specify a script (the bullet 1. you mentioned) inside SLURM,
and use the license server ITSELF as the authoritative source of
license count? Sure, it won't be perfect, e.g. race conditions in
license acquisition can still cause failures, but the database won't
be fixing that
I must be missing something

Alternatively, can one update the license count of local license with
a scontrol command, rather than changing the slurm.conf and
reconfigure? That could make what I say possible

Thanks

On Fri, Sep 16, 2022 at 9:25 AM Brian Andrus  wrote:

Davide,

You have it pretty correct. While the database itself is not part of the
slurm suite, slurmdbd (which would access the database) is.

As far as writing something that keeps things updated, I'm sure many
have done this. However, it would be unique to your installation. The
specific number of licenses, naming them, what license server is being
used, etc.
All of that could easily be a few lines in a script that you have in a
cron job or other trigger (eg prolog/epilog). You would just:

1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil
lmstat output)
2) Update the database (sacctmgr command)

As you can see, that 1st step would be highly dependent on you and your
environment. The 2nd step would be dependent on what things you are
tracking within that.

Brian Andrus


On 9/16/2022 5:01 AM, Davide DelVento wrote:

So if I understand correctly, this "remote database" is something that
is neither part of slurm itself, nor part of the license server per
se, correct?

Regarding the "if you got creative", has anybody on this list done
that already? I can't believe I'm the first one wanting this feature!
Matching the number in that database with the actual number the
license server knows would be extremely helpful! We use various
license servers (for various licensed software), so each one of them
would be useful. I can probably 

Re: [slurm-users] remote license

[Davide DelVento]

Hi Brian,

>From your response, I speculate that my wording sounded harsh or
unrespectful. That was not my intention and therefore I sincerely
apologize for it.

In fact my perplexity is certainly due to my ignorance (as it must be
very clear by the number and "quality" of queries that I am posting on
this mailing list). It seemed to me that what is currently available
is a special edge case, whereas the simpler one is not covered, so I
was (perhaps still am) convinced that I must have misunderstood how
things work. Perhaps my "am I missing something?" sounded rhetorical
rather than sincere which was the spirit I wrote it with. Sorry about
that.

I find it a nice coincidence that you suggested paying SchedMD for
this, because after the clarifications which I am trying to get in
this thread, I thought to ask my management to do exactly that!!!

For the local license, are you suggesting to programmatically change
slurm.conf and reconfigure e.g. in a cron?

Thanks a lot for your help and have a great weekend

Davide






On Fri, Sep 16, 2022 at 11:52 AM Brian Andrus  wrote:
>
> Davide,
>
> I'll not engage on this. If you want a feature, pay SchedMD for support
> and they will prioritize it and work on it.  You are already using a
> very impressive bit of software for free.
>
> As far as local license updates, yes, you can do the local license and
> reconfigure regularly. Feel free to do that. It is not something that
> scales well, but it looks like you have a rather beginner cluster that
> would never be impacted by such choices.
>
> Brian Andrus
>
>
> On 9/16/2022 10:00 AM, Davide DelVento wrote:
> > Thanks Brian.
> >
> > I am still perplexed. What is a database to install, administer,
> > patch, update, could break, be down, etc buying us? I see limited use
> > cases, e.g. a license server which does not provide the license
> > count/use in a parsable way, and that someone wants to use with
> > multiple SLURM installations (if it's on a single one, the local
> > license is perfect). Wouldn't it much, much easier for everybody if
> > one could specify a script (the bullet 1. you mentioned) inside SLURM,
> > and use the license server ITSELF as the authoritative source of
> > license count? Sure, it won't be perfect, e.g. race conditions in
> > license acquisition can still cause failures, but the database won't
> > be fixing that
> > I must be missing something
> >
> > Alternatively, can one update the license count of local license with
> > a scontrol command, rather than changing the slurm.conf and
> > reconfigure? That could make what I say possible
> >
> > Thanks
> >
> > On Fri, Sep 16, 2022 at 9:25 AM Brian Andrus  wrote:
> >> Davide,
> >>
> >> You have it pretty correct. While the database itself is not part of the
> >> slurm suite, slurmdbd (which would access the database) is.
> >>
> >> As far as writing something that keeps things updated, I'm sure many
> >> have done this. However, it would be unique to your installation. The
> >> specific number of licenses, naming them, what license server is being
> >> used, etc.
> >> All of that could easily be a few lines in a script that you have in a
> >> cron job or other trigger (eg prolog/epilog). You would just:
> >>
> >> 1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil
> >> lmstat output)
> >> 2) Update the database (sacctmgr command)
> >>
> >> As you can see, that 1st step would be highly dependent on you and your
> >> environment. The 2nd step would be dependent on what things you are
> >> tracking within that.
> >>
> >> Brian Andrus
> >>
> >>
> >> On 9/16/2022 5:01 AM, Davide DelVento wrote:
> >>> So if I understand correctly, this "remote database" is something that
> >>> is neither part of slurm itself, nor part of the license server per
> >>> se, correct?
> >>>
> >>> Regarding the "if you got creative", has anybody on this list done
> >>> that already? I can't believe I'm the first one wanting this feature!
> >>> Matching the number in that database with the actual number the
> >>> license server knows would be extremely helpful! We use various
> >>> license servers (for various licensed software), so each one of them
> >>> would be useful. I can probably script/develop one of these myself,
> >>> but I am not sure I've got the time...
> >>>
> >>> Thanks!
> >>>
> >>> On Thu, Sep 15, 2022 at 6:04 PM Brian Andrus  wrote:
>  So if you follow the links to: https://slurm.schedmd.com/licenses.html
>  you should see the difference.
> 
>  Local licenses are just a counter that is setup in slurm.conf
>  Remote liceneses are a counter in a database (the database is "remote"),
>  so you can change/update it dynamically. So, you could change their
>  allocation with a sacctmgr command. It is especially useful when you are
>  managing multiple clusters that share licenses. You can allocate that a
>  certain number are allowed by each cluster and change that if needed.
> 
>  If you 

Re: [slurm-users] remote license

[Brian Andrus]

Davide,

I'll not engage on this. If you want a feature, pay SchedMD for support 
and they will prioritize it and work on it.  You are already using a 
very impressive bit of software for free.


As far as local license updates, yes, you can do the local license and 
reconfigure regularly. Feel free to do that. It is not something that 
scales well, but it looks like you have a rather beginner cluster that 
would never be impacted by such choices.


Brian Andrus


On 9/16/2022 10:00 AM, Davide DelVento wrote:

Thanks Brian.

I am still perplexed. What is a database to install, administer,
patch, update, could break, be down, etc buying us? I see limited use
cases, e.g. a license server which does not provide the license
count/use in a parsable way, and that someone wants to use with
multiple SLURM installations (if it's on a single one, the local
license is perfect). Wouldn't it much, much easier for everybody if
one could specify a script (the bullet 1. you mentioned) inside SLURM,
and use the license server ITSELF as the authoritative source of
license count? Sure, it won't be perfect, e.g. race conditions in
license acquisition can still cause failures, but the database won't
be fixing that
I must be missing something

Alternatively, can one update the license count of local license with
a scontrol command, rather than changing the slurm.conf and
reconfigure? That could make what I say possible

Thanks

On Fri, Sep 16, 2022 at 9:25 AM Brian Andrus  wrote:

Davide,

You have it pretty correct. While the database itself is not part of the
slurm suite, slurmdbd (which would access the database) is.

As far as writing something that keeps things updated, I'm sure many
have done this. However, it would be unique to your installation. The
specific number of licenses, naming them, what license server is being
used, etc.
All of that could easily be a few lines in a script that you have in a
cron job or other trigger (eg prolog/epilog). You would just:

1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil
lmstat output)
2) Update the database (sacctmgr command)

As you can see, that 1st step would be highly dependent on you and your
environment. The 2nd step would be dependent on what things you are
tracking within that.

Brian Andrus


On 9/16/2022 5:01 AM, Davide DelVento wrote:

So if I understand correctly, this "remote database" is something that
is neither part of slurm itself, nor part of the license server per
se, correct?

Regarding the "if you got creative", has anybody on this list done
that already? I can't believe I'm the first one wanting this feature!
Matching the number in that database with the actual number the
license server knows would be extremely helpful! We use various
license servers (for various licensed software), so each one of them
would be useful. I can probably script/develop one of these myself,
but I am not sure I've got the time...

Thanks!

On Thu, Sep 15, 2022 at 6:04 PM Brian Andrus  wrote:

So if you follow the links to: https://slurm.schedmd.com/licenses.html
you should see the difference.

Local licenses are just a counter that is setup in slurm.conf
Remote liceneses are a counter in a database (the database is "remote"),
so you can change/update it dynamically. So, you could change their
allocation with a sacctmgr command. It is especially useful when you are
managing multiple clusters that share licenses. You can allocate that a
certain number are allowed by each cluster and change that if needed.

If you got creative, you could keep the license count that is in the
database updated to match the number free from flexlm to stop license
starvation due to users outside slurm using them up so they really
aren't available to slurm.

Brian Andrus


On 9/15/2022 3:34 PM, Davide DelVento wrote:

I am a bit confused by remote licenses.

https://lists.schedmd.com/pipermail/slurm-users/2020-September/006049.html
(which is only 2 years old) claims that they are just a counter, so
like local licenses. Then why call them remote?

Only a few days after, this
https://lists.schedmd.com/pipermail/slurm-users/2020-September/006081.html
appeared to imply (but not clearly stated) that the remote license are
not simply a counter, but then it's not clear how they are different.

The current documentation (and attempts to run the "add resource"
command) says that one must use the license count, which seems to
imply they are just a simple counter (but then what do they need the
server for)?

So what is what?

In my cursory past experience with this, it seemed that it were
possible to query a license server (at least some of them) to get the
actual number of available licenses and schedule (or let jobs pending)
accordingly. Which would be very helpful for the not-too-uncommon
situation in which the same license server provides licenses for both
the HPC cluster and other non-slurm-controlled resources, such a
user's workstations. Was that impression wrong, or 

[slurm-users] Configure account after installing Slurm

[Alejandro Acuña]

Hi ladies and gentlemen. 
I am finding some problems while configuring sacct on an existing slurm-wlm 
19.05.5 installation. I can not start MySQL Server for begin account 
configuration becose DB server have root password (don't remember set it up). 
If I try to remove and reinstall MySQL/MariaDB, I lose Slurm installation. I 
need constantly functional squeue while I configure account. It is possible? 
What am I missing? 

Thanks, have a nice weekend everyone. 
Alex 

Re: [slurm-users] remote license

[Davide DelVento]

Thanks Brian.

I am still perplexed. What is a database to install, administer,
patch, update, could break, be down, etc buying us? I see limited use
cases, e.g. a license server which does not provide the license
count/use in a parsable way, and that someone wants to use with
multiple SLURM installations (if it's on a single one, the local
license is perfect). Wouldn't it much, much easier for everybody if
one could specify a script (the bullet 1. you mentioned) inside SLURM,
and use the license server ITSELF as the authoritative source of
license count? Sure, it won't be perfect, e.g. race conditions in
license acquisition can still cause failures, but the database won't
be fixing that
I must be missing something

Alternatively, can one update the license count of local license with
a scontrol command, rather than changing the slurm.conf and
reconfigure? That could make what I say possible

Thanks

On Fri, Sep 16, 2022 at 9:25 AM Brian Andrus  wrote:
>
> Davide,
>
> You have it pretty correct. While the database itself is not part of the
> slurm suite, slurmdbd (which would access the database) is.
>
> As far as writing something that keeps things updated, I'm sure many
> have done this. However, it would be unique to your installation. The
> specific number of licenses, naming them, what license server is being
> used, etc.
> All of that could easily be a few lines in a script that you have in a
> cron job or other trigger (eg prolog/epilog). You would just:
>
> 1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil
> lmstat output)
> 2) Update the database (sacctmgr command)
>
> As you can see, that 1st step would be highly dependent on you and your
> environment. The 2nd step would be dependent on what things you are
> tracking within that.
>
> Brian Andrus
>
>
> On 9/16/2022 5:01 AM, Davide DelVento wrote:
> > So if I understand correctly, this "remote database" is something that
> > is neither part of slurm itself, nor part of the license server per
> > se, correct?
> >
> > Regarding the "if you got creative", has anybody on this list done
> > that already? I can't believe I'm the first one wanting this feature!
> > Matching the number in that database with the actual number the
> > license server knows would be extremely helpful! We use various
> > license servers (for various licensed software), so each one of them
> > would be useful. I can probably script/develop one of these myself,
> > but I am not sure I've got the time...
> >
> > Thanks!
> >
> > On Thu, Sep 15, 2022 at 6:04 PM Brian Andrus  wrote:
> >> So if you follow the links to: https://slurm.schedmd.com/licenses.html
> >> you should see the difference.
> >>
> >> Local licenses are just a counter that is setup in slurm.conf
> >> Remote liceneses are a counter in a database (the database is "remote"),
> >> so you can change/update it dynamically. So, you could change their
> >> allocation with a sacctmgr command. It is especially useful when you are
> >> managing multiple clusters that share licenses. You can allocate that a
> >> certain number are allowed by each cluster and change that if needed.
> >>
> >> If you got creative, you could keep the license count that is in the
> >> database updated to match the number free from flexlm to stop license
> >> starvation due to users outside slurm using them up so they really
> >> aren't available to slurm.
> >>
> >> Brian Andrus
> >>
> >>
> >> On 9/15/2022 3:34 PM, Davide DelVento wrote:
> >>> I am a bit confused by remote licenses.
> >>>
> >>> https://lists.schedmd.com/pipermail/slurm-users/2020-September/006049.html
> >>> (which is only 2 years old) claims that they are just a counter, so
> >>> like local licenses. Then why call them remote?
> >>>
> >>> Only a few days after, this
> >>> https://lists.schedmd.com/pipermail/slurm-users/2020-September/006081.html
> >>> appeared to imply (but not clearly stated) that the remote license are
> >>> not simply a counter, but then it's not clear how they are different.
> >>>
> >>> The current documentation (and attempts to run the "add resource"
> >>> command) says that one must use the license count, which seems to
> >>> imply they are just a simple counter (but then what do they need the
> >>> server for)?
> >>>
> >>> So what is what?
> >>>
> >>> In my cursory past experience with this, it seemed that it were
> >>> possible to query a license server (at least some of them) to get the
> >>> actual number of available licenses and schedule (or let jobs pending)
> >>> accordingly. Which would be very helpful for the not-too-uncommon
> >>> situation in which the same license server provides licenses for both
> >>> the HPC cluster and other non-slurm-controlled resources, such a
> >>> user's workstations. Was that impression wrong, or perhaps somebody
> >>> scripted it in some way? If the latter, does anybody know if those
> >>> scripts are publicly available anywhere?
> >>>
> >>> Thanks
> >>>
>

Re: [slurm-users] remote license

[Brian Andrus]

Davide,

You have it pretty correct. While the database itself is not part of the 
slurm suite, slurmdbd (which would access the database) is.


As far as writing something that keeps things updated, I'm sure many 
have done this. However, it would be unique to your installation. The 
specific number of licenses, naming them, what license server is being 
used, etc.
All of that could easily be a few lines in a script that you have in a 
cron job or other trigger (eg prolog/epilog). You would just:


1) Read/parse current licenses/use (eg: if you are using flexlm, lmutil 
lmstat output)

2) Update the database (sacctmgr command)

As you can see, that 1st step would be highly dependent on you and your 
environment. The 2nd step would be dependent on what things you are 
tracking within that.


Brian Andrus


On 9/16/2022 5:01 AM, Davide DelVento wrote:

So if I understand correctly, this "remote database" is something that
is neither part of slurm itself, nor part of the license server per
se, correct?

Regarding the "if you got creative", has anybody on this list done
that already? I can't believe I'm the first one wanting this feature!
Matching the number in that database with the actual number the
license server knows would be extremely helpful! We use various
license servers (for various licensed software), so each one of them
would be useful. I can probably script/develop one of these myself,
but I am not sure I've got the time...

Thanks!

On Thu, Sep 15, 2022 at 6:04 PM Brian Andrus  wrote:

So if you follow the links to: https://slurm.schedmd.com/licenses.html
you should see the difference.

Local licenses are just a counter that is setup in slurm.conf
Remote liceneses are a counter in a database (the database is "remote"),
so you can change/update it dynamically. So, you could change their
allocation with a sacctmgr command. It is especially useful when you are
managing multiple clusters that share licenses. You can allocate that a
certain number are allowed by each cluster and change that if needed.

If you got creative, you could keep the license count that is in the
database updated to match the number free from flexlm to stop license
starvation due to users outside slurm using them up so they really
aren't available to slurm.

Brian Andrus


On 9/15/2022 3:34 PM, Davide DelVento wrote:

I am a bit confused by remote licenses.

https://lists.schedmd.com/pipermail/slurm-users/2020-September/006049.html
(which is only 2 years old) claims that they are just a counter, so
like local licenses. Then why call them remote?

Only a few days after, this
https://lists.schedmd.com/pipermail/slurm-users/2020-September/006081.html
appeared to imply (but not clearly stated) that the remote license are
not simply a counter, but then it's not clear how they are different.

The current documentation (and attempts to run the "add resource"
command) says that one must use the license count, which seems to
imply they are just a simple counter (but then what do they need the
server for)?

So what is what?

In my cursory past experience with this, it seemed that it were
possible to query a license server (at least some of them) to get the
actual number of available licenses and schedule (or let jobs pending)
accordingly. Which would be very helpful for the not-too-uncommon
situation in which the same license server provides licenses for both
the HPC cluster and other non-slurm-controlled resources, such a
user's workstations. Was that impression wrong, or perhaps somebody
scripted it in some way? If the latter, does anybody know if those
scripts are publicly available anywhere?

Thanks

Re: [slurm-users] How to debug a prolog script?

[Davide DelVento]

Thanks a lot.

> > Does it need the execution permission? For root alone sufficient?
>
> slurmd runs as root, so it only need exec perms for root.

Perfect. That must have been then, since my script (like the example
one) did not have the execution permission on.

> I'm curious: What kind of disruption did it cause for your production
> jobs?

All jobs failed and went in pending/held with "launch failed requeued
held" status, all nodes where the jobs were scheduled went draining.

The logs only said "error: validate_node_specs: Prolog or job env
setup failure on node , draining the node". I guess if they said
"-bash: /path/to/prolog: Permission denied" I would have caught the
problem myself.

In hindsight it is obvious, but I don't think even the documentation
mentions that, does it? After all you can execute a file with a
non-executable with with "sh filename", so I made the incorrect
assumption that slurm would have invoked the prolog that way.

Thanks!

Re: [slurm-users] Providing users with info on wait time vs. run time

[Ümit Seren]

On Fri, Sep 16, 2022 at 3:43 PM Sebastian Potthoff <
s.potth...@uni-muenster.de> wrote:

> Hi Hermann,
>
> So you both are happily(?) ignoring this warning the "Prolog and Epilog
> Guide",
> right? :-)
>
> "Prolog and Epilog scripts [...] should not call Slurm commands (e.g.
> squeue,
> scontrol, sacctmgr, etc)."
>
>
> We have probably been doing this since before the warning was added to
> the documentation.  So we are "ignorantly ignoring" the advice :-/
>
>
> Same here :) But if $SLURM_JOB_STDOUT is not defined as documented … what
> can you do.
>

FYI: SLURM_JOB_STDOUT among other ENV variables was added in 22.05 (see
https://slurm.schedmd.com/news.html) so it might not be available if you
have an older SLURM version.



>
> May I ask how big your clusters are (number of nodes) and how heavily they
> are
> used (submitted jobs per hour)?
>
>
> We have around 500 nodes (mostly 2x18 cores). Jobs ending (i.e. calling
> the epilog script) varies quite a lot between 1000 and 15k a day, so
> something in between 40 and 625 Jobs/hour. During those peaks Slurm can
> become noticeably slower, however usually it runs fine.
>
> Sebastian
>
> Am 16.09.2022 um 15:15 schrieb Loris Bennett :
>
> Hi Hermann,
>
> Hermann Schwärzler  writes:
>
> Hi Loris,
> hi Sebastian,
>
> thanks for the information on how you are doing this.
> So you both are happily(?) ignoring this warning the "Prolog and Epilog
> Guide",
> right? :-)
>
> "Prolog and Epilog scripts [...] should not call Slurm commands (e.g.
> squeue,
> scontrol, sacctmgr, etc)."
>
>
> We have probably been doing this since before the warning was added to
> the documentation.  So we are "ignorantly ignoring" the advice :-/
>
> May I ask how big your clusters are (number of nodes) and how heavily they
> are
> used (submitted jobs per hour)?
>
>
> We have around 190 32-core nodes.  I don't know how I would easily find
> out the average number of jobs per hour.  The only problems we have had
> with submission have been when people have written their own mechanisms
> for submitting thousands of jobs.  Once we get them to use job array,
> such problems generally disappear.
>
> Cheers,
>
> Loris
>
> Regards,
> Hermann
>
> On 9/16/22 9:09 AM, Loris Bennett wrote:
>
> Hi Hermann,
> Sebastian Potthoff  writes:
>
> Hi Hermann,
>
> I happened to read along this conversation and was just solving this issue
> today. I added this part to the epilog script to make it work:
>
> # Add job report to stdout
> StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut |
> /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')
>
> NODELIST=($(/usr/bin/scontrol show hostnames))
>
> # Only add to StdOut file if it exists and if we are the first node
> if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
> then
>   echo "# JOB REPORT
> ##" >> $StdOut
>   /usr/bin/seff $SLURM_JOB_ID >> $StdOut
>   echo
> "###"
> >> $StdOut
> fi
>
> We do something similar.  At the end of our script pointed to by
> EpilogSlurmctld we have
>   OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
>   if [ ! -f "$OUT" ]; then
> exit
>   fi
>   printf "\n== Epilog Slurmctld
> ==\n\n" >>  ${OUT}
>   seff ${SLURM_JOB_ID} >> ${OUT}
>   printf
>
> "\n==\n"
>
> ${OUT}
>
>   chown ${user} ${OUT}
> Cheers,
> Loris
>
>   Contrary to what it says in the slurm docs
> https://slurm.schedmd.com/prolog_epilog.html  I was not able to use the
> env var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I
> had to
> make sure it is only called by the „leading“ node as the epilog script
> will be called by ALL nodes of a multinode job and they would all call seff
> and clutter up the output. Last thing was to check if StdOut is
> not of length zero (i.e. it exists). Interactive jobs would otherwise
> cause the node to drain.
>
> Maybe this helps.
>
> Kind regards
> Sebastian
>
> PS: goslmailer looks quite nice with its recommendations! Will definitely
> look into it.
>
> --
> Westfälische Wilhelms-Universität (WWU) Münster
> WWU IT
> Sebastian Potthoff (eScience / HPC)
>
>  Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler <
> hermann.schwaerz...@uibk.ac.at>:
>
>  Hi Ole,
>
>  On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>
>  On 15-09-2022 16:08, Hermann Schwärzler wrote:
>
>  Just out of curiosity: how do you insert the output of seff into the
> out-file of a job?
>
>  Use the "smail" tool from the slurm-contribs RPM and set this in
> slurm.conf:
>  MailProg=/usr/bin/smail
>
>  Maybe I am missing something but from what I can tell smail sends an
> email and does *not* change or append to the .out file of a job...
>
>  Regards,
>  Hermann
>
>
>
> --
> Dr. Loris Bennett (Herr/Mr)
> ZEDAT, Freie 

Re: [slurm-users] Providing users with info on wait time vs. run time

[Paul Edmon]

We also call scontrol in our scripts (a little as we can manage) and we 
run at the scale of 1500 nodes.  It hasn't really caused many issues, 
but we try to limit it as much as we possibly can.


-Paul Edmon-

On 9/16/22 9:41 AM, Sebastian Potthoff wrote:

Hi Hermann,

So you both are happily(?) ignoring this warning the "Prolog and 
Epilog Guide",

right? :-)

"Prolog and Epilog scripts [...] should not call Slurm commands 
(e.g. squeue,

scontrol, sacctmgr, etc)."


We have probably been doing this since before the warning was added to
the documentation.  So we are "ignorantly ignoring" the advice :-/


Same here :) But if $SLURM_JOB_STDOUT is not defined as documented … 
what can you do.


May I ask how big your clusters are (number of nodes) and how 
heavily they are

used (submitted jobs per hour)?


We have around 500 nodes (mostly 2x18 cores). Jobs ending (i.e. 
calling the epilog script) varies quite a lot between 1000 and 15k a 
day, so something in between 40 and 625 Jobs/hour. During those peaks 
Slurm can become noticeably slower, however usually it runs fine.


Sebastian

Am 16.09.2022 um 15:15 schrieb Loris Bennett 
:


Hi Hermann,

Hermann Schwärzler  writes:


Hi Loris,
hi Sebastian,

thanks for the information on how you are doing this.
So you both are happily(?) ignoring this warning the "Prolog and 
Epilog Guide",

right? :-)

"Prolog and Epilog scripts [...] should not call Slurm commands 
(e.g. squeue,

scontrol, sacctmgr, etc)."


We have probably been doing this since before the warning was added to
the documentation.  So we are "ignorantly ignoring" the advice :-/

May I ask how big your clusters are (number of nodes) and how 
heavily they are

used (submitted jobs per hour)?


We have around 190 32-core nodes.  I don't know how I would easily find
out the average number of jobs per hour.  The only problems we have had
with submission have been when people have written their own mechanisms
for submitting thousands of jobs.  Once we get them to use job array,
such problems generally disappear.

Cheers,

Loris


Regards,
Hermann

On 9/16/22 9:09 AM, Loris Bennett wrote:

Hi Hermann,
Sebastian Potthoff  writes:


Hi Hermann,

I happened to read along this conversation and was just solving 
this issue today. I added this part to the epilog script to make 
it work:


# Add job report to stdout
StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep 
StdOut | /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { 
print $2 }')


NODELIST=($(/usr/bin/scontrol show hostnames))

# Only add to StdOut file if it exists and if we are the first node
if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z 
"${StdOut}" ]

then
  echo "# JOB REPORT 
##" >> $StdOut

  /usr/bin/seff $SLURM_JOB_ID >> $StdOut
  echo 
"###" 
>> $StdOut

fi

We do something similar.  At the end of our script pointed to by
EpilogSlurmctld we have
  OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
  if [ ! -f "$OUT" ]; then
exit
  fi
  printf "\n== Epilog Slurmctld
==\n\n" >>  ${OUT}
  seff ${SLURM_JOB_ID} >> ${OUT}
  printf
"\n==\n"

${OUT}

  chown ${user} ${OUT}
Cheers,
Loris

  Contrary to what it says in the slurm docs 
https://slurm.schedmd.com/prolog_epilog.html  I was not able to 
use the env var SLURM_JOB_STDOUT, so I had to fetch it via 
scontrol. In addition I had to
make sure it is only called by the „leading“ node as the epilog 
script will be called by ALL nodes of a multinode job and they 
would all call seff and clutter up the output. Last thing was to 
check if StdOut is
not of length zero (i.e. it exists). Interactive jobs would 
otherwise cause the node to drain.


Maybe this helps.

Kind regards
Sebastian

PS: goslmailer looks quite nice with its recommendations! Will 
definitely look into it.


--
Westfälische Wilhelms-Universität (WWU) Münster
WWU IT
Sebastian Potthoff (eScience / HPC)

 Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
:


 Hi Ole,

 On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:

 On 15-09-2022 16:08, Hermann Schwärzler wrote:

 Just out of curiosity: how do you insert the output of seff into 
the out-file of a job?


 Use the "smail" tool from the slurm-contribs RPM and set this in 
slurm.conf:

 MailProg=/usr/bin/smail

 Maybe I am missing something but from what I can tell smail sends 
an email and does *not* change or append to the .out file of a job...


 Regards,
 Hermann





--
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin emailloris.benn...@fu-berlin.de

Re: [slurm-users] Providing users with info on wait time vs. run time

[Sebastian Potthoff]

Hi Hermann,

>> So you both are happily(?) ignoring this warning the "Prolog and Epilog 
>> Guide",
>> right? :-)
>> 
>> "Prolog and Epilog scripts [...] should not call Slurm commands (e.g. squeue,
>> scontrol, sacctmgr, etc)."
> 
> We have probably been doing this since before the warning was added to
> the documentation.  So we are "ignorantly ignoring" the advice :-/

Same here :) But if $SLURM_JOB_STDOUT is not defined as documented … what can 
you do.

>> May I ask how big your clusters are (number of nodes) and how heavily they 
>> are
>> used (submitted jobs per hour)?


We have around 500 nodes (mostly 2x18 cores). Jobs ending (i.e. calling the 
epilog script) varies quite a lot between 1000 and 15k a day, so something in 
between 40 and 625 Jobs/hour. During those peaks Slurm can become noticeably 
slower, however usually it runs fine.

Sebastian 

> Am 16.09.2022 um 15:15 schrieb Loris Bennett :
> 
> Hi Hermann,
> 
> Hermann Schwärzler  > writes:
> 
>> Hi Loris,
>> hi Sebastian,
>> 
>> thanks for the information on how you are doing this.
>> So you both are happily(?) ignoring this warning the "Prolog and Epilog 
>> Guide",
>> right? :-)
>> 
>> "Prolog and Epilog scripts [...] should not call Slurm commands (e.g. squeue,
>> scontrol, sacctmgr, etc)."
> 
> We have probably been doing this since before the warning was added to
> the documentation.  So we are "ignorantly ignoring" the advice :-/
> 
>> May I ask how big your clusters are (number of nodes) and how heavily they 
>> are
>> used (submitted jobs per hour)?
> 
> We have around 190 32-core nodes.  I don't know how I would easily find
> out the average number of jobs per hour.  The only problems we have had
> with submission have been when people have written their own mechanisms
> for submitting thousands of jobs.  Once we get them to use job array,
> such problems generally disappear.
> 
> Cheers,
> 
> Loris
> 
>> Regards,
>> Hermann
>> 
>> On 9/16/22 9:09 AM, Loris Bennett wrote:
>>> Hi Hermann,
>>> Sebastian Potthoff  writes:
>>> 
 Hi Hermann,
 
 I happened to read along this conversation and was just solving this issue 
 today. I added this part to the epilog script to make it work:
 
 # Add job report to stdout
 StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | 
 /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')
 
 NODELIST=($(/usr/bin/scontrol show hostnames))
 
 # Only add to StdOut file if it exists and if we are the first node
 if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
 then
   echo "# JOB REPORT 
 ##" >> $StdOut
   /usr/bin/seff $SLURM_JOB_ID >> $StdOut
   echo 
 "###"
  >> $StdOut
 fi
>>> We do something similar.  At the end of our script pointed to by
>>> EpilogSlurmctld we have
>>>   OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
>>>   if [ ! -f "$OUT" ]; then
>>> exit
>>>   fi
>>>   printf "\n== Epilog Slurmctld
>>> ==\n\n" >>  ${OUT}
>>>   seff ${SLURM_JOB_ID} >> ${OUT}
>>>   printf
>>> "\n==\n"
> ${OUT}
>>>   chown ${user} ${OUT}
>>> Cheers,
>>> Loris
>>> 
   Contrary to what it says in the slurm docs 
 https://slurm.schedmd.com/prolog_epilog.html  I was not able to use the 
 env var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I 
 had to
 make sure it is only called by the „leading“ node as the epilog script 
 will be called by ALL nodes of a multinode job and they would all call 
 seff and clutter up the output. Last thing was to check if StdOut is
 not of length zero (i.e. it exists). Interactive jobs would otherwise 
 cause the node to drain.
 
 Maybe this helps.
 
 Kind regards
 Sebastian
 
 PS: goslmailer looks quite nice with its recommendations! Will definitely 
 look into it.
 
 --
 Westfälische Wilhelms-Universität (WWU) Münster
 WWU IT
 Sebastian Potthoff (eScience / HPC)
 
  Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
 :
 
  Hi Ole,
 
  On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
 
  On 15-09-2022 16:08, Hermann Schwärzler wrote:
 
  Just out of curiosity: how do you insert the output of seff into the 
 out-file of a job?
 
  Use the "smail" tool from the slurm-contribs RPM and set this in 
 slurm.conf:
  MailProg=/usr/bin/smail
 
  Maybe I am missing something but from what I can tell smail sends an 
 email and does *not* change or append to the .out file of a job...
 
  Regards,
  Hermann
>>> 
>> 
> -- 
> Dr. Loris Bennett (Herr/Mr)
> 

Re: [slurm-users] How to debug a prolog script?

[Bjørn-Helge Mevik]

Davide DelVento  writes:

> Does it need the execution permission? For root alone sufficient?

slurmd runs as root, so it only need exec perms for root.

>> > 2. How to debug the issue?
>> I'd try capturing all stdout and stderr from the script into a file on the 
>> compute
>> node, for instance like this:
>>
>> exec &> /root/prolog_slurmd.$$
>> set -x # To print out all commands
>
> Do you mean INSIDE the prologue script itself?

Yes, inside the prolog script itself.

> Yes, this is what I'd have done, if it weren't so disruptive of all my
> production jobs, hence I had to turn it off before wrecking havoc too
> much.

I'm curious: What kind of disruption did it cause for your production
jobs?

We use this in our slurmd prologs (and similar in epilogs) on all our
production clusters, and have not seen any disruption due to it.  (We do
have things like

## Remove log file if we got this far:
rm -f /root/prolog_slurmd.$$

at the bottom of the scripts, though, so as to remove the log file when
the prolog succeeded.)

> Sure, but even "just executing" there is stdout and stderr which could
> be captured and logged rather than thrown away and force one to do the
> above.

True.  But slurmd doesn't, so...

> How do you "install the prolog scripts there"? Isn't the prolog
> setting in slurm.conf global?

I just overwrite the prolog script file itself on the node.  We
don't have them on a shared file system, though.  If you have the
prologs on a shared file system, you'd have to override the slurm config
on the compute node itself.  This can be done in several ways, for
instance by starting slurmd with the "-f "
option.

>> (Otherwise one could always
>> set up a small cluster of VMs and use that for simpler testing.)
>
> Yes, but I need to request that cluster of VM to IT, have the same OS
> installed and configured (and to be 100% identical, it needs to be
> RHEL so license paid), and everything sync'ed with the actual
> cluster I know it'd be very useful, but sadly we don't have the
> resources to do that, so unfortunately this is not an option for me.

I totally agree that VMs instead of a physical test cluster is never
going to be 100 % the same, but some things can be tested even though
the setups are not exactly the same (for instance, in my experience,
CentOS and Rocky are close enough to RHEL for most slurm-related
things).  One takes what one have. :)

-- 
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo



signature.asc
Description: PGP signature

Re: [slurm-users] Providing users with info on wait time vs. run time

[Loris Bennett]

Hi Hermann,

Hermann Schwärzler  writes:

> Hi Loris,
> hi Sebastian,
>
> thanks for the information on how you are doing this.
> So you both are happily(?) ignoring this warning the "Prolog and Epilog 
> Guide",
> right? :-)
>
> "Prolog and Epilog scripts [...] should not call Slurm commands (e.g. squeue,
> scontrol, sacctmgr, etc)."

We have probably been doing this since before the warning was added to
the documentation.  So we are "ignorantly ignoring" the advice :-/

> May I ask how big your clusters are (number of nodes) and how heavily they are
> used (submitted jobs per hour)?

We have around 190 32-core nodes.  I don't know how I would easily find
out the average number of jobs per hour.  The only problems we have had
with submission have been when people have written their own mechanisms
for submitting thousands of jobs.  Once we get them to use job array,
such problems generally disappear.

Cheers,

Loris

> Regards,
> Hermann
>
> On 9/16/22 9:09 AM, Loris Bennett wrote:
>> Hi Hermann,
>> Sebastian Potthoff  writes:
>> 
>>> Hi Hermann,
>>>
>>> I happened to read along this conversation and was just solving this issue 
>>> today. I added this part to the epilog script to make it work:
>>>
>>> # Add job report to stdout
>>> StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | 
>>> /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')
>>>
>>> NODELIST=($(/usr/bin/scontrol show hostnames))
>>>
>>> # Only add to StdOut file if it exists and if we are the first node
>>> if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
>>> then
>>>echo "# JOB REPORT 
>>> ##" >> $StdOut
>>>/usr/bin/seff $SLURM_JOB_ID >> $StdOut
>>>echo 
>>> "###"
>>>  >> $StdOut
>>> fi
>> We do something similar.  At the end of our script pointed to by
>> EpilogSlurmctld we have
>>OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
>>if [ ! -f "$OUT" ]; then
>>  exit
>>fi
>>printf "\n== Epilog Slurmctld
>> ==\n\n" >>  ${OUT}
>>seff ${SLURM_JOB_ID} >> ${OUT}
>>printf
>> "\n==\n"
>> >>  ${OUT}
>>chown ${user} ${OUT}
>> Cheers,
>> Loris
>> 
>>>Contrary to what it says in the slurm docs 
>>> https://slurm.schedmd.com/prolog_epilog.html  I was not able to use the env 
>>> var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I had 
>>> to
>>> make sure it is only called by the „leading“ node as the epilog script will 
>>> be called by ALL nodes of a multinode job and they would all call seff and 
>>> clutter up the output. Last thing was to check if StdOut is
>>> not of length zero (i.e. it exists). Interactive jobs would otherwise cause 
>>> the node to drain.
>>>
>>> Maybe this helps.
>>>
>>> Kind regards
>>> Sebastian
>>>
>>> PS: goslmailer looks quite nice with its recommendations! Will definitely 
>>> look into it.
>>>
>>> --
>>> Westfälische Wilhelms-Universität (WWU) Münster
>>> WWU IT
>>> Sebastian Potthoff (eScience / HPC)
>>>
>>>   Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
>>> :
>>>
>>>   Hi Ole,
>>>
>>>   On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>>>
>>>   On 15-09-2022 16:08, Hermann Schwärzler wrote:
>>>
>>>   Just out of curiosity: how do you insert the output of seff into the 
>>> out-file of a job?
>>>
>>>   Use the "smail" tool from the slurm-contribs RPM and set this in 
>>> slurm.conf:
>>>   MailProg=/usr/bin/smail
>>>
>>>   Maybe I am missing something but from what I can tell smail sends an 
>>> email and does *not* change or append to the .out file of a job...
>>>
>>>   Regards,
>>>   Hermann
>> 
>
-- 
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de

Re: [slurm-users] How to debug a prolog script?

[Davide DelVento]

Thanks to both of you.

> Permissions on the file itself (and the directories in the path to it)

Does it need the execution permission? For root alone sufficient?

> Existence of the script on the nodes (prologue is run on the nodes, not the 
> head)

Yes, it's in a shared filesystem.

> Not sure your error is the prologue script itself. Does everything run fine 
> with no prologue configured?

Yes, everything has been working fine for months and still does as
soon as I take the prolog out of slurm.conf.

> > 2. How to debug the issue?
> I'd try capturing all stdout and stderr from the script into a file on the 
> compute
> node, for instance like this:
>
> exec &> /root/prolog_slurmd.$$
> set -x # To print out all commands

Do you mean INSIDE the prologue script itself? Yes, this is what I'd
have done, if it weren't so disruptive of all my production jobs,
hence I had to turn it off before wrecking havoc too much.


> > Even increasing the debug level the
> > slurmctld.log contains simply a "error: validate_node_specs: Prolog or
> > job env setup failure on node xxx, draining the node" message, without
> > even a line number or anything.
>
> Slurm only executes the prolog script.  It doesn't parse it or evaluate
> it itself, so it has no way of knowing what fails inside the script.

Sure, but even "just executing" there is stdout and stderr which could
be captured and logged rather than thrown away and force one to do the
above.

> > 3. And more generally, how to debug a prolog (and epilog) script
> > without disrupting all production jobs? Unfortunately we can't have
> > another slurm install for testing, is there a sbatch option to force
> > utilizing a prolog script which would not be executed for all the
> > other jobs? Or perhaps making a dedicated queue?
>
> I tend to reserve a node, install the updated prolog scripts there, and
> run test jobs asking for that reservation.

How do you "install the prolog scripts there"? Isn't the prolog
setting in slurm.conf global?

> (Otherwise one could always
> set up a small cluster of VMs and use that for simpler testing.)

Yes, but I need to request that cluster of VM to IT, have the same OS
installed and configured (and to be 100% identical, it needs to be
RHEL so license paid), and everything sync'ed with the actual
cluster I know it'd be very useful, but sadly we don't have the
resources to do that, so unfortunately this is not an option for me.

Thanks again.

Re: [slurm-users] Providing users with info on wait time vs. run time

[Hermann Schwärzler]

Hi Loris,
hi Sebastian,

thanks for the information on how you are doing this.
So you both are happily(?) ignoring this warning the "Prolog and Epilog 
Guide", right? :-)


"Prolog and Epilog scripts [...] should not call Slurm commands (e.g. 
squeue, scontrol, sacctmgr, etc)."


May I ask how big your clusters are (number of nodes) and how heavily 
they are used (submitted jobs per hour)?


Regards,
Hermann

On 9/16/22 9:09 AM, Loris Bennett wrote:

Hi Hermann,

Sebastian Potthoff  writes:


Hi Hermann,

I happened to read along this conversation and was just solving this issue 
today. I added this part to the epilog script to make it work:

# Add job report to stdout
StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | /usr/bin/xargs 
| /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')

NODELIST=($(/usr/bin/scontrol show hostnames))

# Only add to StdOut file if it exists and if we are the first node
if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
then
   echo "# JOB REPORT 
##" >> $StdOut
   /usr/bin/seff $SLURM_JOB_ID >> $StdOut
   echo 
"###" 
>> $StdOut
fi


We do something similar.  At the end of our script pointed to by
EpilogSlurmctld we have

   OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
   if [ ! -f "$OUT" ]; then
 exit
   fi

   printf "\n== Epilog Slurmctld 
==\n\n" >>  ${OUT}

   seff ${SLURM_JOB_ID} >> ${OUT}

   printf 
"\n==\n" >> 
 ${OUT}

   chown ${user} ${OUT}

Cheers,

Loris


   Contrary to what it says in the slurm docs 
https://slurm.schedmd.com/prolog_epilog.html  I was not able to use the env var 
SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I had to
make sure it is only called by the „leading“ node as the epilog script will be 
called by ALL nodes of a multinode job and they would all call seff and clutter 
up the output. Last thing was to check if StdOut is
not of length zero (i.e. it exists). Interactive jobs would otherwise cause the 
node to drain.

Maybe this helps.

Kind regards
Sebastian

PS: goslmailer looks quite nice with its recommendations! Will definitely look 
into it.

--
Westfälische Wilhelms-Universität (WWU) Münster
WWU IT
Sebastian Potthoff (eScience / HPC)

  Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
:

  Hi Ole,

  On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:

  On 15-09-2022 16:08, Hermann Schwärzler wrote:

  Just out of curiosity: how do you insert the output of seff into the out-file 
of a job?

  Use the "smail" tool from the slurm-contribs RPM and set this in slurm.conf:
  MailProg=/usr/bin/smail

  Maybe I am missing something but from what I can tell smail sends an email 
and does *not* change or append to the .out file of a job...

  Regards,
  Hermann

Re: [slurm-users] remote license

[Davide DelVento]

So if I understand correctly, this "remote database" is something that
is neither part of slurm itself, nor part of the license server per
se, correct?

Regarding the "if you got creative", has anybody on this list done
that already? I can't believe I'm the first one wanting this feature!
Matching the number in that database with the actual number the
license server knows would be extremely helpful! We use various
license servers (for various licensed software), so each one of them
would be useful. I can probably script/develop one of these myself,
but I am not sure I've got the time...

Thanks!

On Thu, Sep 15, 2022 at 6:04 PM Brian Andrus  wrote:
>
> So if you follow the links to: https://slurm.schedmd.com/licenses.html
> you should see the difference.
>
> Local licenses are just a counter that is setup in slurm.conf
> Remote liceneses are a counter in a database (the database is "remote"),
> so you can change/update it dynamically. So, you could change their
> allocation with a sacctmgr command. It is especially useful when you are
> managing multiple clusters that share licenses. You can allocate that a
> certain number are allowed by each cluster and change that if needed.
>
> If you got creative, you could keep the license count that is in the
> database updated to match the number free from flexlm to stop license
> starvation due to users outside slurm using them up so they really
> aren't available to slurm.
>
> Brian Andrus
>
>
> On 9/15/2022 3:34 PM, Davide DelVento wrote:
> > I am a bit confused by remote licenses.
> >
> > https://lists.schedmd.com/pipermail/slurm-users/2020-September/006049.html
> > (which is only 2 years old) claims that they are just a counter, so
> > like local licenses. Then why call them remote?
> >
> > Only a few days after, this
> > https://lists.schedmd.com/pipermail/slurm-users/2020-September/006081.html
> > appeared to imply (but not clearly stated) that the remote license are
> > not simply a counter, but then it's not clear how they are different.
> >
> > The current documentation (and attempts to run the "add resource"
> > command) says that one must use the license count, which seems to
> > imply they are just a simple counter (but then what do they need the
> > server for)?
> >
> > So what is what?
> >
> > In my cursory past experience with this, it seemed that it were
> > possible to query a license server (at least some of them) to get the
> > actual number of available licenses and schedule (or let jobs pending)
> > accordingly. Which would be very helpful for the not-too-uncommon
> > situation in which the same license server provides licenses for both
> > the HPC cluster and other non-slurm-controlled resources, such a
> > user's workstations. Was that impression wrong, or perhaps somebody
> > scripted it in some way? If the latter, does anybody know if those
> > scripts are publicly available anywhere?
> >
> > Thanks
> >
>

Re: [slurm-users] Providing users with info on wait time vs. run time

[Loris Bennett]

Hi Sebastian,

Sebastian Potthoff  writes:

> Hi Loris
>
>  We do something similar.  At the end of our script pointed to by
>  EpilogSlurmctld we have
>
> Using EpilogSlurmctld only works if the slurmctld user is root (or slurm with 
> root privileges), right? I opted for the normal Epilog since we wanted to 
> avoid running slurm as root and I don’t have to worry
> about ownership of the output file.

Yes, good point.  We should look into that.

Cheers,

Loris


> Sebastian
>
>  Am 16.09.2022 um 09:09 schrieb Loris Bennett :
>
>  Hi Hermann,
>
>  Sebastian Potthoff  writes:
>
>  Hi Hermann,
>
>  I happened to read along this conversation and was just solving this issue 
> today. I added this part to the epilog script to make it work:
>
>  # Add job report to stdout
>  StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | 
> /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')
>
>  NODELIST=($(/usr/bin/scontrol show hostnames))
>
>  # Only add to StdOut file if it exists and if we are the first node
>  if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
>  then
>   echo "# JOB REPORT 
> ##" >> $StdOut
>   /usr/bin/seff $SLURM_JOB_ID >> $StdOut
>   echo 
> "###"
>  >> $StdOut
>  fi
>
>  We do something similar.  At the end of our script pointed to by
>  EpilogSlurmctld we have
>
>   OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
>   if [ ! -f "$OUT" ]; then
> exit
>   fi
>
>   printf "\n== Epilog Slurmctld 
> ==\n\n" >>  ${OUT}
>
>   seff ${SLURM_JOB_ID} >> ${OUT}
>
>   printf 
> "\n==\n" 
> >>  ${OUT}
>
>   chown ${user} ${OUT}
>
>  Cheers,
>
>  Loris
>
>   Contrary to what it says in the slurm docs 
> https://slurm.schedmd.com/prolog_epilog.html  I was not able to use the env 
> var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I
>  had to
>  make sure it is only called by the „leading“ node as the epilog script will 
> be called by ALL nodes of a multinode job and they would all call seff and 
> clutter up the output. Last thing was to check if
>  StdOut is
>  not of length zero (i.e. it exists). Interactive jobs would otherwise cause 
> the node to drain.
>
>  Maybe this helps. 
>
>  Kind regards
>  Sebastian
>
>  PS: goslmailer looks quite nice with its recommendations! Will definitely 
> look into it.
>
>  --
>  Westfälische Wilhelms-Universität (WWU) Münster
>  WWU IT
>  Sebastian Potthoff (eScience / HPC)
>
>  Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
> :
>
>  Hi Ole,
>
>  On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>
>  On 15-09-2022 16:08, Hermann Schwärzler wrote:
>
>  Just out of curiosity: how do you insert the output of seff into the 
> out-file of a job?
>
>  Use the "smail" tool from the slurm-contribs RPM and set this in slurm.conf:
>  MailProg=/usr/bin/smail
>
>  Maybe I am missing something but from what I can tell smail sends an email 
> and does *not* change or append to the .out file of a job...
>
>  Regards,
>  Hermann
>
-- 
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de

Re: [slurm-users] Providing users with info on wait time vs. run time

[Sebastian Potthoff]

Hi Loris

> We do something similar.  At the end of our script pointed to by
> EpilogSlurmctld we have

Using EpilogSlurmctld only works if the slurmctld user is root (or slurm with 
root privileges), right? I opted for the normal Epilog since we wanted to avoid 
running slurm as root and I don’t have to worry about ownership of the output 
file.

Sebastian


> Am 16.09.2022 um 09:09 schrieb Loris Bennett :
> 
> Hi Hermann,
> 
> Sebastian Potthoff  > writes:
> 
>> Hi Hermann,
>> 
>> I happened to read along this conversation and was just solving this issue 
>> today. I added this part to the epilog script to make it work:
>> 
>> # Add job report to stdout
>> StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | 
>> /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')
>> 
>> NODELIST=($(/usr/bin/scontrol show hostnames))
>> 
>> # Only add to StdOut file if it exists and if we are the first node
>> if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
>> then
>>  echo "# JOB REPORT 
>> ##" >> $StdOut
>>  /usr/bin/seff $SLURM_JOB_ID >> $StdOut
>>  echo 
>> "###"
>>  >> $StdOut
>> fi
> 
> We do something similar.  At the end of our script pointed to by
> EpilogSlurmctld we have
> 
>  OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
>  if [ ! -f "$OUT" ]; then
>exit
>  fi
> 
>  printf "\n== Epilog Slurmctld 
> ==\n\n" >>  ${OUT}
> 
>  seff ${SLURM_JOB_ID} >> ${OUT}
> 
>  printf 
> "\n==\n" 
> >>  ${OUT}
> 
>  chown ${user} ${OUT}
> 
> Cheers,
> 
> Loris
> 
>>  Contrary to what it says in the slurm docs 
>> https://slurm.schedmd.com/prolog_epilog.html  I was not able to use the env 
>> var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I had to
>> make sure it is only called by the „leading“ node as the epilog script will 
>> be called by ALL nodes of a multinode job and they would all call seff and 
>> clutter up the output. Last thing was to check if StdOut is
>> not of length zero (i.e. it exists). Interactive jobs would otherwise cause 
>> the node to drain.
>> 
>> Maybe this helps. 
>> 
>> Kind regards
>> Sebastian
>> 
>> PS: goslmailer looks quite nice with its recommendations! Will definitely 
>> look into it.
>> 
>> --
>> Westfälische Wilhelms-Universität (WWU) Münster
>> WWU IT
>> Sebastian Potthoff (eScience / HPC)
>> 
>> Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
>> :
>> 
>> Hi Ole,
>> 
>> On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>> 
>> On 15-09-2022 16:08, Hermann Schwärzler wrote:
>> 
>> Just out of curiosity: how do you insert the output of seff into the 
>> out-file of a job?
>> 
>> Use the "smail" tool from the slurm-contribs RPM and set this in slurm.conf:
>> MailProg=/usr/bin/smail
>> 
>> Maybe I am missing something but from what I can tell smail sends an email 
>> and does *not* change or append to the .out file of a job...
>> 
>> Regards,
>> Hermann



smime.p7s
Description: S/MIME cryptographic signature

Re: [slurm-users] How to debug a prolog script?

[Bjørn-Helge Mevik]

Davide DelVento  writes:

> 2. How to debug the issue?

I'd try capturing all stdout and stderr from the script into a file on the 
compute
node, for instance like this:

exec &> /root/prolog_slurmd.$$
set -x # To print out all commands

before any other commands in the script.  The "prolog_slurmd." will
then contain a log of all commands executed in the script, along with
all output (stdout and stderr).  If there is no "prolog_slurmd."
file after the job has been scheduled, then as has been pointed out by
others, slurm wasn't able to exec the prolog at all.

> Even increasing the debug level the
> slurmctld.log contains simply a "error: validate_node_specs: Prolog or
> job env setup failure on node xxx, draining the node" message, without
> even a line number or anything.

Slurm only executes the prolog script.  It doesn't parse it or evaluate
it itself, so it has no way of knowing what fails inside the script.

> 3. And more generally, how to debug a prolog (and epilog) script
> without disrupting all production jobs? Unfortunately we can't have
> another slurm install for testing, is there a sbatch option to force
> utilizing a prolog script which would not be executed for all the
> other jobs? Or perhaps making a dedicated queue?

I tend to reserve a node, install the updated prolog scripts there, and
run test jobs asking for that reservation.  (Otherwise one could always
set up a small cluster of VMs and use that for simpler testing.)

-- 
B/H


signature.asc
Description: PGP signature

Re: [slurm-users] Providing users with info on wait time vs. run time

[Loris Bennett]

Hi Hermann,

Sebastian Potthoff  writes:

> Hi Hermann,
>
> I happened to read along this conversation and was just solving this issue 
> today. I added this part to the epilog script to make it work:
>
> # Add job report to stdout
> StdOut=$(/usr/bin/scontrol show job=$SLURM_JOB_ID | /usr/bin/grep StdOut | 
> /usr/bin/xargs | /usr/bin/awk 'BEGIN { FS = "=" } ; { print $2 }')
>
> NODELIST=($(/usr/bin/scontrol show hostnames))
>
> # Only add to StdOut file if it exists and if we are the first node
> if [ "$(/usr/bin/hostname -s)" = "${NODELIST[0]}" -a ! -z "${StdOut}" ]
> then
>   echo "# JOB REPORT 
> ##" >> $StdOut
>   /usr/bin/seff $SLURM_JOB_ID >> $StdOut
>   echo 
> "###"
>  >> $StdOut
> fi

We do something similar.  At the end of our script pointed to by
EpilogSlurmctld we have

  OUT=`scontrol show jobid ${job_id} | awk -F= '/ StdOut/{print $2}'`
  if [ ! -f "$OUT" ]; then
exit
  fi

  printf "\n== Epilog Slurmctld 
==\n\n" >>  ${OUT}

  seff ${SLURM_JOB_ID} >> ${OUT}

  printf 
"\n==\n" >> 
 ${OUT}

  chown ${user} ${OUT}

Cheers,

Loris

>   Contrary to what it says in the slurm docs 
> https://slurm.schedmd.com/prolog_epilog.html  I was not able to use the env 
> var SLURM_JOB_STDOUT, so I had to fetch it via scontrol. In addition I had to
> make sure it is only called by the „leading“ node as the epilog script will 
> be called by ALL nodes of a multinode job and they would all call seff and 
> clutter up the output. Last thing was to check if StdOut is
> not of length zero (i.e. it exists). Interactive jobs would otherwise cause 
> the node to drain.
>
> Maybe this helps. 
>
> Kind regards
> Sebastian
>
> PS: goslmailer looks quite nice with its recommendations! Will definitely 
> look into it.
>
> --
> Westfälische Wilhelms-Universität (WWU) Münster
> WWU IT
> Sebastian Potthoff (eScience / HPC)
>
>  Am 15.09.2022 um 18:07 schrieb Hermann Schwärzler 
> :
>
>  Hi Ole,
>
>  On 9/15/22 5:21 PM, Ole Holm Nielsen wrote:
>
>  On 15-09-2022 16:08, Hermann Schwärzler wrote:
>
>  Just out of curiosity: how do you insert the output of seff into the 
> out-file of a job?
>
>  Use the "smail" tool from the slurm-contribs RPM and set this in slurm.conf:
>  MailProg=/usr/bin/smail
>
>  Maybe I am missing something but from what I can tell smail sends an email 
> and does *not* change or append to the .out file of a job...
>
>  Regards,
>  Hermann

Re: [slurm-users] Providing users with info on wait time vs. run time

[Ole Holm Nielsen]

Hi Hermann,

On 9/15/22 18:07, Hermann Schwärzler wrote:
Use the "smail" tool from the slurm-contribs RPM and set this in 
slurm.conf:


MailProg=/usr/bin/smail


Maybe I am missing something but from what I can tell smail sends an email 
and does *not* change or append to the .out file of a job...


Correct, smail sends usage information along with the other mail information:

$ grep SEFF /usr/bin/smail
SEFF=/usr/bin/seff
$SEFF $jobid | $MAIL -s "$subject" $recipient


/Ole