Re: [slurm-users] License management and invoking scontrol in the prolog
For other poor souls coming to this conversation, here is the conclusion.
$ sbatch --version
slurm 21.08.5
$ # irrelevant parts omitted from copy-paste for brevity
$ cat /opt/slurm/job_submit.lua
log_prefix = 'slurm_job_submit'
function slurm_job_submit(job_desc, part_list, submit_uid)
slurm.log_info("LOOPING")
for key, value in pairs(job_desc) do
slurm.log_info("%s: key=%s value=%s", log_prefix, key, value)
end
slurm.log_info("END LOOPING")
slurm.log_info("%s: user %s(%u) job_name=%s", log_prefix,
job_desc.user_name, submit_uid, job_desc.name)
return slurm.SUCCESS
end
$ sudo tail /var/log/slurm/slurmctld.log
[2022-09-12T08:13:16.310] _slurm_rpc_submit_batch_job: JobId=16028
InitPrio=4294887729 usec=198
[2022-09-12T08:13:17.213] sched: Allocate JobId=16028
NodeList=co49svnode13 #CPUs=16 Partition=compute512
[2022-09-12T08:15:01.090] lua: LOOPING
[2022-09-12T08:15:01.090] lua: END LOOPING
[2022-09-12T08:15:01.090] lua: slurm_job_submit: user
ddvento(57002254) job_name=lic.slurm
[2022-09-12T08:15:01.091] _slurm_rpc_submit_batch_job: JobId=16029
InitPrio=4294887728 usec=204
[2022-09-12T08:15:01.289] sched/backfill: _start_job: Started
JobId=16029 in compute512 on co49svnode14
[2022-09-12T08:15:17.330] _job_complete: JobId=16028 WEXITSTATUS 0
[2022-09-12T08:15:17.330] _job_complete: JobId=16028 done
Note how the looping is ended. This is probably just my naiveness and
ignorance with how lua work.
In conclusion, I am now able to see job_desc.licenses for example with:
slurm.log_info("%s: licenses=%s", log_prefix, job_desc.licenses)
So now I need to just implement my logic.
Thank you everybody in this conversation for helping to sort out this
issue. Greatly appreciated!
On Thu, Sep 8, 2022 at 9:22 AM Davide DelVento wrote:
>
> Thanks Ole, for this clarification, this is very good to know.
>
> However, the problem is that the very example provided by slurm itself
> is the one that has the error. I removed the unpack part with the
> variable arguments and that fixed that part.
>
> Unfortunately, the job_desc table is always empty so the whole
> job_submit.lua seems like a moot point? Or the example is so outdated
> (given that it cannot even log correctly) that this is now performed
> in a different way??
> Davide
>
> On Thu, Sep 8, 2022 at 12:23 AM Ole Holm Nielsen
> wrote:
> >
> > Hi Davide,
> >
> > In your slurmctld log you see an entry "error: job_submit/lua:
> > /opt/slurm/job_submit.lua".
> >
> > What I think happens is that when slurmctld encounters an error in
> > job_submit.lua, it will revert to the last known good script cached by
> > slurmctld and ignore the file on disk from now on, even if it has been
> > corrected. An "scontrol reconfig" may make slurmctld reread the
> > job_submit.lua, please try it.
> >
> > I believe that this slurmctld behavior is undocumented at present. Please
> > see https://bugs.schedmd.com/show_bug.cgi?id=14472#c15 for a description:
> >
> > > And, if after the reconfigure, the job_submit.lua is wrong formatted (or
> > > missing), it will use the previous version of the script (which we have
> > > stored backup previously):
> >
> > /Ole
> >
> >
> > On 9/7/22 14:21, Davide DelVento wrote:
> > > Thanks Ole, your wiki page sheds some light on this mystery.
> > > Very frustrating that even the simple example provided in the release
> > > fails, and it fails at the most basic logging functionality.
> > >
> > > Note that "my" job_submit.lua is now the unmodified, slurm-provided
> > > one and that the luac command returns nothing in my case (this is
> > > Lua 5.3.4) so syntax seems correct?
> > >
> > > Yet the logs report the problem I mentioned rather than the actual
> > > content that the plugin is attempting to log.
> > >
> > > On Wed, Sep 7, 2022 at 2:13 AM Ole Holm Nielsen
> > > wrote:
> > >>
> > >> Hi Davide,
> > >>
> > >> I suggest that you check your job_submit.lua script with the LUA
> > >> compiler:
> > >>
> > >> luac -p /etc/slurm/job_submit.lua
> > >>
> > >> I have written some more details in my Wiki page
> > >> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
> > >>
> > >> Best regards,
> > >> Ole
> > >>
> > >> On 9/7/22 01:51, Davide DelVento wrote:
> > >>> Thanks again to both of you.
> > >>>
> > >>> I actually did not build Slurm myself, otherwise I'd keep extensive
> > >>> logs of what I did. Other people did, so I don't know. However, I get
> > >>> the same grep'ing results as yours.
> > >>>
> > >>> Looking at the logs reveals some info, but it's cryptic.
> > >>>
> > >>> [2022-09-06T17:33:56.513] debug3: job_submit/lua:
> > >>> slurm_lua_loadscript: skipping loading Lua script:
> > >>> /opt/slurm/job_submit.lua
> > >>> [2022-09-06T17:33:56.513] error: job_submit/lua:
> > >>> /opt/slurm/job_submit.lua: [string "slurm.user_msg
> > >>> (string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
> > >>> (no value)
> > >>>
> > >>> As you can see,
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks Ole, for this clarification, this is very good to know.
However, the problem is that the very example provided by slurm itself
is the one that has the error. I removed the unpack part with the
variable arguments and that fixed that part.
Unfortunately, the job_desc table is always empty so the whole
job_submit.lua seems like a moot point? Or the example is so outdated
(given that it cannot even log correctly) that this is now performed
in a different way??
Davide
On Thu, Sep 8, 2022 at 12:23 AM Ole Holm Nielsen
wrote:
>
> Hi Davide,
>
> In your slurmctld log you see an entry "error: job_submit/lua:
> /opt/slurm/job_submit.lua".
>
> What I think happens is that when slurmctld encounters an error in
> job_submit.lua, it will revert to the last known good script cached by
> slurmctld and ignore the file on disk from now on, even if it has been
> corrected. An "scontrol reconfig" may make slurmctld reread the
> job_submit.lua, please try it.
>
> I believe that this slurmctld behavior is undocumented at present. Please
> see https://bugs.schedmd.com/show_bug.cgi?id=14472#c15 for a description:
>
> > And, if after the reconfigure, the job_submit.lua is wrong formatted (or
> > missing), it will use the previous version of the script (which we have
> > stored backup previously):
>
> /Ole
>
>
> On 9/7/22 14:21, Davide DelVento wrote:
> > Thanks Ole, your wiki page sheds some light on this mystery.
> > Very frustrating that even the simple example provided in the release
> > fails, and it fails at the most basic logging functionality.
> >
> > Note that "my" job_submit.lua is now the unmodified, slurm-provided
> > one and that the luac command returns nothing in my case (this is
> > Lua 5.3.4) so syntax seems correct?
> >
> > Yet the logs report the problem I mentioned rather than the actual
> > content that the plugin is attempting to log.
> >
> > On Wed, Sep 7, 2022 at 2:13 AM Ole Holm Nielsen
> > wrote:
> >>
> >> Hi Davide,
> >>
> >> I suggest that you check your job_submit.lua script with the LUA compiler:
> >>
> >> luac -p /etc/slurm/job_submit.lua
> >>
> >> I have written some more details in my Wiki page
> >> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
> >>
> >> Best regards,
> >> Ole
> >>
> >> On 9/7/22 01:51, Davide DelVento wrote:
> >>> Thanks again to both of you.
> >>>
> >>> I actually did not build Slurm myself, otherwise I'd keep extensive
> >>> logs of what I did. Other people did, so I don't know. However, I get
> >>> the same grep'ing results as yours.
> >>>
> >>> Looking at the logs reveals some info, but it's cryptic.
> >>>
> >>> [2022-09-06T17:33:56.513] debug3: job_submit/lua:
> >>> slurm_lua_loadscript: skipping loading Lua script:
> >>> /opt/slurm/job_submit.lua
> >>> [2022-09-06T17:33:56.513] error: job_submit/lua:
> >>> /opt/slurm/job_submit.lua: [string "slurm.user_msg
> >>> (string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
> >>> (no value)
> >>>
> >>> As you can see, there is no line number and there is nothing like
> >>> user_msg in this code. There is indeed an "unpack" which is used in
> >>> the SchedMD-defined logging helper function which has a comment
> >>> "Implicit definition of arg was removed in Lua 5.2" and that's where I
> >>> speculate the error occurs.
> >>>
> >>> I should stress, this is with their own example, not my code. I guess
> >>> I could forgo the logging and move forward, but that won't probably
> >>> lead me very far.
> >>>
> >>> I am contemplating submitting a github issue about it? I did check
> >>> that the version of the job_submit.lua I have is the same currently in
> >>> the repo at
> >>> https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
> >>>
> >>> On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
> >>> wrote:
>
> Did you install all prerequiste packages (including lua) on the server
> where you built the Slurm packages?
>
> On my system I get:
>
> $ strings `which slurmctld ` | grep HAVE_LUA
> HAVE_LUA 1
>
> /Ole
>
> https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
>
> On 9/2/22 05:15, Davide DelVento wrote:
> > Thanks.
> >
> > I did try a lua script as soon as I got your first email, but that
> > never worked (yes, I enabled it in slurm.conf and ran "scontrol
> > reconfigure" after). Slurm simply acted as if there was no job_submit
> > script.
> >
> > After various tests, all unsuccessful, today I found that link which I
> > mentioned saying that lua might not be compiled in, hence all my most
> > recent messages of this thread.
> >
> > That file is indeed there, so that's good news that I don't need to
> > recompile.
> > However I'm puzzled on what might be missing...
> >
> >
> > On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote:
> >>
> >> lua is the language you can use with the job_submit
Re: [slurm-users] License management and invoking scontrol in the prolog
Hi Davide,
In your slurmctld log you see an entry "error: job_submit/lua:
/opt/slurm/job_submit.lua".
What I think happens is that when slurmctld encounters an error in
job_submit.lua, it will revert to the last known good script cached by
slurmctld and ignore the file on disk from now on, even if it has been
corrected. An "scontrol reconfig" may make slurmctld reread the
job_submit.lua, please try it.
I believe that this slurmctld behavior is undocumented at present. Please
see https://bugs.schedmd.com/show_bug.cgi?id=14472#c15 for a description:
And, if after the reconfigure, the job_submit.lua is wrong formatted (or
missing), it will use the previous version of the script (which we have stored
backup previously):
/Ole
On 9/7/22 14:21, Davide DelVento wrote:
Thanks Ole, your wiki page sheds some light on this mystery.
Very frustrating that even the simple example provided in the release
fails, and it fails at the most basic logging functionality.
Note that "my" job_submit.lua is now the unmodified, slurm-provided
one and that the luac command returns nothing in my case (this is
Lua 5.3.4) so syntax seems correct?
Yet the logs report the problem I mentioned rather than the actual
content that the plugin is attempting to log.
On Wed, Sep 7, 2022 at 2:13 AM Ole Holm Nielsen
wrote:
Hi Davide,
I suggest that you check your job_submit.lua script with the LUA compiler:
luac -p /etc/slurm/job_submit.lua
I have written some more details in my Wiki page
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
Best regards,
Ole
On 9/7/22 01:51, Davide DelVento wrote:
Thanks again to both of you.
I actually did not build Slurm myself, otherwise I'd keep extensive
logs of what I did. Other people did, so I don't know. However, I get
the same grep'ing results as yours.
Looking at the logs reveals some info, but it's cryptic.
[2022-09-06T17:33:56.513] debug3: job_submit/lua:
slurm_lua_loadscript: skipping loading Lua script:
/opt/slurm/job_submit.lua
[2022-09-06T17:33:56.513] error: job_submit/lua:
/opt/slurm/job_submit.lua: [string "slurm.user_msg
(string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
(no value)
As you can see, there is no line number and there is nothing like
user_msg in this code. There is indeed an "unpack" which is used in
the SchedMD-defined logging helper function which has a comment
"Implicit definition of arg was removed in Lua 5.2" and that's where I
speculate the error occurs.
I should stress, this is with their own example, not my code. I guess
I could forgo the logging and move forward, but that won't probably
lead me very far.
I am contemplating submitting a github issue about it? I did check
that the version of the job_submit.lua I have is the same currently in
the repo at
https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
wrote:
Did you install all prerequiste packages (including lua) on the server
where you built the Slurm packages?
On my system I get:
$ strings `which slurmctld ` | grep HAVE_LUA
HAVE_LUA 1
/Ole
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
On 9/2/22 05:15, Davide DelVento wrote:
Thanks.
I did try a lua script as soon as I got your first email, but that
never worked (yes, I enabled it in slurm.conf and ran "scontrol
reconfigure" after). Slurm simply acted as if there was no job_submit script.
After various tests, all unsuccessful, today I found that link which I
mentioned saying that lua might not be compiled in, hence all my most
recent messages of this thread.
That file is indeed there, so that's good news that I don't need to recompile.
However I'm puzzled on what might be missing...
On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote:
lua is the language you can use with the job_submit plugin.
I was showing a quick way to see that job_submit capability is indeed in
there.
You can see if lua support is there by looking for the job_submit_lua.so
file is there.
It would be part of the slurm rpm (not the slurm-slurmctl rpm)
Usually it would be found at /usr/lib64/slurm/job_submit_lua.so
If that is there, you should be good with trying out a job_submit lua
script.
Brian Andrus
On 9/1/2022 1:24 PM, Davide DelVento wrote:
Thanks again, Brian, indeed that grep returns many hits, but none of
them includes lua, i.e.
strings `which slurmctld ` | grep -i job_submit | grep -i lua
returns nothing. So I should use the C rather than the more convenient
lua interface, unless I recompile or am I missing something?
On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote:
I would be surprised if it were compiled without the support. However,
you could check and run something like:
strings /sbin/slurmctld | grep job_submit
(or where ever your slurmctld binary is). There should be quite a few
lines with that in it.
Brian Andrus
On 9/1/2022 10:54 AM, Davide DelVento wrote:
Re: [slurm-users] License management and invoking scontrol in the prolog
No, I never moved that file from the Linux cluster (and I do the
editing with VIM which warns me of that possible issue).
On Wed, Sep 7, 2022 at 11:19 AM Brian Andrus wrote:
>
> Possibly way off base, but did you happen to do any of the editing in
> Windows? Maybe running into the cr/lf issue for how windows saves text
> files?
>
> Brian Andrus
>
> On 9/7/2022 5:21 AM, Davide DelVento wrote:
> > Thanks Ole, your wiki page sheds some light on this mystery.
> > Very frustrating that even the simple example provided in the release
> > fails, and it fails at the most basic logging functionality.
> >
> > Note that "my" job_submit.lua is now the unmodified, slurm-provided
> > one and that the luac command returns nothing in my case (this is
> > Lua 5.3.4) so syntax seems correct?
> >
> > Yet the logs report the problem I mentioned rather than the actual
> > content that the plugin is attempting to log.
> >
> > On Wed, Sep 7, 2022 at 2:13 AM Ole Holm Nielsen
> > wrote:
> >> Hi Davide,
> >>
> >> I suggest that you check your job_submit.lua script with the LUA compiler:
> >>
> >> luac -p /etc/slurm/job_submit.lua
> >>
> >> I have written some more details in my Wiki page
> >> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
> >>
> >> Best regards,
> >> Ole
> >>
> >> On 9/7/22 01:51, Davide DelVento wrote:
> >>> Thanks again to both of you.
> >>>
> >>> I actually did not build Slurm myself, otherwise I'd keep extensive
> >>> logs of what I did. Other people did, so I don't know. However, I get
> >>> the same grep'ing results as yours.
> >>>
> >>> Looking at the logs reveals some info, but it's cryptic.
> >>>
> >>> [2022-09-06T17:33:56.513] debug3: job_submit/lua:
> >>> slurm_lua_loadscript: skipping loading Lua script:
> >>> /opt/slurm/job_submit.lua
> >>> [2022-09-06T17:33:56.513] error: job_submit/lua:
> >>> /opt/slurm/job_submit.lua: [string "slurm.user_msg
> >>> (string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
> >>> (no value)
> >>>
> >>> As you can see, there is no line number and there is nothing like
> >>> user_msg in this code. There is indeed an "unpack" which is used in
> >>> the SchedMD-defined logging helper function which has a comment
> >>> "Implicit definition of arg was removed in Lua 5.2" and that's where I
> >>> speculate the error occurs.
> >>>
> >>> I should stress, this is with their own example, not my code. I guess
> >>> I could forgo the logging and move forward, but that won't probably
> >>> lead me very far.
> >>>
> >>> I am contemplating submitting a github issue about it? I did check
> >>> that the version of the job_submit.lua I have is the same currently in
> >>> the repo at
> >>> https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
> >>>
> >>> On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
> >>> wrote:
> Did you install all prerequiste packages (including lua) on the server
> where you built the Slurm packages?
>
> On my system I get:
>
> $ strings `which slurmctld ` | grep HAVE_LUA
> HAVE_LUA 1
>
> /Ole
>
> https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
>
> On 9/2/22 05:15, Davide DelVento wrote:
> > Thanks.
> >
> > I did try a lua script as soon as I got your first email, but that
> > never worked (yes, I enabled it in slurm.conf and ran "scontrol
> > reconfigure" after). Slurm simply acted as if there was no job_submit
> > script.
> >
> > After various tests, all unsuccessful, today I found that link which I
> > mentioned saying that lua might not be compiled in, hence all my most
> > recent messages of this thread.
> >
> > That file is indeed there, so that's good news that I don't need to
> > recompile.
> > However I'm puzzled on what might be missing...
> >
> >
> > On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote:
> >> lua is the language you can use with the job_submit plugin.
> >>
> >> I was showing a quick way to see that job_submit capability is indeed
> >> in
> >> there.
> >>
> >> You can see if lua support is there by looking for the
> >> job_submit_lua.so
> >> file is there.
> >> It would be part of the slurm rpm (not the slurm-slurmctl rpm)
> >>
> >> Usually it would be found at /usr/lib64/slurm/job_submit_lua.so
> >>
> >> If that is there, you should be good with trying out a job_submit lua
> >> script.
> >>
> >> Brian Andrus
> >>
> >> On 9/1/2022 1:24 PM, Davide DelVento wrote:
> >>> Thanks again, Brian, indeed that grep returns many hits, but none of
> >>> them includes lua, i.e.
> >>>
> >>> strings `which slurmctld ` | grep -i job_submit | grep -i lua
> >>>
> >>> returns nothing. So I should use the C rather than the more convenient
> >>> lua interface, unless I recompile or am I missing something?
> >>>
> >>> On
Re: [slurm-users] License management and invoking scontrol in the prolog
Possibly way off base, but did you happen to do any of the editing in
Windows? Maybe running into the cr/lf issue for how windows saves text
files?
Brian Andrus
On 9/7/2022 5:21 AM, Davide DelVento wrote:
Thanks Ole, your wiki page sheds some light on this mystery.
Very frustrating that even the simple example provided in the release
fails, and it fails at the most basic logging functionality.
Note that "my" job_submit.lua is now the unmodified, slurm-provided
one and that the luac command returns nothing in my case (this is
Lua 5.3.4) so syntax seems correct?
Yet the logs report the problem I mentioned rather than the actual
content that the plugin is attempting to log.
On Wed, Sep 7, 2022 at 2:13 AM Ole Holm Nielsen
wrote:
Hi Davide,
I suggest that you check your job_submit.lua script with the LUA compiler:
luac -p /etc/slurm/job_submit.lua
I have written some more details in my Wiki page
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
Best regards,
Ole
On 9/7/22 01:51, Davide DelVento wrote:
Thanks again to both of you.
I actually did not build Slurm myself, otherwise I'd keep extensive
logs of what I did. Other people did, so I don't know. However, I get
the same grep'ing results as yours.
Looking at the logs reveals some info, but it's cryptic.
[2022-09-06T17:33:56.513] debug3: job_submit/lua:
slurm_lua_loadscript: skipping loading Lua script:
/opt/slurm/job_submit.lua
[2022-09-06T17:33:56.513] error: job_submit/lua:
/opt/slurm/job_submit.lua: [string "slurm.user_msg
(string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
(no value)
As you can see, there is no line number and there is nothing like
user_msg in this code. There is indeed an "unpack" which is used in
the SchedMD-defined logging helper function which has a comment
"Implicit definition of arg was removed in Lua 5.2" and that's where I
speculate the error occurs.
I should stress, this is with their own example, not my code. I guess
I could forgo the logging and move forward, but that won't probably
lead me very far.
I am contemplating submitting a github issue about it? I did check
that the version of the job_submit.lua I have is the same currently in
the repo at
https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
wrote:
Did you install all prerequiste packages (including lua) on the server
where you built the Slurm packages?
On my system I get:
$ strings `which slurmctld ` | grep HAVE_LUA
HAVE_LUA 1
/Ole
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
On 9/2/22 05:15, Davide DelVento wrote:
Thanks.
I did try a lua script as soon as I got your first email, but that
never worked (yes, I enabled it in slurm.conf and ran "scontrol
reconfigure" after). Slurm simply acted as if there was no job_submit script.
After various tests, all unsuccessful, today I found that link which I
mentioned saying that lua might not be compiled in, hence all my most
recent messages of this thread.
That file is indeed there, so that's good news that I don't need to recompile.
However I'm puzzled on what might be missing...
On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote:
lua is the language you can use with the job_submit plugin.
I was showing a quick way to see that job_submit capability is indeed in
there.
You can see if lua support is there by looking for the job_submit_lua.so
file is there.
It would be part of the slurm rpm (not the slurm-slurmctl rpm)
Usually it would be found at /usr/lib64/slurm/job_submit_lua.so
If that is there, you should be good with trying out a job_submit lua
script.
Brian Andrus
On 9/1/2022 1:24 PM, Davide DelVento wrote:
Thanks again, Brian, indeed that grep returns many hits, but none of
them includes lua, i.e.
strings `which slurmctld ` | grep -i job_submit | grep -i lua
returns nothing. So I should use the C rather than the more convenient
lua interface, unless I recompile or am I missing something?
On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote:
I would be surprised if it were compiled without the support. However,
you could check and run something like:
strings /sbin/slurmctld | grep job_submit
(or where ever your slurmctld binary is). There should be quite a few
lines with that in it.
Brian Andrus
On 9/1/2022 10:54 AM, Davide DelVento wrote:
Thanks Brian for the suggestion, which I am now exploring.
The documentation is a bit cryptic for me, but exploring a few things
and checking
https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/
I suspect my slurm install (provided by cluster vendor) was not
compiled with the lua plugin installed. Do you know how to verify if
that is the case or if it's something else? I don't see a way to show
if the plugin is actually being "seen" by slurm, and I suspect it's
not.
Does anyone else have other suggestions or comment on either the
plugin or the prolog
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks Ole, your wiki page sheds some light on this mystery.
Very frustrating that even the simple example provided in the release
fails, and it fails at the most basic logging functionality.
Note that "my" job_submit.lua is now the unmodified, slurm-provided
one and that the luac command returns nothing in my case (this is
Lua 5.3.4) so syntax seems correct?
Yet the logs report the problem I mentioned rather than the actual
content that the plugin is attempting to log.
On Wed, Sep 7, 2022 at 2:13 AM Ole Holm Nielsen
wrote:
>
> Hi Davide,
>
> I suggest that you check your job_submit.lua script with the LUA compiler:
>
> luac -p /etc/slurm/job_submit.lua
>
> I have written some more details in my Wiki page
> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
>
> Best regards,
> Ole
>
> On 9/7/22 01:51, Davide DelVento wrote:
> > Thanks again to both of you.
> >
> > I actually did not build Slurm myself, otherwise I'd keep extensive
> > logs of what I did. Other people did, so I don't know. However, I get
> > the same grep'ing results as yours.
> >
> > Looking at the logs reveals some info, but it's cryptic.
> >
> > [2022-09-06T17:33:56.513] debug3: job_submit/lua:
> > slurm_lua_loadscript: skipping loading Lua script:
> > /opt/slurm/job_submit.lua
> > [2022-09-06T17:33:56.513] error: job_submit/lua:
> > /opt/slurm/job_submit.lua: [string "slurm.user_msg
> > (string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
> > (no value)
> >
> > As you can see, there is no line number and there is nothing like
> > user_msg in this code. There is indeed an "unpack" which is used in
> > the SchedMD-defined logging helper function which has a comment
> > "Implicit definition of arg was removed in Lua 5.2" and that's where I
> > speculate the error occurs.
> >
> > I should stress, this is with their own example, not my code. I guess
> > I could forgo the logging and move forward, but that won't probably
> > lead me very far.
> >
> > I am contemplating submitting a github issue about it? I did check
> > that the version of the job_submit.lua I have is the same currently in
> > the repo at
> > https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
> >
> > On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
> > wrote:
> >>
> >> Did you install all prerequiste packages (including lua) on the server
> >> where you built the Slurm packages?
> >>
> >> On my system I get:
> >>
> >> $ strings `which slurmctld ` | grep HAVE_LUA
> >> HAVE_LUA 1
> >>
> >> /Ole
> >>
> >> https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
> >>
> >> On 9/2/22 05:15, Davide DelVento wrote:
> >>> Thanks.
> >>>
> >>> I did try a lua script as soon as I got your first email, but that
> >>> never worked (yes, I enabled it in slurm.conf and ran "scontrol
> >>> reconfigure" after). Slurm simply acted as if there was no job_submit
> >>> script.
> >>>
> >>> After various tests, all unsuccessful, today I found that link which I
> >>> mentioned saying that lua might not be compiled in, hence all my most
> >>> recent messages of this thread.
> >>>
> >>> That file is indeed there, so that's good news that I don't need to
> >>> recompile.
> >>> However I'm puzzled on what might be missing...
> >>>
> >>>
> >>> On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote:
>
> lua is the language you can use with the job_submit plugin.
>
> I was showing a quick way to see that job_submit capability is indeed in
> there.
>
> You can see if lua support is there by looking for the job_submit_lua.so
> file is there.
> It would be part of the slurm rpm (not the slurm-slurmctl rpm)
>
> Usually it would be found at /usr/lib64/slurm/job_submit_lua.so
>
> If that is there, you should be good with trying out a job_submit lua
> script.
>
> Brian Andrus
>
> On 9/1/2022 1:24 PM, Davide DelVento wrote:
> > Thanks again, Brian, indeed that grep returns many hits, but none of
> > them includes lua, i.e.
> >
> > strings `which slurmctld ` | grep -i job_submit | grep -i lua
> >
> > returns nothing. So I should use the C rather than the more convenient
> > lua interface, unless I recompile or am I missing something?
> >
> > On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus
> > wrote:
> >> I would be surprised if it were compiled without the support. However,
> >> you could check and run something like:
> >>
> >> strings /sbin/slurmctld | grep job_submit
> >>
> >> (or where ever your slurmctld binary is). There should be quite a few
> >> lines with that in it.
> >>
> >> Brian Andrus
> >>
> >> On 9/1/2022 10:54 AM, Davide DelVento wrote:
> >>> Thanks Brian for the suggestion, which I am now exploring.
> >>>
> >>> The documentation is a bit cryptic for me, but exploring a few things
> >>> and checking
> >>>
Re: [slurm-users] License management and invoking scontrol in the prolog
Hi Davide,
I suggest that you check your job_submit.lua script with the LUA compiler:
luac -p /etc/slurm/job_submit.lua
I have written some more details in my Wiki page
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#job-submit-plugins
Best regards,
Ole
On 9/7/22 01:51, Davide DelVento wrote:
Thanks again to both of you.
I actually did not build Slurm myself, otherwise I'd keep extensive
logs of what I did. Other people did, so I don't know. However, I get
the same grep'ing results as yours.
Looking at the logs reveals some info, but it's cryptic.
[2022-09-06T17:33:56.513] debug3: job_submit/lua:
slurm_lua_loadscript: skipping loading Lua script:
/opt/slurm/job_submit.lua
[2022-09-06T17:33:56.513] error: job_submit/lua:
/opt/slurm/job_submit.lua: [string "slurm.user_msg
(string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
(no value)
As you can see, there is no line number and there is nothing like
user_msg in this code. There is indeed an "unpack" which is used in
the SchedMD-defined logging helper function which has a comment
"Implicit definition of arg was removed in Lua 5.2" and that's where I
speculate the error occurs.
I should stress, this is with their own example, not my code. I guess
I could forgo the logging and move forward, but that won't probably
lead me very far.
I am contemplating submitting a github issue about it? I did check
that the version of the job_submit.lua I have is the same currently in
the repo at
https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
wrote:
Did you install all prerequiste packages (including lua) on the server
where you built the Slurm packages?
On my system I get:
$ strings `which slurmctld ` | grep HAVE_LUA
HAVE_LUA 1
/Ole
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
On 9/2/22 05:15, Davide DelVento wrote:
Thanks.
I did try a lua script as soon as I got your first email, but that
never worked (yes, I enabled it in slurm.conf and ran "scontrol
reconfigure" after). Slurm simply acted as if there was no job_submit script.
After various tests, all unsuccessful, today I found that link which I
mentioned saying that lua might not be compiled in, hence all my most
recent messages of this thread.
That file is indeed there, so that's good news that I don't need to recompile.
However I'm puzzled on what might be missing...
On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote:
lua is the language you can use with the job_submit plugin.
I was showing a quick way to see that job_submit capability is indeed in
there.
You can see if lua support is there by looking for the job_submit_lua.so
file is there.
It would be part of the slurm rpm (not the slurm-slurmctl rpm)
Usually it would be found at /usr/lib64/slurm/job_submit_lua.so
If that is there, you should be good with trying out a job_submit lua
script.
Brian Andrus
On 9/1/2022 1:24 PM, Davide DelVento wrote:
Thanks again, Brian, indeed that grep returns many hits, but none of
them includes lua, i.e.
strings `which slurmctld ` | grep -i job_submit | grep -i lua
returns nothing. So I should use the C rather than the more convenient
lua interface, unless I recompile or am I missing something?
On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote:
I would be surprised if it were compiled without the support. However,
you could check and run something like:
strings /sbin/slurmctld | grep job_submit
(or where ever your slurmctld binary is). There should be quite a few
lines with that in it.
Brian Andrus
On 9/1/2022 10:54 AM, Davide DelVento wrote:
Thanks Brian for the suggestion, which I am now exploring.
The documentation is a bit cryptic for me, but exploring a few things
and checking
https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/
I suspect my slurm install (provided by cluster vendor) was not
compiled with the lua plugin installed. Do you know how to verify if
that is the case or if it's something else? I don't see a way to show
if the plugin is actually being "seen" by slurm, and I suspect it's
not.
Does anyone else have other suggestions or comment on either the
plugin or the prolog workaround?
Thanks!
On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote:
Not sure if you can do all the things you intend, but the job_submit
script is precisely where you want to check submission options.
https://slurm.schedmd.com/job_submit_plugins.html
Brian Andrus
On 8/30/2022 12:58 PM, Davide DelVento wrote:
Hi,
I would like to soft-enforce license utilization only when the -L is
set. My idea: check in the prolog if the license was requested and
only if it were, set the environmental variables needed for the
license.
I looked at all environmental variables set by slurm and did not find
any related to the license as I was hoping.
As a workaround, I could check
scontrol show job $SLURM_JOB_ID | grep License
and that would
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks again to both of you.
I actually did not build Slurm myself, otherwise I'd keep extensive
logs of what I did. Other people did, so I don't know. However, I get
the same grep'ing results as yours.
Looking at the logs reveals some info, but it's cryptic.
[2022-09-06T17:33:56.513] debug3: job_submit/lua:
slurm_lua_loadscript: skipping loading Lua script:
/opt/slurm/job_submit.lua
[2022-09-06T17:33:56.513] error: job_submit/lua:
/opt/slurm/job_submit.lua: [string "slurm.user_msg
(string.format(table.unpack({...})))"]:1: bad argument #2 to 'format'
(no value)
As you can see, there is no line number and there is nothing like
user_msg in this code. There is indeed an "unpack" which is used in
the SchedMD-defined logging helper function which has a comment
"Implicit definition of arg was removed in Lua 5.2" and that's where I
speculate the error occurs.
I should stress, this is with their own example, not my code. I guess
I could forgo the logging and move forward, but that won't probably
lead me very far.
I am contemplating submitting a github issue about it? I did check
that the version of the job_submit.lua I have is the same currently in
the repo at
https://github.com/SchedMD/slurm/blob/master/etc/job_submit.lua.example
On Thu, Sep 1, 2022 at 11:55 PM Ole Holm Nielsen
wrote:
>
> Did you install all prerequiste packages (including lua) on the server
> where you built the Slurm packages?
>
> On my system I get:
>
> $ strings `which slurmctld ` | grep HAVE_LUA
> HAVE_LUA 1
>
> /Ole
>
> https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites
>
> On 9/2/22 05:15, Davide DelVento wrote:
> > Thanks.
> >
> > I did try a lua script as soon as I got your first email, but that
> > never worked (yes, I enabled it in slurm.conf and ran "scontrol
> > reconfigure" after). Slurm simply acted as if there was no job_submit
> > script.
> >
> > After various tests, all unsuccessful, today I found that link which I
> > mentioned saying that lua might not be compiled in, hence all my most
> > recent messages of this thread.
> >
> > That file is indeed there, so that's good news that I don't need to
> > recompile.
> > However I'm puzzled on what might be missing...
> >
> >
> > On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote:
> >>
> >> lua is the language you can use with the job_submit plugin.
> >>
> >> I was showing a quick way to see that job_submit capability is indeed in
> >> there.
> >>
> >> You can see if lua support is there by looking for the job_submit_lua.so
> >> file is there.
> >> It would be part of the slurm rpm (not the slurm-slurmctl rpm)
> >>
> >> Usually it would be found at /usr/lib64/slurm/job_submit_lua.so
> >>
> >> If that is there, you should be good with trying out a job_submit lua
> >> script.
> >>
> >> Brian Andrus
> >>
> >> On 9/1/2022 1:24 PM, Davide DelVento wrote:
> >>> Thanks again, Brian, indeed that grep returns many hits, but none of
> >>> them includes lua, i.e.
> >>>
> >>>strings `which slurmctld ` | grep -i job_submit | grep -i lua
> >>>
> >>> returns nothing. So I should use the C rather than the more convenient
> >>> lua interface, unless I recompile or am I missing something?
> >>>
> >>> On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote:
> I would be surprised if it were compiled without the support. However,
> you could check and run something like:
>
> strings /sbin/slurmctld | grep job_submit
>
> (or where ever your slurmctld binary is). There should be quite a few
> lines with that in it.
>
> Brian Andrus
>
> On 9/1/2022 10:54 AM, Davide DelVento wrote:
> > Thanks Brian for the suggestion, which I am now exploring.
> >
> > The documentation is a bit cryptic for me, but exploring a few things
> > and checking
> > https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/
> > I suspect my slurm install (provided by cluster vendor) was not
> > compiled with the lua plugin installed. Do you know how to verify if
> > that is the case or if it's something else? I don't see a way to show
> > if the plugin is actually being "seen" by slurm, and I suspect it's
> > not.
> >
> > Does anyone else have other suggestions or comment on either the
> > plugin or the prolog workaround?
> >
> > Thanks!
> >
> >
> > On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus
> > wrote:
> >> Not sure if you can do all the things you intend, but the job_submit
> >> script is precisely where you want to check submission options.
> >>
> >> https://slurm.schedmd.com/job_submit_plugins.html
> >>
> >> Brian Andrus
> >>
> >> On 8/30/2022 12:58 PM, Davide DelVento wrote:
> >>> Hi,
> >>>
> >>> I would like to soft-enforce license utilization only when the -L is
> >>> set. My idea: check in the prolog if the license was requested and
> >>> only if it were, set the environmental
Re: [slurm-users] License management and invoking scontrol in the prolog
Did you install all prerequiste packages (including lua) on the server where you built the Slurm packages? On my system I get: $ strings `which slurmctld ` | grep HAVE_LUA HAVE_LUA 1 /Ole https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites On 9/2/22 05:15, Davide DelVento wrote: Thanks. I did try a lua script as soon as I got your first email, but that never worked (yes, I enabled it in slurm.conf and ran "scontrol reconfigure" after). Slurm simply acted as if there was no job_submit script. After various tests, all unsuccessful, today I found that link which I mentioned saying that lua might not be compiled in, hence all my most recent messages of this thread. That file is indeed there, so that's good news that I don't need to recompile. However I'm puzzled on what might be missing... On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote: lua is the language you can use with the job_submit plugin. I was showing a quick way to see that job_submit capability is indeed in there. You can see if lua support is there by looking for the job_submit_lua.so file is there. It would be part of the slurm rpm (not the slurm-slurmctl rpm) Usually it would be found at /usr/lib64/slurm/job_submit_lua.so If that is there, you should be good with trying out a job_submit lua script. Brian Andrus On 9/1/2022 1:24 PM, Davide DelVento wrote: Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern
Re: [slurm-users] License management and invoking scontrol in the prolog
Try setting logging to debug mode, then you can get some info from the logs. Brian Andrus On 9/1/2022 8:15 PM, Davide DelVento wrote: Thanks. I did try a lua script as soon as I got your first email, but that never worked (yes, I enabled it in slurm.conf and ran "scontrol reconfigure" after). Slurm simply acted as if there was no job_submit script. After various tests, all unsuccessful, today I found that link which I mentioned saying that lua might not be compiled in, hence all my most recent messages of this thread. That file is indeed there, so that's good news that I don't need to recompile. However I'm puzzled on what might be missing... On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote: lua is the language you can use with the job_submit plugin. I was showing a quick way to see that job_submit capability is indeed in there. You can see if lua support is there by looking for the job_submit_lua.so file is there. It would be part of the slurm rpm (not the slurm-slurmctl rpm) Usually it would be found at /usr/lib64/slurm/job_submit_lua.so If that is there, you should be good with trying out a job_submit lua script. Brian Andrus On 9/1/2022 1:24 PM, Davide DelVento wrote: Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks. I did try a lua script as soon as I got your first email, but that never worked (yes, I enabled it in slurm.conf and ran "scontrol reconfigure" after). Slurm simply acted as if there was no job_submit script. After various tests, all unsuccessful, today I found that link which I mentioned saying that lua might not be compiled in, hence all my most recent messages of this thread. That file is indeed there, so that's good news that I don't need to recompile. However I'm puzzled on what might be missing... On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote: > > lua is the language you can use with the job_submit plugin. > > I was showing a quick way to see that job_submit capability is indeed in > there. > > You can see if lua support is there by looking for the job_submit_lua.so > file is there. > It would be part of the slurm rpm (not the slurm-slurmctl rpm) > > Usually it would be found at /usr/lib64/slurm/job_submit_lua.so > > If that is there, you should be good with trying out a job_submit lua > script. > > Brian Andrus > > On 9/1/2022 1:24 PM, Davide DelVento wrote: > > Thanks again, Brian, indeed that grep returns many hits, but none of > > them includes lua, i.e. > > > > strings `which slurmctld ` | grep -i job_submit | grep -i lua > > > > returns nothing. So I should use the C rather than the more convenient > > lua interface, unless I recompile or am I missing something? > > > > On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: > >> I would be surprised if it were compiled without the support. However, > >> you could check and run something like: > >> > >> strings /sbin/slurmctld | grep job_submit > >> > >> (or where ever your slurmctld binary is). There should be quite a few > >> lines with that in it. > >> > >> Brian Andrus > >> > >> On 9/1/2022 10:54 AM, Davide DelVento wrote: > >>> Thanks Brian for the suggestion, which I am now exploring. > >>> > >>> The documentation is a bit cryptic for me, but exploring a few things > >>> and checking > >>> https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ > >>> I suspect my slurm install (provided by cluster vendor) was not > >>> compiled with the lua plugin installed. Do you know how to verify if > >>> that is the case or if it's something else? I don't see a way to show > >>> if the plugin is actually being "seen" by slurm, and I suspect it's > >>> not. > >>> > >>> Does anyone else have other suggestions or comment on either the > >>> plugin or the prolog workaround? > >>> > >>> Thanks! > >>> > >>> > >>> On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: > Not sure if you can do all the things you intend, but the job_submit > script is precisely where you want to check submission options. > > https://slurm.schedmd.com/job_submit_plugins.html > > Brian Andrus > > On 8/30/2022 12:58 PM, Davide DelVento wrote: > > Hi, > > > > I would like to soft-enforce license utilization only when the -L is > > set. My idea: check in the prolog if the license was requested and > > only if it were, set the environmental variables needed for the > > license. > > > > I looked at all environmental variables set by slurm and did not find > > any related to the license as I was hoping. > > > > As a workaround, I could check > > > > scontrol show job $SLURM_JOB_ID | grep License > > > > and that would work, but (as discussed in other messages in this list) > > the documentation at https://slurm.schedmd.com/prolog_epilog.html say > > > >> Prolog and Epilog scripts should be designed to be as short as possible > >> and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, > >> etc). [...] Slurm commands in these scripts can potentially lead to > >> performance > >> issues and should not be used. > > This is a bit of a concern, since the prolog would be invoked for > > every job on the cluster, and it's a prolog (rather than the epilogue > > like discussed in earlier messages). > > > > So two questions: > > > > 1) is there a better workaround to check in the prolog if the current > > job requested a license and/or > > 2) would this kind of use of scontrol be okay or is indeed a concern > > > > Thanks! > > >
Re: [slurm-users] License management and invoking scontrol in the prolog
lua is the language you can use with the job_submit plugin. I was showing a quick way to see that job_submit capability is indeed in there. You can see if lua support is there by looking for the job_submit_lua.so file is there. It would be part of the slurm rpm (not the slurm-slurmctl rpm) Usually it would be found at /usr/lib64/slurm/job_submit_lua.so If that is there, you should be good with trying out a job_submit lua script. Brian Andrus On 9/1/2022 1:24 PM, Davide DelVento wrote: Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: > > I would be surprised if it were compiled without the support. However, > you could check and run something like: > > strings /sbin/slurmctld | grep job_submit > > (or where ever your slurmctld binary is). There should be quite a few > lines with that in it. > > Brian Andrus > > On 9/1/2022 10:54 AM, Davide DelVento wrote: > > Thanks Brian for the suggestion, which I am now exploring. > > > > The documentation is a bit cryptic for me, but exploring a few things > > and checking > > https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ > > I suspect my slurm install (provided by cluster vendor) was not > > compiled with the lua plugin installed. Do you know how to verify if > > that is the case or if it's something else? I don't see a way to show > > if the plugin is actually being "seen" by slurm, and I suspect it's > > not. > > > > Does anyone else have other suggestions or comment on either the > > plugin or the prolog workaround? > > > > Thanks! > > > > > > On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: > >> Not sure if you can do all the things you intend, but the job_submit > >> script is precisely where you want to check submission options. > >> > >> https://slurm.schedmd.com/job_submit_plugins.html > >> > >> Brian Andrus > >> > >> On 8/30/2022 12:58 PM, Davide DelVento wrote: > >>> Hi, > >>> > >>> I would like to soft-enforce license utilization only when the -L is > >>> set. My idea: check in the prolog if the license was requested and > >>> only if it were, set the environmental variables needed for the > >>> license. > >>> > >>> I looked at all environmental variables set by slurm and did not find > >>> any related to the license as I was hoping. > >>> > >>> As a workaround, I could check > >>> > >>> scontrol show job $SLURM_JOB_ID | grep License > >>> > >>> and that would work, but (as discussed in other messages in this list) > >>> the documentation at https://slurm.schedmd.com/prolog_epilog.html say > >>> > Prolog and Epilog scripts should be designed to be as short as possible > and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, > etc). [...] Slurm commands in these scripts can potentially lead to > performance > issues and should not be used. > >>> This is a bit of a concern, since the prolog would be invoked for > >>> every job on the cluster, and it's a prolog (rather than the epilogue > >>> like discussed in earlier messages). > >>> > >>> So two questions: > >>> > >>> 1) is there a better workaround to check in the prolog if the current > >>> job requested a license and/or > >>> 2) would this kind of use of scontrol be okay or is indeed a concern > >>> > >>> Thanks! > >>> >
Re: [slurm-users] License management and invoking scontrol in the prolog
I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: > > Not sure if you can do all the things you intend, but the job_submit > script is precisely where you want to check submission options. > > https://slurm.schedmd.com/job_submit_plugins.html > > Brian Andrus > > On 8/30/2022 12:58 PM, Davide DelVento wrote: > > Hi, > > > > I would like to soft-enforce license utilization only when the -L is > > set. My idea: check in the prolog if the license was requested and > > only if it were, set the environmental variables needed for the > > license. > > > > I looked at all environmental variables set by slurm and did not find > > any related to the license as I was hoping. > > > > As a workaround, I could check > > > > scontrol show job $SLURM_JOB_ID | grep License > > > > and that would work, but (as discussed in other messages in this list) > > the documentation at https://slurm.schedmd.com/prolog_epilog.html say > > > >> Prolog and Epilog scripts should be designed to be as short as possible > >> and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, > >> etc). [...] Slurm commands in these scripts can potentially lead to > >> performance > >> issues and should not be used. > > This is a bit of a concern, since the prolog would be invoked for > > every job on the cluster, and it's a prolog (rather than the epilogue > > like discussed in earlier messages). > > > > So two questions: > > > > 1) is there a better workaround to check in the prolog if the current > > job requested a license and/or > > 2) would this kind of use of scontrol be okay or is indeed a concern > > > > Thanks! > > >
Re: [slurm-users] License management and invoking scontrol in the prolog
Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
[slurm-users] License management and invoking scontrol in the prolog
Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say > Prolog and Epilog scripts should be designed to be as short as possible > and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, > etc). [...] Slurm commands in these scripts can potentially lead to > performance > issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
