Re: [slurm-users] License management and invoking scontrol in the prolog
Did you install all prerequiste packages (including lua) on the server where you built the Slurm packages? On my system I get: $ strings `which slurmctld ` | grep HAVE_LUA HAVE_LUA 1 /Ole https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#install-prerequisites On 9/2/22 05:15, Davide DelVento wrote: Thanks. I did try a lua script as soon as I got your first email, but that never worked (yes, I enabled it in slurm.conf and ran "scontrol reconfigure" after). Slurm simply acted as if there was no job_submit script. After various tests, all unsuccessful, today I found that link which I mentioned saying that lua might not be compiled in, hence all my most recent messages of this thread. That file is indeed there, so that's good news that I don't need to recompile. However I'm puzzled on what might be missing... On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote: lua is the language you can use with the job_submit plugin. I was showing a quick way to see that job_submit capability is indeed in there. You can see if lua support is there by looking for the job_submit_lua.so file is there. It would be part of the slurm rpm (not the slurm-slurmctl rpm) Usually it would be found at /usr/lib64/slurm/job_submit_lua.so If that is there, you should be good with trying out a job_submit lua script. Brian Andrus On 9/1/2022 1:24 PM, Davide DelVento wrote: Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern
Re: [slurm-users] License management and invoking scontrol in the prolog
Try setting logging to debug mode, then you can get some info from the logs. Brian Andrus On 9/1/2022 8:15 PM, Davide DelVento wrote: Thanks. I did try a lua script as soon as I got your first email, but that never worked (yes, I enabled it in slurm.conf and ran "scontrol reconfigure" after). Slurm simply acted as if there was no job_submit script. After various tests, all unsuccessful, today I found that link which I mentioned saying that lua might not be compiled in, hence all my most recent messages of this thread. That file is indeed there, so that's good news that I don't need to recompile. However I'm puzzled on what might be missing... On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote: lua is the language you can use with the job_submit plugin. I was showing a quick way to see that job_submit capability is indeed in there. You can see if lua support is there by looking for the job_submit_lua.so file is there. It would be part of the slurm rpm (not the slurm-slurmctl rpm) Usually it would be found at /usr/lib64/slurm/job_submit_lua.so If that is there, you should be good with trying out a job_submit lua script. Brian Andrus On 9/1/2022 1:24 PM, Davide DelVento wrote: Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks. I did try a lua script as soon as I got your first email, but that never worked (yes, I enabled it in slurm.conf and ran "scontrol reconfigure" after). Slurm simply acted as if there was no job_submit script. After various tests, all unsuccessful, today I found that link which I mentioned saying that lua might not be compiled in, hence all my most recent messages of this thread. That file is indeed there, so that's good news that I don't need to recompile. However I'm puzzled on what might be missing... On Thu, Sep 1, 2022 at 6:33 PM Brian Andrus wrote: > > lua is the language you can use with the job_submit plugin. > > I was showing a quick way to see that job_submit capability is indeed in > there. > > You can see if lua support is there by looking for the job_submit_lua.so > file is there. > It would be part of the slurm rpm (not the slurm-slurmctl rpm) > > Usually it would be found at /usr/lib64/slurm/job_submit_lua.so > > If that is there, you should be good with trying out a job_submit lua > script. > > Brian Andrus > > On 9/1/2022 1:24 PM, Davide DelVento wrote: > > Thanks again, Brian, indeed that grep returns many hits, but none of > > them includes lua, i.e. > > > > strings `which slurmctld ` | grep -i job_submit | grep -i lua > > > > returns nothing. So I should use the C rather than the more convenient > > lua interface, unless I recompile or am I missing something? > > > > On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: > >> I would be surprised if it were compiled without the support. However, > >> you could check and run something like: > >> > >> strings /sbin/slurmctld | grep job_submit > >> > >> (or where ever your slurmctld binary is). There should be quite a few > >> lines with that in it. > >> > >> Brian Andrus > >> > >> On 9/1/2022 10:54 AM, Davide DelVento wrote: > >>> Thanks Brian for the suggestion, which I am now exploring. > >>> > >>> The documentation is a bit cryptic for me, but exploring a few things > >>> and checking > >>> https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ > >>> I suspect my slurm install (provided by cluster vendor) was not > >>> compiled with the lua plugin installed. Do you know how to verify if > >>> that is the case or if it's something else? I don't see a way to show > >>> if the plugin is actually being "seen" by slurm, and I suspect it's > >>> not. > >>> > >>> Does anyone else have other suggestions or comment on either the > >>> plugin or the prolog workaround? > >>> > >>> Thanks! > >>> > >>> > >>> On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: > Not sure if you can do all the things you intend, but the job_submit > script is precisely where you want to check submission options. > > https://slurm.schedmd.com/job_submit_plugins.html > > Brian Andrus > > On 8/30/2022 12:58 PM, Davide DelVento wrote: > > Hi, > > > > I would like to soft-enforce license utilization only when the -L is > > set. My idea: check in the prolog if the license was requested and > > only if it were, set the environmental variables needed for the > > license. > > > > I looked at all environmental variables set by slurm and did not find > > any related to the license as I was hoping. > > > > As a workaround, I could check > > > > scontrol show job $SLURM_JOB_ID | grep License > > > > and that would work, but (as discussed in other messages in this list) > > the documentation at https://slurm.schedmd.com/prolog_epilog.html say > > > >> Prolog and Epilog scripts should be designed to be as short as possible > >> and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, > >> etc). [...] Slurm commands in these scripts can potentially lead to > >> performance > >> issues and should not be used. > > This is a bit of a concern, since the prolog would be invoked for > > every job on the cluster, and it's a prolog (rather than the epilogue > > like discussed in earlier messages). > > > > So two questions: > > > > 1) is there a better workaround to check in the prolog if the current > > job requested a license and/or > > 2) would this kind of use of scontrol be okay or is indeed a concern > > > > Thanks! > > >
Re: [slurm-users] License management and invoking scontrol in the prolog
lua is the language you can use with the job_submit plugin. I was showing a quick way to see that job_submit capability is indeed in there. You can see if lua support is there by looking for the job_submit_lua.so file is there. It would be part of the slurm rpm (not the slurm-slurmctl rpm) Usually it would be found at /usr/lib64/slurm/job_submit_lua.so If that is there, you should be good with trying out a job_submit lua script. Brian Andrus On 9/1/2022 1:24 PM, Davide DelVento wrote: Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks again, Brian, indeed that grep returns many hits, but none of them includes lua, i.e. strings `which slurmctld ` | grep -i job_submit | grep -i lua returns nothing. So I should use the C rather than the more convenient lua interface, unless I recompile or am I missing something? On Thu, Sep 1, 2022 at 12:30 PM Brian Andrus wrote: > > I would be surprised if it were compiled without the support. However, > you could check and run something like: > > strings /sbin/slurmctld | grep job_submit > > (or where ever your slurmctld binary is). There should be quite a few > lines with that in it. > > Brian Andrus > > On 9/1/2022 10:54 AM, Davide DelVento wrote: > > Thanks Brian for the suggestion, which I am now exploring. > > > > The documentation is a bit cryptic for me, but exploring a few things > > and checking > > https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ > > I suspect my slurm install (provided by cluster vendor) was not > > compiled with the lua plugin installed. Do you know how to verify if > > that is the case or if it's something else? I don't see a way to show > > if the plugin is actually being "seen" by slurm, and I suspect it's > > not. > > > > Does anyone else have other suggestions or comment on either the > > plugin or the prolog workaround? > > > > Thanks! > > > > > > On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: > >> Not sure if you can do all the things you intend, but the job_submit > >> script is precisely where you want to check submission options. > >> > >> https://slurm.schedmd.com/job_submit_plugins.html > >> > >> Brian Andrus > >> > >> On 8/30/2022 12:58 PM, Davide DelVento wrote: > >>> Hi, > >>> > >>> I would like to soft-enforce license utilization only when the -L is > >>> set. My idea: check in the prolog if the license was requested and > >>> only if it were, set the environmental variables needed for the > >>> license. > >>> > >>> I looked at all environmental variables set by slurm and did not find > >>> any related to the license as I was hoping. > >>> > >>> As a workaround, I could check > >>> > >>> scontrol show job $SLURM_JOB_ID | grep License > >>> > >>> and that would work, but (as discussed in other messages in this list) > >>> the documentation at https://slurm.schedmd.com/prolog_epilog.html say > >>> > Prolog and Epilog scripts should be designed to be as short as possible > and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, > etc). [...] Slurm commands in these scripts can potentially lead to > performance > issues and should not be used. > >>> This is a bit of a concern, since the prolog would be invoked for > >>> every job on the cluster, and it's a prolog (rather than the epilogue > >>> like discussed in earlier messages). > >>> > >>> So two questions: > >>> > >>> 1) is there a better workaround to check in the prolog if the current > >>> job requested a license and/or > >>> 2) would this kind of use of scontrol be okay or is indeed a concern > >>> > >>> Thanks! > >>> >
Re: [slurm-users] License management and invoking scontrol in the prolog
I would be surprised if it were compiled without the support. However, you could check and run something like: strings /sbin/slurmctld | grep job_submit (or where ever your slurmctld binary is). There should be quite a few lines with that in it. Brian Andrus On 9/1/2022 10:54 AM, Davide DelVento wrote: Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: Not sure if you can do all the things you intend, but the job_submit script is precisely where you want to check submission options. https://slurm.schedmd.com/job_submit_plugins.html Brian Andrus On 8/30/2022 12:58 PM, Davide DelVento wrote: Hi, I would like to soft-enforce license utilization only when the -L is set. My idea: check in the prolog if the license was requested and only if it were, set the environmental variables needed for the license. I looked at all environmental variables set by slurm and did not find any related to the license as I was hoping. As a workaround, I could check scontrol show job $SLURM_JOB_ID | grep License and that would work, but (as discussed in other messages in this list) the documentation at https://slurm.schedmd.com/prolog_epilog.html say Prolog and Epilog scripts should be designed to be as short as possible and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, etc). [...] Slurm commands in these scripts can potentially lead to performance issues and should not be used. This is a bit of a concern, since the prolog would be invoked for every job on the cluster, and it's a prolog (rather than the epilogue like discussed in earlier messages). So two questions: 1) is there a better workaround to check in the prolog if the current job requested a license and/or 2) would this kind of use of scontrol be okay or is indeed a concern Thanks!
Re: [slurm-users] License management and invoking scontrol in the prolog
Thanks Brian for the suggestion, which I am now exploring. The documentation is a bit cryptic for me, but exploring a few things and checking https://funinit.wordpress.com/2018/06/07/how-to-use-job_submit_lua-with-slurm/ I suspect my slurm install (provided by cluster vendor) was not compiled with the lua plugin installed. Do you know how to verify if that is the case or if it's something else? I don't see a way to show if the plugin is actually being "seen" by slurm, and I suspect it's not. Does anyone else have other suggestions or comment on either the plugin or the prolog workaround? Thanks! On Tue, Aug 30, 2022 at 3:01 PM Brian Andrus wrote: > > Not sure if you can do all the things you intend, but the job_submit > script is precisely where you want to check submission options. > > https://slurm.schedmd.com/job_submit_plugins.html > > Brian Andrus > > On 8/30/2022 12:58 PM, Davide DelVento wrote: > > Hi, > > > > I would like to soft-enforce license utilization only when the -L is > > set. My idea: check in the prolog if the license was requested and > > only if it were, set the environmental variables needed for the > > license. > > > > I looked at all environmental variables set by slurm and did not find > > any related to the license as I was hoping. > > > > As a workaround, I could check > > > > scontrol show job $SLURM_JOB_ID | grep License > > > > and that would work, but (as discussed in other messages in this list) > > the documentation at https://slurm.schedmd.com/prolog_epilog.html say > > > >> Prolog and Epilog scripts should be designed to be as short as possible > >> and should not call Slurm commands (e.g. squeue, scontrol, sacctmgr, > >> etc). [...] Slurm commands in these scripts can potentially lead to > >> performance > >> issues and should not be used. > > This is a bit of a concern, since the prolog would be invoked for > > every job on the cluster, and it's a prolog (rather than the epilogue > > like discussed in earlier messages). > > > > So two questions: > > > > 1) is there a better workaround to check in the prolog if the current > > job requested a license and/or > > 2) would this kind of use of scontrol be okay or is indeed a concern > > > > Thanks! > > >
