[slurm-users] Re: errors compiling Slurm 18 on RHEL 9: [Makefile:577: scancel] Error 1 & It's not recommended to have unversioned Obsoletes
Slurm 18? Isn't that a bit outdated?
On Fri, Sep 27, 2024 at 9:41 AM Robert Kudyba via slurm-users <
slurm-users@lists.schedmd.com> wrote:
> We're in the process of upgrading but first we're moving to RHEL 9. My
> attempt to compile using rpmbuild -v -ta --define "_lto_cflags %{nil}"
> slurm-18.08.9.tar.bz2 (H/T to Brian for this flag
> ).
> I've stumped Google and the Slurm mailing list with the scancel error so
> hoping someone here knows of a work around.
>
> /bin/ld:
> opt.o:/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel/../../src/scancel/scancel.h:78:
> multiple definition of `opt';
> scancel.o:/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel/../../src/scancel/scancel.h:78:
> first defined here
> collect2: error: ld returned 1 exit status
> make[3]: *** [Makefile:577: scancel] Error 1
> make[3]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9/src/scancel'
> make[2]: *** [Makefile:563: all-recursive] Error 1
> make[2]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9/src'
> make[1]: *** [Makefile:690: all-recursive] Error 1
> make[1]: Leaving directory '/root/rpmbuild/BUILD/slurm-18.08.9'
> make: *** [Makefile:589: all] Error 2
> error: Bad exit status from /var/tmp/rpm-tmp.jhiGyR (%build)
>
>
> RPM build errors:
> Macro expanded in comment on line 22: %_prefix path install path for
> commands, libraries, etc.
>
> line 70: It's not recommended to have unversioned Obsoletes:
> Obsoletes: slurm-lua slurm-munge slurm-plugins
> Macro expanded in comment on line 158: %define
> _unpackaged_files_terminate_build 0
>
> line 224: It's not recommended to have unversioned Obsoletes:
> Obsoletes: slurm-sql
> line 256: It's not recommended to have unversioned Obsoletes:
> Obsoletes: slurm-sjobexit slurm-sjstat slurm-seff
> line 275: It's not recommended to have unversioned Obsoletes:
> Obsoletes: pam_slurm
> Bad exit status from /var/tmp/rpm-tmp.jhiGyR (%build)
>
> #!/bin/sh
>
> RPM_SOURCE_DIR="/root"
> RPM_BUILD_DIR="/root/rpmbuild/BUILD"
> RPM_OPT_FLAGS="-O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall
> -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64
> -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables
> -fstack-clash-protection -fcf-protection"
> RPM_LD_FLAGS="-Wl,-z,relro -Wl,--as-needed "-Wl,-z,lazy"
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 "
> RPM_ARCH="x86_64"
> RPM_OS="linux"
> RPM_BUILD_NCPUS="48"
> export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_LD_FLAGS RPM_ARCH
> RPM_OS RPM_BUILD_NCPUS RPM_LD_FLAGS
> RPM_DOC_DIR="/usr/share/doc"
> export RPM_DOC_DIR
> RPM_PACKAGE_NAME="slurm"
> RPM_PACKAGE_VERSION="18.08.9"
> RPM_PACKAGE_RELEASE="1.el9"
> export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE
> LANG=C
> export LANG
> unset CDPATH DISPLAY ||:
> RPM_BUILD_ROOT="/root/rpmbuild/BUILDROOT/slurm-18.08.9-1.el9.x86_64"
> export RPM_BUILD_ROOT
>
>
> PKG_CONFIG_PATH="${PKG_CONFIG_PATH}:/usr/lib64/pkgconfig:/usr/share/pkgconfig"
> export PKG_CONFIG_PATH
> CONFIG_SITE=${CONFIG_SITE:-NONE}
> export CONFIG_SITE
>
> set -x
> umask 022
> cd "/root/rpmbuild/BUILD"
> cd 'slurm-18.08.9'
>
>
> CFLAGS="${CFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall
> -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64
> -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables
> -fstack-clash-protection -fcf-protection}" ; export CFLAGS ;
> CXXFLAGS="${CXXFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe
> -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy"
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2
> -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
> -fcf-protection}" ; export CXXFLAGS ;
> FFLAGS="${FFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall
> -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS
> "-Wl,-z,lazy" -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64
> -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables
> -fstack-clash-protection -fcf-protection -I/usr/lib64/gfortran/modules}" ;
> export FFLAGS ;
> FCFLAGS="${FCFLAGS:--O2 -fexceptions -g -grecord-gcc-switches -pipe
> -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> -Wp,-D_GLIBCXX_ASSERTIONS "-Wl,-z,lazy"
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2
> -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection
> -fcf-protection -I/usr/lib64/gfortran/modules}" ; export FCFLAGS ;
> LDFLAGS="${LDFLAGS:--Wl,-z,relro -Wl,--as-needed "-Wl,-z,lazy"
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 }" ; export LDFLAGS ;
> LT_SYS_LIBRARY_PATH="${LT_SYS_LIBRARY_PATH:-/usr/lib64:}" ; expor
[slurm-users] Re: Print Slurm Stats on Login
Thanks everybody once again and especially Paul: your job_summary script
was exactly what I needed, served on a golden plate. I just had to
modify/customize the date range and change the following line (I can make a
PR if you want, but it's such a small change that it'd take more time to
deal with the PR than just typing it)
-Timelimit =
time_to_float(Timelimit.replace('UNLIMITED','365-00:00:00'))
+Timelimit =
time_to_float(Timelimit.replace('UNLIMITED','365-00:00:00').replace('Partition_Limit','365-00:00:00'))
Cheers,
Davide
On Tue, Aug 27, 2024 at 1:40 PM Paul Edmon via slurm-users <
slurm-users@lists.schedmd.com> wrote:
> This thread when a bunch of different directions. However I ran with
> Jeffrey's suggestion and wrote up a profile.d script along with other
> supporting scripts to pull the data. The setup I put together is here
> for the community to use as they see fit:
>
> https://github.com/fasrc/puppet-slurm_stats
>
> While this is written as a puppet module the scripts there in can be
> used by anyone as its a pretty straightforward set up and the templates
> have obvious places to do a find and replace.
>
> Naturally I'm happy to take additional merge requests. Thanks for all
> the interesting conversation about this. Lots of great ideas.
>
> -Paul Edmon-
>
> On 8/9/24 12:04 PM, Jeffrey T Frey wrote:
> > You'd have to do this within e.g. the system's bashrc infrastructure.
> The simplest idea would be to add to e.g. /etc/profile.d/zzz-slurmstats.sh
> and have some canned commands/scripts running. That does introduce load to
> the system and Slurm on every login, though, and slows the startup of login
> shells based on how responsive slurmctld/slurmdbd are at that moment.
> >
> > Another option would be to run the commands/scripts for all users on
> some timed schedule — e.g. produce per-user stats every 30 minutes. So
> long as the stats are publicly-visible anyway, put those summaries in a
> shared file system with open read access. Name the files by uid number.
> Now your /etc/profile.d script just cat's ${STATS_DIR}/$(id -u).
> >
> >
> >
> >
> >> On Aug 9, 2024, at 11:11, Paul Edmon via slurm-users <
> slurm-users@lists.schedmd.com> wrote:
> >>
> >> We are working to make our users more aware of their usage. One of the
> ideas we came up with was to having some basic usage stats printed at login
> (usage over past day, fairshare, job efficiency, etc). Does anyone have any
> scripts or methods that they use to do this? Before baking my own I was
> curious what other sites do and if they would be willing to share their
> scripts and methodology.
> >>
> >> -Paul Edmon-
> >>
> >>
> >> --
> >> slurm-users mailing list -- slurm-users@lists.schedmd.com
> >> To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
>
> --
> slurm-users mailing list -- slurm-users@lists.schedmd.com
> To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
>
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Spread a multistep job across clusters
Ciao Fabio, That for sure is syntactically incorrect, because the way sbatch parsing works: as soon as it finds a non-empy non-comment line (your first srun) it will stop parsing for #SBATCH directives. So assuming this is a single file as it looks from the formatting, the second hetjob and the cluster3 are ignored. Now, if these are two separate files, they would be two separate jobs, so that's not going to work either. More specifically to your question, I can't help because I don't have experience with federated clusters. On Mon, Aug 26, 2024 at 9:43 AM Di Bernardini, Fabio via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hi everyone, for accounting reasons, I need to create only one job across > two or more federated clusters with two or more srun steps. > > I’m trying with hetjobs but it's not clear to me from the documentation ( > https://slurm.schedmd.com/heterogeneous_jobs.html) if this is possible > and how to do it. > > I'm trying with this script, but the steps are executed on only the first > cluster. > > Can you tell me if there is a mistake in the hetjob or if it has to be > done in another way? > > > > #!/bin/bash > > > > #SBATCH hetjob > > #SBATCH --clusters=cluster2 > > srun -v --het-group=0 hostname > > > > #SBATCH hetjob > > #SBATCH --clusters=cluster3 > > srun -v --het-group=1 hostname > > > > NICE SRL, viale Monte Grappa 3/5, 20124 Milano, Italia, Registro delle > Imprese di Milano Monza Brianza Lodi REA n. 2096882, Capitale Sociale: > 10.329,14 EUR i.v., Cod. Fisc. e P.IVA 01133050052, Societa con Socio Unico > > > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Slurmdbd purge and reported downtime
Thanks Ole, this is very helpful. I was unaware of that issue. From the bug report it's not clear to me if it was just a sreport (display) issue, or if the problem was in the way the data was stored. In fact I am running 23.11.5 which I installed in April. The numbers I see for the last few months (including April) are fine. The earlier numbers (when I was running an earlier version) are the ones affected by this problem. So if the issue was the way the data was stored, that explains it and I can live with it (even if I can't provide an accurate report for my management now) knowing that the problem won't happen again in the future. Thanks and have a great weekend On Fri, Aug 23, 2024 at 8:00 AM Ole Holm Nielsen via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hi Davide, > > On 8/22/24 21:30, Davide DelVento via slurm-users wrote: > > I am confused by the reported amount of Down and PLND Down by sreport. > > According to it, our cluster would have had a significant amount of > > downtime, which I know didn't happen (or, according to the documentation > > "time that slurmctld was not responding", see > > https://slurm.schedmd.com/sreport.html > > <https://slurm.schedmd.com/sreport.html>) > > > > Could it be my purge settings causing this problem? How can I check > (maybe > > in some logs, maybe in the future) if actually slurmctld was not > > responding? The expected long-term numbers should be less than the ones > > reported for last month when we had an issue with a few nodes > > Which version of Slurm are you using? There was an sreport bug that > should be fixed in 23.11: > https://support.schedmd.com/show_bug.cgi?id=17689 > > /Ole > > > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Slurmdbd purge and reported downtime
I am confused by the reported amount of Down and PLND Down by sreport. According to it, our cluster would have had a significant amount of downtime, which I know didn't happen (or, according to the documentation "time that slurmctld was not responding", see https://slurm.schedmd.com/sreport.html) Could it be my purge settings causing this problem? How can I check (maybe in some logs, maybe in the future) if actually slurmctld was not responding? The expected long-term numbers should be less than the ones reported for last month when we had an issue with a few nodes Thanks! [davide@login ~]$ grep Purge /opt/slurm/slurmdbd.conf #JobPurge=12 #StepPurge=1 PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/22 Cluster Utilization 2022-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total Cluster TRES Name AllocatedDown PLND Down Idle Planned Reported - -- --- --- -- - clustercpu 19.50% 12.07% 3.92% 64.36% 0.15% 100.03% clustermem 16.13% 13.17% 4.56% 66.13% 0.00% 99.99% [davide@login ~]$sreport -t percent -T cpu,mem cluster utilization start=2/1/23 Cluster Utilization 2023-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total Cluster TRES Name AllocatedDown PLND Down Idle Planned Reported - -- --- --- -- --- - clustercpu 28.74% 18.80% 6.44% 45.77% 0.24% 100.02% clustermem 22.52% 20.54% 7.38% 49.55% 0.00% 99.98% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=2/1/24 Cluster Utilization 2024-02-01T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total Cluster TRES Name AllocatedDown PLND Down Idle PlannedReported - -- -- --- -- --- --- clustercpu 29.92% 24.88% 17.73% 27.45%0.02% 100.00% clustermem 20.07% 28.60% 19.57% 31.76%0.00% 100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=8/8/24 Cluster Utilization 2024-08-08T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported - -- - -- -- clustercpu15.96%2.53%0.00% 81.51% 0.00%100.00% clustermem 9.18%2.22%0.00% 88.60% 0.00%100.00% [davide@login ~]$ sreport -t percent -T cpu,mem cluster utilization start=7/7/24 Cluster Utilization 2024-07-07T00:00:00 - 2024-08-21T23:59:59 Usage reported in Percentage of Total Cluster TRES Name Allocated Down PLND Dow Idle Planned Reported - -- -- - -- -- clustercpu 27.07% 2.57%0.00% 70.34%0.02%100.00% clustermem 17.35% 2.26%0.00% 80.40%0.00%100.00% -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Print Slurm Stats on Login
Hi Ole, On Wed, Aug 21, 2024 at 1:06 PM Ole Holm Nielsen via slurm-users < slurm-users@lists.schedmd.com> wrote: > The slurmacct script can actually break down statistics by partition, > which I guess is what you're asking for? The usage of the command is: > Yes, this is almost what I was asking for. And admittedly I now realize that with perhaps some minor algebra (using the TOTAL-all line) I could get what I need. What confused me is that running it from everything or one partition reported the same beginning, rather than a partition-specific beginning: [davide ~]$ slurmacct -s 0101 -e 0202 Start date 0101 End date 0202 Report generated to file /tmp/Slurm_report_acct_0101_0202 [davide ~]$ cat /tmp/Slurm_report_acct_0101_0202 Cluster Utilization 01-Jan-2024_00:00 - 01-Feb-2024_23:59 Usage reported in Percentage of Total Cluster Allocated Down PLND Dow Idle Planned Reported - -- -- - -- cluster 23.25% 67.85%0.00% 8.89%0.01%100.00% Usage sorted by top users: (omitted) [davide ~]$ slurmacct -s 0101 -e 0202 Start date 0101 End date 0202 Print only accounting in Slurm partition gpu Report generated to file /tmp/Slurm_report_acct_0101_0202 [davide ~]$ cat /tmp/Slurm_report_acct_0101_0202 Cluster Utilization 01-Jan-2024_00:00 - 01-Feb-2024_23:59 Usage reported in Percentage of Total Cluster Allocated Down PLND Dow Idle Planned Reported - -- -- - -- cluster 23.25% 67.85%0.00% 8.89%0.01%100.00% Partition selected: gpu Usage sorted by top users: (omitted) Also, what you label "Wallclock hours" in the table of users is actually core-hours? Not even node-hours, otherwise I am reading things incorrectly. The Start_time and End_time values specify the date/time interval of > job completion/termination (see "man sacct"). > > Hint: Specify Start/End time as MMDD (Month and Date) > Small suggestion: change this to Hint: Specify Start/End time as MMDD (Month and Day) or as MMDDYY (Month and Day and Year) since sreport accepts it and your tool appears to otherwise understand that format. > > > - statistics about wait-in-queue for jobs, due to unavailable > > resources > > The slurmacct report prints "Average q-hours" (starttime minus submittime). > Ahaha! That's it! Super useful, I was wondering what "q" was (wait-in-Queue, I guess). You are super. We use the "topreports" script to gather weekly, monthly and yearly > reports (using slurmacct) for management (professors at our university). > I knew that I must not have been the only one with this need ;-) Thanks again! -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Print Slurm Stats on Login
Thanks, Ole! Your tools and what you do for the community is fantastic, we all appreciate you! Of course, I did look (and use) your script. But I need more info. And no, this is not something that users would run *ever* (let alone at every login). This is something I *myself* (the cluster administrator) need to run, once a quarter, or perhaps even just once a year, to inform my managers of cluster utilization to keep them apprised on the status of the affairs, and justify change in funding for future hardware purchases. Sorry for not making this clear, given the initial message I replied to. Thanks for any suggestion you might have. On Wed, Aug 21, 2024 at 12:19 AM Ole Holm Nielsen via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hi Davide, > > Did you already check out what the slurmacct script can do for you? See > > https://github.com/OleHolmNielsen/Slurm_tools/blob/master/slurmacct/slurmacct > > What you're asking for seems like a pretty heavy task regarding system > resources and Slurm database requests. You don't imagine this to run > every time a user makes a login shell? Some users might run "bash -l" > inside jobs to emulate a login session, causing a heavy load on your > servers. > > /Ole > > On 8/21/24 01:13, Davide DelVento via slurm-users wrote: > > Thanks Kevin and Simon, > > > > The full thing that you do is indeed overkill, however I was able to > learn > > how to collect/parse some of the information I need. > > > > What I am still unable to get is: > > > > - utilization by queue (or list of node names), to track actual use of > > expensive resources such as GPUs, high memory nodes, etc > > - statistics about wait-in-queue for jobs, due to unavailable resources > > > > hopefully both in a sreport-like format by user and by overall system > > > > I suspect this information is available in sacct, but needs some > > massaging/consolidation to become useful for what I am looking for. > > Perhaps either (or both) of your scripts already do that in some place > > that I did not find? That would be terrific, and I'd appreciate it if > you > > can point me to its place. > > > > Thanks again! > > > > On Tue, Aug 20, 2024 at 9:09 AM Kevin Broch via slurm-users > > mailto:slurm-users@lists.schedmd.com>> > wrote: > > > > Heavyweight solution (although if you have grafana and prometheus > > going already a little less so): > > https://github.com/rivosinc/prometheus-slurm-exporter > > <https://github.com/rivosinc/prometheus-slurm-exporter> > > > > On Tue, Aug 20, 2024 at 12:40 AM Simon Andrews via slurm-users > > mailto:slurm-users@lists.schedmd.com > >> > > wrote: > > > > Possibly a bit more elaborate than you want but I wrote a web > > based monitoring system for our cluster. It mostly uses standard > > slurm commands for job monitoring, but I've also added storage > > monitoring which requires a separate cron job to run every > night. > > It was written for our cluster, but probably wouldn't take much > > work to adapt to another cluster with similar structure. > > > > You can see the code and some screenshots at: > > > > https://github.com/s-andrews/capstone_monitor > > <https://github.com/s-andrews/capstone_monitor> > > > > ..and there's a video walk through at: > > > > https://vimeo.com/982985174 <https://vimeo.com/982985174> > > > > We've also got more friendly scripts for monitoring current and > > past jobs on the command line. These are in a private repository > > as some of the other information there is more sensitive but I'm > > happy to share those scripts. You can see the scripts being used > > in https://vimeo.com/982986202 <https://vimeo.com/982986202> > > > > Simon. > > > > -Original Message- > > From: Paul Edmon via slurm-users > <mailto:slurm-users@lists.schedmd.com>> > > Sent: 09 August 2024 16:12 > > To: slurm-users@lists.schedmd.com > > <mailto:slurm-users@lists.schedmd.com> > > Subject: [slurm-users] Print Slurm Stats on Login > > > > We are working to make our users more aware of their usage. One > of > > the ideas we came up with was to having some basic usage stats > > printed at login (usage over pa
[slurm-users] Re: Print Slurm Stats on Login
Thanks Kevin and Simon, The full thing that you do is indeed overkill, however I was able to learn how to collect/parse some of the information I need. What I am still unable to get is: - utilization by queue (or list of node names), to track actual use of expensive resources such as GPUs, high memory nodes, etc - statistics about wait-in-queue for jobs, due to unavailable resources hopefully both in a sreport-like format by user and by overall system I suspect this information is available in sacct, but needs some massaging/consolidation to become useful for what I am looking for. Perhaps either (or both) of your scripts already do that in some place that I did not find? That would be terrific, and I'd appreciate it if you can point me to its place. Thanks again! On Tue, Aug 20, 2024 at 9:09 AM Kevin Broch via slurm-users < slurm-users@lists.schedmd.com> wrote: > Heavyweight solution (although if you have grafana and prometheus going > already a little less so): > https://github.com/rivosinc/prometheus-slurm-exporter > > On Tue, Aug 20, 2024 at 12:40 AM Simon Andrews via slurm-users < > slurm-users@lists.schedmd.com> wrote: > >> Possibly a bit more elaborate than you want but I wrote a web based >> monitoring system for our cluster. It mostly uses standard slurm commands >> for job monitoring, but I've also added storage monitoring which requires a >> separate cron job to run every night. It was written for our cluster, but >> probably wouldn't take much work to adapt to another cluster with similar >> structure. >> >> You can see the code and some screenshots at: >> >> https://github.com/s-andrews/capstone_monitor >> >> ..and there's a video walk through at: >> >> https://vimeo.com/982985174 >> >> We've also got more friendly scripts for monitoring current and past jobs >> on the command line. These are in a private repository as some of the >> other information there is more sensitive but I'm happy to share those >> scripts. You can see the scripts being used in >> https://vimeo.com/982986202 >> >> Simon. >> >> -Original Message- >> From: Paul Edmon via slurm-users >> Sent: 09 August 2024 16:12 >> To: slurm-users@lists.schedmd.com >> Subject: [slurm-users] Print Slurm Stats on Login >> >> We are working to make our users more aware of their usage. One of the >> ideas we came up with was to having some basic usage stats printed at login >> (usage over past day, fairshare, job efficiency, etc). Does anyone have any >> scripts or methods that they use to do this? Before baking my own I was >> curious what other sites do and if they would be willing to share their >> scripts and methodology. >> >> -Paul Edmon- >> >> >> -- >> slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe >> send an email to slurm-users-le...@lists.schedmd.com >> >> >> This email has been scanned for spam & viruses. If you believe this email >> should have been stopped by our filters, click the following link to report >> it ( >> https://portal-uk.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1zaW1vbi5hbmRyZXdzQGJhYnJhaGFtLmFjLnVrO3RzPTE3MjMyMTY5MzA7dXVpZD02NkI2MzQyMTY5MzU2Q0YwRThDQzI5RTY4MkMxOEY5Mjt0b2tlbj01MjI1ZmJmYzJjODgzNWM3ZDE2ZGRiOTE2ZjIxYzk4MjliMjY2MjA0Ow%3D%3D >> ). >> >> -- >> slurm-users mailing list -- slurm-users@lists.schedmd.com >> To unsubscribe send an email to slurm-users-le...@lists.schedmd.com >> > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Unable to run sequential jobs simultaneously on the same node
Since each instance of the program is independent and you are using one core for each, it'd be better to leave slurm deal with that and schedule them concurrently as it sees fit. Maybe you simply need to add some directive to allow shared jobs on the same node. Alternatively (if at your site jobs must be exclusive) you have to check what it is their recommended way to perform this. Some sites prefer dask, some other an MPI-based serial-job consolidation (often called "command file") some others a technique similar to what you are doing, but instead of reinventing the wheel I suggest to check what your site recommends in this situation On Mon, Aug 19, 2024 at 2:24 AM Arko Roy via slurm-users < slurm-users@lists.schedmd.com> wrote: > Dear Loris, > > I just checked removing the & > it didn't work. > > On Mon, Aug 19, 2024 at 1:43 PM Loris Bennett > wrote: > >> Dear Arko, >> >> Arko Roy writes: >> >> > Thanks Loris and Gareth. here is the job submission script. if you find >> any errors please let me know. >> > since i am not the admin but just an user, i think i dont have access >> to the prolog and epilogue files. >> > >> > If the jobs are independent, why do you want to run them all on the same >> > node? >> > I am running sequential codes. Essentially 50 copies of the same node >> with a variation in parameter. >> > Since I am using the Slurm scheduler, the nodes and cores are allocated >> depending upon the >> > available resources. So there are instances, when 20 of them goes to 20 >> free cores located on a particular >> > node and the rest 30 goes to the free 30 cores on another node. It >> turns out that only 1 job out of 20 and 1 job >> > out of 30 are completed succesfully with exitcode 0 and the rest gets >> terminated with exitcode 9. >> > for information, i run sjobexitmod -l jobid to check the exitcodes. >> > >> > -- >> > the submission script is as follows: >> > >> > #!/bin/bash >> > >> > # Setting slurm options >> > >> > >> > # lines starting with "#SBATCH" define your jobs parameters >> > # requesting the type of node on which to run job >> > ##SBATCH --partition >> > #SBATCH --partition=standard >> > >> > # telling slurm how many instances of this job to spawn (typically 1) >> > >> > ##SBATCH --ntasks >> > ##SBATCH --ntasks=1 >> > #SBATCH --nodes=1 >> > ##SBATCH -N 1 >> > ##SBATCH --ntasks-per-node=1 >> > >> > # setting number of CPUs per task (1 for serial jobs) >> > >> > ##SBATCH --cpus-per-task >> > >> > ##SBATCH --cpus-per-task=1 >> > >> > # setting memory requirements >> > >> > ##SBATCH --mem-per-cpu >> > #SBATCH --mem-per-cpu=1G >> > >> > # propagating max time for job to run >> > >> > ##SBATCH --time >> > ##SBATCH --time >> > ##SBATCH --time >> > #SBATCH --time 10:0:0 >> > #SBATCH --job-name gstate >> > >> > #module load compiler/intel/2018_4 >> > module load fftw-3.3.10-intel-2021.6.0-ppbepka >> > echo "Running on $(hostname)" >> > echo "We are in $(pwd)" >> > >> > >> > # run the program >> > >> > /home/arkoroy.sps.iitmandi/ferro-detun/input1/a_1.out & >> >> You should not write >> >> & >> >> at the end of the above command. This will run your program in the >> background, which will cause the submit script to terminate, which in >> turn will terminate your job. >> >> Regards >> >> Loris >> >> -- >> Dr. Loris Bennett (Herr/Mr) >> FUB-IT, Freie Universität Berlin >> > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Print Slurm Stats on Login
This is wonderful, thanks Josef and Ole! I will need to familiarize myself with it, but on a cursory glance it looks almost exactly what I was looking for! On Wed, Aug 14, 2024 at 1:44 AM Josef Dvořáček via slurm-users < slurm-users@lists.schedmd.com> wrote: > > I too would be interested in some lightweight scripts > > For lightweight stats I tend to use this excellent script: slurmacct. > Author is member of this mailinglist too. (hi): > > > https://github.com/OleHolmNielsen/Slurm_tools/blob/master/slurmacct/slurmacct > > Currently I am in process of writing prometheus exporter as the one I've > used for years (https://github.com/vpenso/prometheus-slurm-exporter) > provides suboptimal results with Slurm 24.04+. > (we use looong job arrays at our system breaking somehow the exporter, > which is parsing text output of squeue command) > > cheers > > josef > > -- > *From:* Davide DelVento via slurm-users > *Sent:* Wednesday, 14 August 2024 01:52 > *To:* Paul Edmon > *Cc:* Reid, Andrew C.E. (Fed) ; Jeffrey T Frey < > f...@udel.edu>; slurm-users@lists.schedmd.com < > slurm-users@lists.schedmd.com> > *Subject:* [slurm-users] Re: Print Slurm Stats on Login > > I too would be interested in some lightweight scripts. XDMOD in my > experience has been very intense in workload to install, maintain and > learn. It's great if one needs that level of interactivity, granularity and > detail, but for some "quick and dirty" summary in a small dept it's not > only overkill, it's also impossible given the available staffing. > ... > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Print Slurm Stats on Login
I too would be interested in some lightweight scripts. XDMOD in my
experience has been very intense in workload to install, maintain and
learn. It's great if one needs that level of interactivity, granularity and
detail, but for some "quick and dirty" summary in a small dept it's not
only overkill, it's also impossible given the available staffing.
On Fri, Aug 9, 2024 at 10:31 AM Paul Edmon via slurm-users <
slurm-users@lists.schedmd.com> wrote:
> Yup, we have that installed already. It's been very beneficial for over
> all monitoring.
>
> -Paul Edmon-
>
> On 8/9/2024 12:27 PM, Reid, Andrew C.E. (Fed) wrote:
> >Maybe a heavier lift than you had in mind, but check
> > out xdmod, open.xdmod.org.
> >
> >It was developed by the NSF as part of the now-shuttered
> > XSEDE program, and is useful for both system and user monitoring.
> >
> >-- A.
> >
> > On Fri, Aug 09, 2024 at 12:12:08PM -0400, Paul Edmon via slurm-users
> wrote:
> >> Yeah, I was contemplating doing that so I didn't have a dependency on
> the
> >> scheduler being up or down or busy.
> >>
> >> What I was more curious about is if any one had an prebaked scripts for
> >> that.
> >>
> >> -Paul Edmon-
> >>
> >> On 8/9/2024 12:04 PM, Jeffrey T Frey wrote:
> >>> You'd have to do this within e.g. the system's bashrc infrastructure.
> The simplest idea would be to add to e.g. /etc/profile.d/zzz-slurmstats.sh
> and have some canned commands/scripts running. That does introduce load to
> the system and Slurm on every login, though, and slows the startup of login
> shells based on how responsive slurmctld/slurmdbd are at that moment.
> >>>
> >>> Another option would be to run the commands/scripts for all users on
> some timed schedule — e.g. produce per-user stats every 30 minutes. So
> long as the stats are publicly-visible anyway, put those summaries in a
> shared file system with open read access. Name the files by uid number.
> Now your /etc/profile.d script just cat's ${STATS_DIR}/$(id -u).
> >>>
> >>>
> >>>
> >>>
> On Aug 9, 2024, at 11:11, Paul Edmon via slurm-users <
> slurm-users@lists.schedmd.com> wrote:
>
> We are working to make our users more aware of their usage. One of
> the ideas we came up with was to having some basic usage stats printed at
> login (usage over past day, fairshare, job efficiency, etc). Does anyone
> have any scripts or methods that they use to do this? Before baking my own
> I was curious what other sites do and if they would be willing to share
> their scripts and methodology.
>
> -Paul Edmon-
>
>
> --
> slurm-users mailing list -- slurm-users@lists.schedmd.com
> To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
> >> --
> >> slurm-users mailing list -- slurm-users@lists.schedmd.com
> >> To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
>
> --
> slurm-users mailing list -- slurm-users@lists.schedmd.com
> To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
>
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Seeking Commercial SLURM Subscription Provider
How about SchedMD itself? They are the ones doing most (if not all) of the development, and they are great. In my experience, the best options are either SchedMD or the vendor of your hardware. On Mon, Aug 12, 2024 at 11:17 PM John Joseph via slurm-users < slurm-users@lists.schedmd.com> wrote: > Dear All, > > Good morning. > > We successfully implemented a 4-node SLURM cluster with shared storage > using GlusterFS and were able to run COMSOL programs on it. After this > learning experience, we've determined that it would be beneficial to switch > to a commercial SLURM subscription for better support. > > We are currently seeking a solution provider who can offer support based > on a commercial subscription. I would like to reach out to the group for > recommendations or advice on how we can avail these services commercially. > Thank you. > Joseph John > > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: With slurm, how to allocate a whole node for a single multi-threaded process?
I am pretty sure with vanilla slurm is impossible. What it might be possible (maybe) is submitting 5 core jobs and using some pre-post scripts which immediately before the job start change the requested number of cores to "however are currently available on the node where it is scheduled to run". That feels like a nightmare script to write, prone to race conditions (e.g. what is slurm has scheduled another job on the same node to start almost at the same time?). It also may be impractical (the modified job will probably need to be rescheduled, possibly landing on another node with a different number of idle cores) or impossible (maybe slurm does not offer the possibility of changing the requested nodes after the job has been assigned a node, only at other times, such as submission time). What is theoretically possible would be to use slurm only as a "dummy bean counter": submit the job as a 5 core job and let it land and start on a node. The job itself does nothing other than counting the number of idle nodes on that core and submitting *another* slurm job of the highest priority targeting that specific node (option -w) and that number of cores. If the second job starts, then by some other mechanism, probably external to slurm, the actual computational job will start on the appropriate nodes. If that happens outside of slurm, it would be very hard to get right (with the appropriate cgroup for example). If that happens inside of slurm, it needs some functionality which I am not aware exists, but it sounds more likely than "changing the number of cores at the moment the job start". For example the two jobs could merge into one. Or the two jobs could stay separate, but share some MPI communicator or thread space (but again have troubles with the separate cgroups they live in). So in conclusion if this is just a few jobs where you are trying to be more efficient, I think it's better to give up. If this is something of really large scale and important, then my recommendation would be to purchase official Slurm support and get assistance from them On Fri, Aug 2, 2024 at 8:37 AM Laura Hild via slurm-users < slurm-users@lists.schedmd.com> wrote: > My read is that Henrique wants to specify a job to require a variable > number of CPUs on one node, so that when the job is at the front of the > queue, it will run opportunistically on however many happen to be available > on a single node as long as there are at least five. > > I don't personally know of a way to specify such a job, and wouldn't be > surprised if there isn't one, since as other posters have suggested, > usually there's a core-count sweet spot that should be used, achieving a > performance goal while making efficient use of resources. A cluster > administrator may in fact not want you using extra cores, even if there's a > bit more speed-up to be had, when those cores could be used more > efficiently by another job. I'm also not sure how one would set a > judicious TimeLimit on a job that would have such a variable wall-time. > > So there is the question of whether it is possible, and whether it is > advisable. > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: With slurm, how to allocate a whole node for a single multi-threaded process?
In part, it depends on how it's been configured, but have you tried --exclusive? On Thu, Aug 1, 2024 at 7:39 AM Henrique Almeida via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hello, everyone, with slurm, how to allocate a whole node for a > single multi-threaded process? > > > https://stackoverflow.com/questions/78818547/with-slurm-how-to-allocate-a-whole-node-for-a-single-multi-threaded-process > > > -- > Henrique Dante de Almeida > hda...@gmail.com > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Can SLURM queue different jobs to start concurrently?
I think the best way to do it would be to schedule the 10 things to be a single slurm job and then use some of the various MPMD ways (the nitty gritty details depend if each executable is serial, OpenMP, MPI or hybrid). On Mon, Jul 8, 2024 at 2:20 PM Dan Healy via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hi there, > > I've received a question from an end user, which I presume the answer is > "No", but would like to ask the community first. > > Scenario: The user wants to create a series of jobs that all need to start > at the same time. Example: there are 10 different executable applications > which have varying CPU and RAM constraints, all of which need to > communicate via TCP/IP. Of course the user could design some type of > idle/statusing mechanism to wait until all jobs are *randomly *started, > then begin execution, but this feels like a waste of resources. The > complete execution of these 10 applications would be considered a single > simulation. The goal would be to distribute these 10 applications across > the cluster and not necessarily require them all to execute on a single > node. > > Is there a good architecture for this using SLURM? If so, please kindly > point me in the right direction. > > -- > Thanks, > > Daniel Healy > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Best practice for jobs resuming from suspended state
I don't really have an answer for you, just responding to make your message pop out in the "flood" of other topics we've got since you posted. On our cluster we configure cancelling our jobs because it makes more sense for our situation, so I have no experience with that resume from being suspended. I can think of two possible reasons for this: - one is memory (have you checked your memory logs and see if there is a correlation between node memory occupation and jobs not resuming correctly) - the second one is some resources disappearing (temp files? maybe in some circumstances slurm totally wipes out /tmp the second job -- if so, that would be a slurm bug, obviously) Assuming that you're stuck without finding a root cause which you can address, I guess it depends on what "doesn't recover" means. It's one thing if it crashes immediately. It's another if it just stalls without even starting but slurm still thinks it's running and the users are charged their allocation -- even worse if your cluster does not enforce a wallclock limit (or has a very long one). Depending on frequency of the issue, size of your cluster and other conditions, you may want to consider writing a watchdog script which would search for these jobs and cancel them? As I said, not really an answer, just my $0.02 cents (or even less) On Wed, May 15, 2024 at 1:54 AM Paul Jones via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hi, > > We use PreemptMode and PriorityTier within Slurm to suspend low priority > jobs when more urgent work needs to be done. This generally works well, but > on occasion resumed jobs fail to restart - which is to say Slurm sets the > job status to running but the actual code doesn't recover from being > suspended. > > Technically everything is working as expected, but I wondered if there was > any best practice to pass onto users about how to cope with this state? > Obviously not a direct Slurm question, but wondered if others had > experience with this and any advice on how best to limit the impact? > > Thanks, > Paul > > -- > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: memory high water mark reporting
Not exactly the answer to your question (which I don't know) but if you can get to prefix whatever is executed with this https://github.com/NCAR/peak_memusage (which also uses getrusage) or a variant you will be able to do that. On Thu, May 16, 2024 at 4:10 PM Emyr James via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hi, > > We are trying out slurm having been running grid engine for a long while. > In grid engine, the cgroups peak memory and max_rss are generated at the > end of a job and recorded. It logs the information from the cgroup > hierarchy as well as doing a getrusage call right at the end on the parent > pid of the whole job "container" before cleaning up. > With slurm it seems that the only way memory is recorded is by the acct > gather polling. I am trying to add something in an epilog script to get the > memory.peak but It looks like the cgroup hierarchy has been destroyed by > the time the epilog is run. > Where in the code is the cgroup hierarchy cleared up ? Is there no way to > add something in so that the accounting is updated during the job cleanup > process so that peak memory usage can be accurately logged ? > > I can reduce the polling interval from 30s to 5s but don't know if this > causes a lot of overhead and in any case this seems to not be a sensible > way to get values that should just be determined right at the end by an > event rather than using polling. > > Many thanks, > > Emyr > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Partition Preemption Configuration Question
{
"emoji": "👍",
"version": 1
}
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: StateSaveLocation and Slurm HA
Are you seeking something simple rather than sophisticated? If so, you can use the controller local disk for StateSaveLocation and place a cron job (on the same node or somewhere else) to take that data out via e.g. rsync and put it where you need it (NFS?) for the backup control node to use if/when needed. That obviously introduces a time delay which might or might not be problematic depending on what kind of failures you are trying to protect from and with what level of guarantee you wish the HA would have: you will not be protected in every possible scenario. On the other hand, given the size of the cluster that might be adequate and it's basically zero effort, so it might be "good enough" for you. On Tue, May 7, 2024 at 4:44 AM Pierre Abele via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hi all, > > I am looking for a clean way to set up Slurms native high availability > feature. I am managing a Slurm cluster with one control node (hosting > both slurmctld and slurmdbd), one login node and a few dozen compute > nodes. I have a virtual machine that I want to set up as a backup > control node. > > The Slurm documentation says the following about the StateSaveLocation > directory: > > > The directory used should be on a low-latency local disk to prevent file > system delays from affecting Slurm performance. If using a backup host, the > StateSaveLocation should reside on a file system shared by the two hosts. > We do not recommend using NFS to make the directory accessible to both > hosts, but do recommend a shared mount that is accessible to the two > controllers and allows low-latency reads and writes to the disk. If a > controller comes up without access to the state information, queued and > running jobs will be cancelled. [1] > > My question: How do I implement the shared file system for the > StateSaveLocation? > > I do not want to introduce a single point of failure by having a single > node that hosts the StateSaveLocation, neither do I want to put that > directory on the clusters NFS storage since outages/downtime of the > storage system will happen at some point and I do not want that to cause > an outage of the Slurm controller. > > Any help or ideas would be appreciated. > > Best, > Pierre > > > [1] https://slurm.schedmd.com/quickstart_admin.html#Config > > -- > Pierre Abele, M.Sc. > > HPC Administrator > Max-Planck-Institute for Evolutionary Anthropology > Department of Primate Behavior and Evolution > > Deutscher Platz 6 > 04103 Leipzig > > Room: U2.80 > E-Mail: pierre_ab...@eva.mpg.de > Phone: +49 (0) 341 3550 245 > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Partition Preemption Configuration Question
Hi Jason, I wanted exactly the same and was confused exactly like you. For a while it did not work, regardless of what I tried, but eventually (with some help) I figured it out. What I set up and it is working fine is this globally PreemptType = preempt/partition_prio PreemptMode=REQUEUE and then individually each partition definition has either PreemptMode=off or PreemptMode=cancel It took me a while to make it work, and the problem in my case was that I did not include the requeue line because (as I am describing) I did not want requeue, but without that line slurm preemption simply would not work. Since it's overridden in each partition, then it works as if it's not there, but it must be there. Very simple once you know it. Hope this helps On Thu, May 2, 2024 at 9:16 AM Jason Simms via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hello all, > > The Slurm docs have me a bit confused... I'm wanting to enable job > preemption on certain partitions but not others. I *presume* I would > set PreemptType=preempt/partition_prio globally, but then on the partitions > where I don't want jobs to be able to be preempted, I would set > PreemptMode=off within the configuration for that specific partition. > > The documentation, however, says that setting PreemptMode=off at a > partition level "is only compatible with PreemptType=preempt/none at a > global level" yet then immediately says that doing so is a "common use case > for this parameter is to set it on a partition to disable preemption for > that partition," which indicates preemption would still be allowable for > other partitions. > > If PreemptType is set to preempt/none globally, and I *cannot* set that as > an option for a given partition (at least, the documentation doesn't > indicate that is a valid parameter for a partition), wouldn't preemption be > disabled globally anyway? The wording seems odd to me and almost > contradictory. > > Is it possible to have PreemptType=preempt/partition_prio set globally, > yet also disable it on specific partitions with PreemptMode=off? Is > PreemptType actually a valid configuration option for specific partitions? > > Thanks for any guidance. > > Warmest regards, > Jason > > -- > *Jason L. Simms, Ph.D., M.P.H.* > Manager of Research Computing > Swarthmore College > Information Technology Services > (610) 328-8102 > Schedule a meeting: https://calendly.com/jlsimms > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Recover Batch Script Error
Yes, that is what we are also doing and it works well. Note that requesting a batch script for another user, one sees nothing (rather than an error message saying that one does not have permissions) On Fri, Feb 16, 2024 at 12:48 PM Paul Edmon via slurm-users < slurm-users@lists.schedmd.com> wrote: > Are you using the job_script storage option? If so then you should be able > to get at it by doing: > > sacct -B j JOBID > > https://slurm.schedmd.com/sacct.html#OPT_batch-script > > -Paul Edmon- > On 2/16/2024 2:41 PM, Jason Simms via slurm-users wrote: > > Hello all, > > I've used the "scontrol write batch_script" command to output the job > submission script from completed jobs in the past, but for some reason, no > matter which job I specify, it tells me it is invalid. Any way to > troubleshoot this? Alternatively, is there another way - even if a manual > database query - to recover the job script, assuming it exists in the > database? > > sacct --jobs=38960 > JobID JobName PartitionAccount AllocCPUS State > ExitCode > -- -- -- -- -- > > 38960amr_run_v+ tsmith2lab tsmith2lab 72 COMPLETED > 0:0 > 38960.batch batchtsmith2lab 40 COMPLETED > 0:0 > 38960.extern externtsmith2lab 72 COMPLETED > 0:0 > 38960.0 hydra_pmi+tsmith2lab 72 COMPLETED > 0:0 > > scontrol write batch_script 38960 > job script retrieval failed: Invalid job id specified > > Warmest regards, > Jason > > -- > *Jason L. Simms, Ph.D., M.P.H.* > Manager of Research Computing > Swarthmore College > Information Technology Services > (610) 328-8102 > Schedule a meeting: https://calendly.com/jlsimms > > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Need help managing licence
The simple answer is to just add a line such as Licenses=whatever:20 and then request your users to use the -L option as described at https://slurm.schedmd.com/licenses.html This works very well, however it does not do enforcement like Slurm does with other resources. You will find posts in this list from me trying to achieve such enforcement with prolog, but I ended up banging too much my head on the keyboard and so I eventually gave up. User education was easier for me. Depending on your user community, banging your head on the keyboard might be easier than educating your users -- if so please share how you solve the issue On Fri, Feb 16, 2024 at 7:48 AM Sylvain MARET via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hello everyone ! > > Recently our users bought a cplex dynamic license and want to use it on > our slurm cluster. > I've installed the paid version of cplex within modules so authorized > user can load it with a simple module load cplex/2111 command but I > don't know how to manage and ensure slurm doesn't launch a job if 20 > people are already running code with this license. > > How do you guys manage paid licenses on your cluster ? Any advice would > be appreciated ! > > Regards, > Sylvain Maret > > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
[slurm-users] Re: Compilation question
Hi Sylvain, For the series better late than never, is this still a problem? If so, is this a new install or an update? Whan environment/compiler are you using? The error undefined reference to `__nv_init_env' seems to indicate that you are doing something cuda-related which I think you should not be doing? In any case, most people run on a RHEL (or compatible) distro and use rpmbuild rather than straight configure/make, e.g. a variant of what is described at https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/ Hope this helps, On Wed, Jan 17, 2024 at 8:36 AM Sylvain MARET wrote: > Hello everyone ! > > I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi > support > > I've seen the documentation so I've done the configure step : > > ./configure --with-pmix=$PMIXHOME --with-ucx=$UCXHOME > --with-nvml=$NVMLHOME --prefix=$SLURMHOME --with-freeipmi=/usr > > but when I run make I end up with the following error : > > /bin/sh ../../../../../libtool --tag=CC --mode=link gcc > -DNUMA_VERSION1_COMPATIBILITY -g -O2 -fno-omit-frame-pointer -pthread > -ggdb3 -Wall -g -O1 -fno-strict-aliasing -export-dynamic -L/usr/lib64 > -lhdf5_hl -lhdf5 -lsz -lz -ldl -lm -o sh5util sh5util.o > -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm > -L../../../../../src/api/.libs -lslurmfull -ldl ../libhdf5_api.la > -lpthread -lm -lresolv > libtool: link: gcc -DNUMA_VERSION1_COMPATIBILITY -g -O2 > -fno-omit-frame-pointer -pthread -ggdb3 -Wall -g -O1 > -fno-strict-aliasing -o .libs/sh5util sh5util.o > -Wl,-rpath=/softs/batch/slurm/22.05.11/lib/slurm -Wl,--export-dynamic > -L/usr/lib64 -L../../../../../src/api/.libs > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so > ../.libs/libhdf5_api.a -lhdf5_hl -lhdf5 -lsz -lz -ldl -lpthread -lm > -lresolv -pthread -Wl,-rpath -Wl,/softs/batch/slurm/22.05.11/lib/slurm > sh5util.o:(.init_array+0x0): undefined reference to `__nv_init_env' > sh5util.o:(.init_array+0x8): undefined reference to `__flushz' > sh5util.o:(.init_array+0x10): undefined reference to `__daz' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_list_transfer_unique' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_sort_key_pairs' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_xstrchr' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_unsetenvp' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_list_sort' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_list_for_each' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `__builtin__pgi_isnanld' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_get_extra_conf_path' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `__blt_pgi_ctzll' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_running_in_slurmctld' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `__c_mcopy1' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `__blt_pgi_clzll' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_list_create' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_list_count' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `__builtin_va_gparg1' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_destroy_config_key_pair' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_xfree_ptr' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_getenvp' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_free_buf' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_get_log_level' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `__c_mset8' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_xstrdup_printf' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_list_delete_first' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_list_append' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_error' > /softs/batch/slurm/slurm-22.05.11/src/api/.libs/libslurmfull.so: > undefined reference to `slurm_init_buf' > /softs/batch/slu
[slurm-users] Re: Memory used per node
If you would like the high watermark memory utilization after the job completes, https://github.com/NCAR/peak_memusage is a great tool. Of course it has the limitation that you need to know that you want that information *before* starting the job, which might or might not a problem for your use case On Fri, Feb 9, 2024 at 10:07 AM Gerhard Strangar via slurm-users < slurm-users@lists.schedmd.com> wrote: > Hello, > > I'm wondering if there's a way to tell how much memory my job is using > per node. I'm doing > > #SBATCH -n 256 > srun solver inputfile > > When I run sacct -o maxvmsize, the result apparently is the maxmimum VSZ > of the largest solver process, not the maximum of the sum of them all > (unlike when calling mpirun instead). When I sstat -o TresUsageInMax, I > get the memory summed up over all nodes being used. Can I get the > maximum VSZ per node? > > > Gerhard > > -- > slurm-users mailing list -- slurm-users@lists.schedmd.com > To unsubscribe send an email to slurm-users-le...@lists.schedmd.com > -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-le...@lists.schedmd.com
