[slurm-users] Re: Slurmd enabled crash with CgroupV2

[Josef Dvoracek via slurm-users]

thanks for hint.

so you end with two "slurmstepd infinity" processes like me when I tried 
this workaround?


[root@node ~]# ps aux | grep slurm
root    1833  0.0  0.0  33716  2188 ?    Ss   21:02   0:00 
/usr/sbin/slurmstepd infinity
root    2259  0.0  0.0 236796 12108 ?    Ss   21:02   0:00 
/usr/sbin/slurmd --systemd
root    2331  0.0  0.0  33716  1124 ?    S    21:02   0:00 
/usr/sbin/slurmstepd infinity
root    2953  0.0  0.0 221944  1092 pts/0    S+   21:12   0:00 grep 
--color=auto slurm

[root@node ~]#

BTW, I found mention of change in slurm cgroupsv2 code in changelog of 
slurm for next release,


https://github.com/SchedMD/slurm/blob/master/NEWS

one can see here the commit

https://github.com/SchedMD/slurm/commit/c21b48e724ec6f36d82c8efb1b81b6025ede240d

referring to bug

https://bugs.schedmd.com/show_bug.cgi?id=19157

but as the bug is private, I can not see the bug description.

So perhaps with Slurm 24.xx release we'll see something new.

cheers

josef








On 11. 04. 24 19:53, Williams, Jenny Avis wrote:


There needs to be a slurmstepd infinity process running before slurmd 
starts.


This doc goes into it:
https://slurm.schedmd.com/cgroup_v2.html

Probably a better way to do this, but this is what we do to deal with 
that:


::

files/slurm-cgrepair.service

::

[Unit]

Before=slurmd.service slurmctld.service

After=nas-longleaf.mount remote-fs.target system.slice

[Service]

Type=oneshot

ExecStart=/callback/slurm-cgrepair.sh

[Install]

WantedBy=default.target

::

files/slurm-cgrepair.sh

::

#!/bin/bash

/usr/bin/echo +cpu +cpuset +memory >> 
/sys/fs/cgroup/cgroup.subtree_control && \


/usr/bin/echo +cpu +cpuset +memory >> 
/sys/fs/cgroup/system.slice/cgroup.subtree_control


/usr/sbin/slurmstepd infinity &

*From:*Josef Dvoracek via slurm-users 
*Sent:* Thursday, April 11, 2024 11:14 AM
*To:* slurm-users@lists.schedmd.com
*Subject:* [slurm-users] Re: Slurmd enabled crash with CgroupV2

I observe same behavior on slurm 23.11.5 Rocky Linux8.9..

> [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control
> memory pids
> [root@compute ~]# systemctl disable slurmd
> Removed /etc/systemd/system/multi-user.target.wants/slurmd.service.
> [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control
> cpuset cpu io memory pids
> [root@compute ~]# systemctl enable slurmd
> Created symlink 
/etc/systemd/system/multi-user.target.wants/slurmd.service → 
/usr/lib/systemd/system/slurmd.service.

> [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control
> cpuset cpu io memory pids

over time (i see this thread is ~1 year old, is here better / new 
understanding of this?


cheers

josef

On 23. 05. 23 12:46, Alan Orth wrote:

I notice the exact same behavior as Tristan. My CentOS Stream 8
system is in full unified cgroupv2 mode, the slurmd.service has a
"Delegate=Yes" override added to it, and all cgroup stuff is added
to slurm.conf and cgroup.conf, yet slurmd does not start after
reboot. I don't understand what is happening, but I see the exact
same behavior regarding the cgroup subtree_control with disabling
/ re-enabling slurmd.



smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: Slurmd enabled crash with CgroupV2

[Josef Dvoracek via slurm-users]

I observe same behavior on slurm 23.11.5 Rocky Linux8.9..

> [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control
> memory pids
> [root@compute ~]# systemctl disable slurmd
> Removed /etc/systemd/system/multi-user.target.wants/slurmd.service.
> [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control
> cpuset cpu io memory pids
> [root@compute ~]# systemctl enable slurmd
> Created symlink 
/etc/systemd/system/multi-user.target.wants/slurmd.service → 
/usr/lib/systemd/system/slurmd.service.

> [root@compute ~]# cat /sys/fs/cgroup/cgroup.subtree_control
> cpuset cpu io memory pids

over time (i see this thread is ~1 year old, is here better / new 
understanding of this?


cheers

josef


On 23. 05. 23 12:46, Alan Orth wrote:
I notice the exact same behavior as Tristan. My CentOS Stream 8 system 
is in full unified cgroupv2 mode, the slurmd.service has a 
"Delegate=Yes" override added to it, and all cgroup stuff is added to 
slurm.conf and cgroup.conf, yet slurmd does not start after reboot. I 
don't understand what is happening, but I see the exact same behavior 
regarding the cgroup subtree_control with disabling / re-enabling slurmd.




smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] visualisation of JobComp and JobacctGather data with Grafana - screenshots, ideas?

[Josef Dvoracek via slurm-users]

Is here anybody having nice visualization of JobComp and JobacctGather 
data in Grafana?


I save JobComp data in Elasticsearch, JobacctGather data in influxDB, 
and thinking about how to provide meaningful insights to $users.


Things I'd like to show..: especially memory & cpu utilization, job 
result, possible malicious effects like OOMs...


Any screenshots, ideas, experience welcomed!

cheers

Josef




smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: cgroups_exporter for slurm on rhel9 (cgroups-v2)

[Josef Dvoracek via slurm-users]

I use telegraf (which supports "exporter" output format as well) to 
capture cgroupsv2 job data:


https://github.com/jose-d/telegraf-configs/tree/master/slurm-cgroupsv2

I had to rework it when changing from cgroupsv1 to cgroupsv2, as the 
format/structure of textfiles changed a bit.


cheers

josef

On 21. 03. 24 18:55, Saluja, Prabhjyot via slurm-users wrote:

...


smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: Slurm suspend preemption not working

[Josef Dvoracek via slurm-users]

I think you need set reasonable "DefMemPerCPU" - otherwise jobs will 
take all memory by default, and there is no remaining memory for the 
second job.



We calculated DefMemPerCPU in such way, that the default allocated 
memory of full node is slightly under half of total node memory. So 
there is always space for second job. And if user needs more, uses 
"--mem" option in sbatch and gets what needs.



cheers


josef



On 15. 03. 24 16:05, Verma, Nischey (HPC ENG,RAL,LSCI) via slurm-users 
wrote:

...

smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: [EXTERN] Re: sbatch and cgroup v2

[Josef Dvoracek via slurm-users]

> I'm running slurm 22.05.11 which is available with OpenHCP 3.x
> Do you think an upgrade is needed?

I feel that lot of slurm operators tend to not use 3rd party sources of 
slurm binaries, as you do not have the build environment fully in your 
hands.


But before making such a complex decision, perhaps look for build logs 
of slurm you use (somewhere in OpenHPC buildsystem?) and check if it was 
built with libraries needed to have cgroupsv2 working..


Not having cgroupsv2 dependencies during build-time is only one of all 
possible causes..


josef






smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: slurmdbd error - Symbol `slurm_conf' has different size in shared object

[Josef Dvoracek via slurm-users]

I think installing/upgrading "slurm" rpm will replace this shared lib.

Indeed, as always, test it first at not-so-critical system, use vm 
snapshots to be able to travel back in time ... as once you'll upgrade 
DB schema (if part of upgrade) you AFAIK can not go back.


josef

On 28. 02. 24 15:51, Miriam Olmi via slurm-users wrote:

I installed the new version of slurm 23.11.0-1 by rpm.
How can I fix this?


smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: slurmdbd error - Symbol `slurm_conf' has different size in shared object

[Josef Dvoracek via slurm-users]

I see this question unanswered so far.. so I'll give you my 2 cents:

Quick check reveals that mentioned symbol is in libslurmfull.so :

[root@slurmserver2 ~]# nm -gD /usr/lib64/slurm/libslurmfull.so | grep 
"slurm_conf$"

000d2c06 T free_slurm_conf
000d3345 T init_slurm_conf
0041d000 B slurm_conf
[root@slurmserver2 ~]#

can not be that this dynamic lib is still the old one?

Depending if you install slurm by rpms, manual in-place build, or 
something else, the reasons why there is old lib in place may vary..


cheers

josef


On 28. 02. 24 11:16, Miriam Olmi via slurm-users wrote:

`slurm_conf' has different size in shared object, consider re-linking

smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: sbatch and cgroup v2

[Josef Dvoracek via slurm-users]

Hi Dietmar;

I tried this on ${my cluster}, as I switched to cgroupsv2 quite recently..

I must say that on my setup it looks it works as expected, see the 
grepped stdout from your reproducer below.


I use recent slurm 23.11.4 .

Wild guess.. Has your build machine bpt and dbus devel packages installed?
(both packages are fine to be absent when doing build for cgroupsv1 - 
slurm..)


cheers

josef

[jose@koios1 test_cgroups]$ cat slurm-7177217.out | grep eli
ValueError: CPU number 7 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 4 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 5 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 11 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 9 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 10 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 14 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 8 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 12 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 6 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 13 is not eligible; choose between [0, 1, 2, 3]
ValueError: CPU number 15 is not eligible; choose between [0, 1, 2, 3]
[jose@koios1 test_cgroups]$

On 28. 02. 24 14:28, Dietmar Rieder via slurm-users wrote:
...


smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: canonical way to run longer shell/bash interactive job (instead of srun inside of screen/tmux at front-end)?

[Josef Dvoracek via slurm-users]

From unclear reason "--wrap" was not part of my /repertoire/ so far.

thanks

On 26. 02. 24 9:47, Ward Poelmans via slurm-users wrote:

sbatch --wrap 'screen -D -m'
srun --jobid  --pty screen -rd

smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] canonical way to run longer shell/bash interactive job (instead of srun inside of screen/tmux at front-end)?

[Josef Dvoracek via slurm-users]

What is the recommended way to run longer interactive job at your systems?

Our how-to includes starting screen at front-end node and running srun 
with bash/zsh inside,
but that indeed brings dependency between login node (with screen) and 
the compute node job.


On systems with multiple front-ends users need to remember the login 
node where they have their screen session..


Are you anybody using something more advanced and still understandable 
by casual user of HPC?


(I know Open On Demand, but often the use of native console has certain 
benefits. )


cheers

josef






smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: Question about IB and Ethernet networks

[Josef Dvoracek via slurm-users]

> Just looking for some feedback, please. Is this OK? Is there a better 
way?


> I’m tempted to spec all new HPCs with only a high speed (200Gbps) IB 
network,


Well you need Ethernet for OOB management (bmc/ipmi/ilo/whatever) 
anyway.. or?


cheers

josef

On 25. 02. 24 21:12, Dan Healy via slurm-users wrote:


This question is not slurm-specific, but it might develop into that.


smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: Compilation question

[Josef Dvoracek via slurm-users]

isn't your /softs.. filesystem eg. some cluster network filesystem mount?

It happened to me multiple times, that I was attempting to build some 
scientific software, and because of building on top of BeeGFS (I think 
hardlinks are not fully supported), or NFS ( caching), I was getting 
_interesting_ or suboptimal results..


so TL/DR: do the build on local NVME/ssd/hdd FS.

Or, you can fork my https://github.com/jose-d/build-slurm-rocky8 build 
"CI" using github actions, and implement freeipmi and UCX into that..


cheers

josef

On 17. 01. 24 16:33, Sylvain MARET wrote:

Hello everyone !

I'm trying to compile slurm 22.05.11 on Rocky linux 8.7 with freeipmi 
support


...


smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com

[slurm-users] Re: Why is Slurm 20 the latest RPM in RHEL 8/Fedora repo?

[Josef Dvoracek via slurm-users]

My impression is, that there are multiple challenges why it's not easy 
to create good-for-all recent slurm RPM:


- NVML dependency - different sites use different NVML lib version with 
varying update cycle
- pmi* deps - some sites (like mine) is using only one reasonable recent 
openpmix, I know other sites, using eg. Impi with different pmi 
configuration requirements

- EL release cycle vs. Slurm release schedule is (fortunately!) not aligned

BTw. there is HPC-SIG group in Rocky Linux interested in packaging slurm..
OpenHPC has also own distribution of slurm rpms, and there used to be 
some in EPEL too.


I ended writing GitHub action building the rpms for me.. and I must say 
that SchedMD maintains quite stable and predictable build procedure, so 
usually bumping version(s) is enough.


cheers

josef


On 29. 01. 24 17:00, Robert Kudyba wrote:

According to these links:
https://rpmfind.net/linux/rpm2html/search.php?query=slurm
https://src.fedoraproject.org/rpms/slurm

Why doesn't RHEL 8 get a newer version? Can someone update the repo 
maintainer Philip Kovacs  
? There was a ticket at 
https://bugzilla.redhat.com/show_bug.cgi?id=1912491 but no movement on 
RHEL 8.

smime.p7s
Description: S/MIME Cryptographic Signature

-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-le...@lists.schedmd.com