Re: [slurm-users] Correct way to do logrotation
Schedmd has docs about how to do this at:
https://slurm.schedmd.com/slurm.conf.html#SECTION_LOGGING
Our config at HMS looks like this:
/var/log/slurm/slurmctld.log {
create 0640 slurm root
daily
dateext
nocompress
notifempty
rotate 10
sharedscripts
postrotate
/bin/pkill -x --signal SIGUSR2 slurmctld >/dev/null 2>&1
endscript
prerotate
/bin/sh /usr/local/bin/pre_rotate.sh >/dev/null 2>&1
endscript
}
Our pre_rotate scripts rsync's the logs to an NFS mount.
Cheers
--Mick
From: slurm-users on behalf of Davide
DelVento
Sent: Tuesday, October 17, 2023 10:26 AM
To: Slurm User Community List
Subject: Re: [slurm-users] Correct way to do logrotation
I'd be interested in this too, and I'm reposting only because the message was
flagged as both "dangerous email" and "spam", so people may not have seen it
(hopefully my reply will not suffer the same downfall...)
On Mon, Oct 16, 2023 at 3:26 AM Taras Shapovalov
mailto:tshapova...@nvidia.com>> wrote:
Hello,
In the past it was recommended to reconfigure slurm daemons in logrotate
script, sending a signal I believe was also the way to go. But recently I
retested manual logrotation and I see that a removal of log file (for
slurmctld, slurmdbd or slurmd) does not affect the logging of the daemons. The
dameons just recreate the log files and continue to write logs there. What is
the right way to go in case of the modern Slurm versions?
Best regards,
Taras
Re: [slurm-users] Correct way to do logrotation
Thanks for doing that, as I did not see this original message, and I also am having to look at configuring our log for rotation. We once accidentally turned on debug5 and didn't notice until other things started failing because the drive was full...from that ONE file. I did find this conversation about it, but haven't had a chance to try it. https://groups.google.com/g/slurm-users/c/WZfyVh2CXK8?pli=1 Rob From: slurm-users on behalf of Davide DelVento Sent: Tuesday, October 17, 2023 10:26 AM To: Slurm User Community List Subject: Re: [slurm-users] Correct way to do logrotation I'd be interested in this too, and I'm reposting only because the message was flagged as both "dangerous email" and "spam", so people may not have seen it (hopefully my reply will not suffer the same downfall...) On Mon, Oct 16, 2023 at 3:26 AM Taras Shapovalov mailto:tshapova...@nvidia.com>> wrote: Hello, In the past it was recommended to reconfigure slurm daemons in logrotate script, sending a signal I believe was also the way to go. But recently I retested manual logrotation and I see that a removal of log file (for slurmctld, slurmdbd or slurmd) does not affect the logging of the daemons. The dameons just recreate the log files and continue to write logs there. What is the right way to go in case of the modern Slurm versions? Best regards, Taras
Re: [slurm-users] Correct way to do logrotation
I'd be interested in this too, and I'm reposting only because the message was flagged as both "dangerous email" and "spam", so people may not have seen it (hopefully my reply will not suffer the same downfall...) On Mon, Oct 16, 2023 at 3:26 AM Taras Shapovalov wrote: > Hello, > > In the past it was recommended to reconfigure slurm daemons in logrotate > script, sending a signal I believe was also the way to go. But recently I > retested manual logrotation and I see that a removal of log file (for > slurmctld, slurmdbd or slurmd) does not affect the logging of the daemons. > The dameons just recreate the log files and continue to write logs there. > What is the right way to go in case of the modern Slurm versions? > > Best regards, > > Taras >
[slurm-users] Correct way to do logrotation
Hello, In the past it was recommended to reconfigure slurm daemons in logrotate script, sending a signal I believe was also the way to go. But recently I retested manual logrotation and I see that a removal of log file (for slurmctld, slurmdbd or slurmd) does not affect the logging of the daemons. The dameons just recreate the log files and continue to write logs there. What is the right way to go in case of the modern Slurm versions? Best regards, Taras
