On 10-12-2023 17:29, Ryan Novosielski wrote:
This is basically always somebody filling up /tmp and /tmp residing on the same
filesystem as the actual SlurmdSpoolDirectory.
/tmp, without modifications, it’s almost certainly the wrong place for
temporary HPC files. Too large.
Agreed! That's why temporary job directories may be configured in
Slurm, see the Wiki page for a summary:
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#temporary-job-directories
/Ole
On Dec 8, 2023, at 10:02, Xaver Stiensmeier <xaverstiensme...@gmx.de> wrote:
Dear slurm-user list,
during a larger cluster run (the same I mentioned earlier 242 nodes), I
got the error "SlurmdSpoolDir full". The SlurmdSpoolDir is apparently a
directory on the workers that is used for job state information
(https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdSpoolDir). However,
I was unable to find more precise information on that dictionary. We
compute all data on another volume so SlurmdSpoolDir has roughly 38 GB
of free space where nothing is intentionally put during the run. This
error only occurred on very few nodes.
I would like to understand what Slurmd is placing in this dir that fills
up the space. Do you have any ideas? Due to the workflow used, we have a
hard time reconstructing the exact scenario that caused this error. I
guess, the "fix" is to just pick a bit larger disk, but I am unsure
whether Slurm behaves normal here.
Best regards
Xaver Stiensmeier
