Slurm versions 23.02.6 and 22.05.10 are now available to address a
number of filesystem race conditions that could let an attacker take
control of an arbitrary file, or remove entire directories' contents
(CVE-2023-41914).
SchedMD customers were informed on September 27th and provided a patch
on request; this process is documented in our security policy [1].
--------
CVE-2023-41914:
A number of race conditions have been identified within the
slurmd/slurmstepd processes that can lead to the user taking ownership
of an arbitrary file on the system. A related issue can lead to the user
overwriting an arbitrary file on the compute node (although with data
that is not directly under their control). A related issue can also lead
to the user deleting all files and sub-directories of an arbitrary
target directory on the compute node.
Thank you to François Diakhate (CEA) for reporting the original issue to
us. A number of related issues were found during an extensive audit of
Slurm's filesystem handling code in reaction to that report, and are
included here in this same disclosure.
--------
SchedMD only issues security fixes for the supported releases (currently
23.02 and 22.05). Due to the complexity of these fixes, we do not
recommend attempting to backport the fixes to older releases, and
strongly encourage sites to upgrade to fixed versions immediately.
Downloads are available at https://www.schedmd.com/downloads.php .
Release notes follow below.
- Tim
[1] https://www.schedmd.com/security.php
--
Tim Wickberg
Chief Technology Officer, SchedMD LLC
Commercial Slurm Development and Support
* Changes in Slurm 23.02.6
==========================
-- Fix CpusPerTres= not upgreadable with scontrol update
-- Fix unintentional gres removal when validating the gres job state.
-- Fix --without-hpe-slingshot configure option.
-- Fix cgroup v2 memory calculations when transparent huge pages are used.
-- Fix parsing of sgather --timeout option.
-- Fix regression from 22.05.0 that caused srun --cpu-bind "=verbose" and "=v"
options give different CPU bind masks.
-- Fix "_find_node_record: lookup failure for node" error message appearing
for all dynamic nodes during reconfigure.
-- Avoid segfault if loading serializer plugin fails.
-- slurmrestd - Correct OpenAPI format for 'GET /slurm/v0.0.39/licenses'.
-- slurmrestd - Correct OpenAPI format for 'GET /slurm/v0.0.39/job/{job_id}'.
-- slurmrestd - Change format to multiple fields in 'GET
/slurmdb/v0.0.39/assocations' and 'GET /slurmdb/v0.0.39/qos' to handle
infinite and unset states.
-- When a node fails in a job with --no-kill, preserve the extern step on the
remaining nodes to avoid breaking features that rely on the extern step
such as pam_slurm_adopt, x11, and job_container/tmpfs.
-- auth/jwt - Ignore 'x5c' field in JWKS files.
-- auth/jwt - Treat 'alg' field as optional in JWKS files.
-- Allow job_desc.selinux_context to be read from the job_submit.lua script.
-- Skip check in slurmstepd that causes a large number of errors in the munge
log: "Unauthorized credential for client UID=0 GID=0". This error will
still appear on slurmd/slurmctld/slurmdbd start up and is not a cause for
concern.
-- slurmctld - Allow startup with zero partitions.
-- Fix some mig profile names in slurm not matching nvidia mig profiles.
-- Prevent slurmscriptd processing delays from blocking other threads in
slurmctld while trying to launch {Prolog|Epilog}Slurmctld.
-- Fix sacct printing ReqMem field when memory doesn't exist in requested TRES.
-- Fix how heterogenous steps in an allocation with CR_PACK_NODE or -mpack are
created.
-- Fix slurmctld crash from race condition within job_submit_throttle plugin.
-- Fix --with-systemdsystemunitdir when requesting a default location.
-- Fix not being able to cancel an array task by the jobid (i.e. not
<jobid>_<taskid>) through scancel, job launch failure or prolog failure.
-- Fix cancelling the whole array job when the array task is the meta job and
it fails job or prolog launch and is not requeable. Cancel only the
specific task instead.
-- Fix regression in 21.08.2 where MailProg did not run for mail-type=end for
jobs with non-zero exit codes.
-- Fix incorrect setting of memory.swap.max in cgroup/v2.
-- Fix jobacctgather/cgroup collection of disk/io, gpumem, gpuutil TRES values.
-- Fix -d singleton for heterogeneous jobs.
-- Downgrade info logs about a job meeting a "maximum node limit" in the
select plugin to DebugFlags=SelectType. These info logs could spam the
slurmctld log file under certain circumstances.
-- prep/script - Fix [Srun|Task]<Prolog|Epilog> missing SLURM_JOB_NODELIST.
-- gres - Rebuild GRES core bitmap for nodes at startup. This fixes error:
"Core bitmaps size mismatch on node [HOSTNAME]", which causes jobs to enter
state "Requested node configuration is not available".
-- slurmctd - Allow startup with zero nodes.
-- Fix filesystem handling race conditions that could lead to an attacker
taking control of an arbitrary file, or removing entire directories'
contents. CVE-2023-41914.
* Changes in Slurm 22.05.10
===========================
-- Fix filesystem handling race conditions that could lead to an attacker
taking control of an arbitrary file, or removing entire directories'
contents. CVE-2023-41914.
