You can push a new conf file and issue an "scontrol reconfigure" on
the fly as needed... I do it on our cluster as needed, do the nodes
first then login nodes then the slurm controller... you are making a
huge issue of a very basic task...
Sid
On Tue, 4 May 2021, 22:28 Tina Friedrich, <tina.friedr...@it.ox.ac.uk
<mailto:tina.friedr...@it.ox.ac.uk>> wrote:
Hello,
a lot of people already gave very good answer to how to tackle this.
Still, I thought it worth pointing this out - you said 'you need to
basically shut down slurm, update the slurm.conf file, then restart'.
That makes it sound like a major operation with lots of prep
required.
It's not like that at all. Updating slurm.conf is not a major
operation.
There's absolutely no reason to shut things down first & then change
the
file. You can edit the file / ship out a new version (however you
like)
and then restart the daemons.
The daemons do not have to all be restarted simultaneously. It is
of no
consequence if they're running with out-of-sync config files for a
bit,
really. (There's a flag you can set if you want to suppress the
warning
- 'NO_CONF_HASH' debug flag I think).
Restarting the dameons (slurmctld, slurmd, ...) is safe. It does not
require cluster downtime or anything.
I control slurm.conf using configuration management; the config
management process restarts the appropriate daemon (slurmctld,
slurmd,
slurmdbd) if the file changed. This certainly never happens at the
same
time; there's splay in that. It doesn't even necessarily happen on
the
controller first, or anything like that.
What I'm trying to get across - I have a feeling this 'updating the
cluster wide config file' and 'file must be the same on all nodes'
is a
lot less of a procedure (and a lot less strict) than you currently
imagine it to be :)
Tina
On 27/04/2021 19:35, David Henkemeyer wrote:
> Hello,
>
> I'm new to Slurm (coming from PBS), and so I will likely have a
few
> questions over the next several weeks, as I work to transition my
> infrastructure from PBS to Slurm.
>
> My first question has to do with *_adding nodes to Slurm_*.
According
> to the FAQ (and other articles I've read), you need to basically
shut
> down slurm, update the slurm.conf file /*on all nodes in the
cluster*/,
> then restart slurm.
>
> - Why do all nodes need to know about all other nodes? From what
I have
> read, its Slurm does a checksum comparison of the slurm.conf file
across
> all nodes. Is this the only reason all nodes need to know
about all
> other nodes?
> - Can I create a symlink that points <sysconfdir>/slurm.conf to a
> slurm.conf file on an NFS mount point, which is mounted on all the
> nodes? This way, I would only need to update a single file, then
> restart Slurm across the entire cluster.
> - Any additional help/resources for adding/removing nodes to
Slurm would
> be much appreciated. Perhaps there is a "toolkit" out there to
automate
> some of these operations (which is what I already have for PBS,
and will
> create for Slurm, if something doesn't already exist).
>
> Thank you all,
>
> David
-- Tina Friedrich, Advanced Research Computing Snr HPC Systems
Administrator
