On 08/01/2017 04:11 PM, Golpayegani, Navid (GSFC-6190) wrote:
Hi all,
Is there a way to submit submit maintenance jobs in a rolling fashion? What
I’m thinking is the ability to run a job on every node in a slurm cluster/queue
in exclusive mode but X at a time.
For example, say I want to reformat the local scratch space on every slurm
node. I’d want to submit a job to reformat the local scratch space. This job
would run exclusively on X nodes at a time. As it finishes on one node another
job is scheduled for the next node until all nodes in the cluster/queue have
run the job.
I’ve been looking at slurm reservations but if I understand them correctly I
would be reserving the whole cluster to run the job. This would block the whole
cluster until maintenance is done. Alternatively, I would have to manually
create reservations of a subset of the cluster run the job on those and then
reserve the next part of the cluster.
Navid
Hi Navid,
This can be done in several ways. I will shortly present what
we currently are doing at UPPMAX.
We have added a few lines to the file /etc/rc.local, that says:
if [ -x /etc/maintenance_script ] ; then
/bin/logger -s -p local1.info -- "/etc/rc.local: Finds file
/etc/maintenance_script, runs and removes it now."
/bin/bash -c /etc/maintenance_script
/bin/rm -f /etc/maintenance_script
fi
So, we copy a maintenance script to the compute node, and
drains it with reason=please_reboot .
A second local system, named resume_drained_nodes, reboots
nodes with this reason when they are fully drained (implicitly
meaning that the maintenance script is run at boot time), and
puts the nodes back in production when they have successfully
run a set of tests. (If they do not pass the tests, the script
resume_drained_nodes will get back to them regularly until
they are back in production.)
Cheers,
-- Lennart Karlsson, UPPMAX, Uppsala University, Sweden
http://uppmax.uu.se
