And, what is hopefully my final update on this:
Unfortunately I missed including a single last-minute commit in the
21.08.8 release. That missing commit fixes a communication issue between
a mix of patched and unpatched slurmd processes that could lead to nodes
being incorrectly marked as offline.
That patch was included in 20.11.9. That missing commit is included in a
new 21.08.8-2 release which is on our download page now.
If you've already starting rolling out 21.08.8 on your systems, the best
path forward it to restart all slurmd processes in the cluster immediately.
- Tim
