Turns out that when we started the 3 OSDs it did “out” the rest on the same
host, so their reweight was 0.
Thus when I started the singular OSD on that host, it tried to put all the PGs
on the other OSDs onto this one (which failed for lack of disk space) and
because of that it also consumed
“Friday fun”… not!
We set mon_osd_down_out_subtree_limit=host some time ago. Now we needed to take
down all OSDs on one host and as expected nothing happened (noout was _not_
set). All the PGs showed as stuck degraded.
Then we took 3 OSDs on the host up and then down again because of slow
