OK figured this one out - the cephx keys are missing a permission which allows them to see blacklisted clients - as a result they can't deal with a hard crash:
mon 'allow command "osd blacklist"' This is a charm issue after all. As a workaround you can manually update the existing client keys for nova-compute using: sudo ceph auth caps client.nova-compute mon 'allow r, allow command "osd blacklist"' osd 'allow rwx' from any mon unit. ** Changed in: nova Status: New => Invalid ** Changed in: ceph (Ubuntu) Status: New => Invalid ** Changed in: nova (Ubuntu) Status: New => Invalid ** Changed in: cloud-archive Status: New => Invalid ** Changed in: qemu (Ubuntu) Status: New => Invalid ** Also affects: charm-ceph-mon Importance: Undecided Status: New ** Also affects: charms.ceph Importance: Undecided Status: New ** Changed in: charms.ceph Status: New => Triaged ** Changed in: charm-ceph-mon Status: New => Triaged ** Changed in: charms.ceph Importance: Undecided => High ** Changed in: charm-ceph-mon Importance: Undecided => High ** Changed in: charm-ceph-mon Milestone: None => 18.08 ** Changed in: cloud-archive Assignee: Sean Feole (sfeole) => (unassigned) ** Changed in: nova (Ubuntu) Assignee: Sean Feole (sfeole) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs