[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
** Changed in: tripleo Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
Fixed in TripleO via change I9639d606bd538f6776c368a4f34aa6783ab91abb ** Also affects: tripleo Importance: Undecided Status: New ** Changed in: tripleo Status: New => Fix Committed ** Changed in: tripleo Importance: Undecided => High ** Changed in: tripleo Assignee: (unassigned) => Giulio Fidente (gfidente) ** Changed in: tripleo Milestone: None => rocky-3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
I can confirm this resolved my issue - I tested several times by 'kill -9' an instance before and after the config change to client.cinder Ceph permissions. *** WARNING!!! If you make the change manually, as I did, be careful because you need to explicitly define all permissions for the client.cinder user as it overwrites all current permissions. For reference, I issued: ceph auth caps client.cinder mon 'allow r, allow command "osd blacklist"' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images' mds '' This retained all existing permissions. You may need to change your pool names and Ceph user. Thanks for the help James, very useful. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
** Changed in: charm-ceph-mon Status: Fix Committed => Fix Released ** Changed in: charm-ceph-mon Milestone: 18.08 => 18.05 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
** Changed in: charm-ceph-mon Status: Fix Committed => Fix Released ** Changed in: charm-ceph-mon Milestone: 18.08 => 18.05 -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
Reviewed: https://review.openstack.org/573664 Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=e071486acf8e90f293ce5ce8880047384527c9f0 Submitter: Zuul Branch:master commit e071486acf8e90f293ce5ce8880047384527c9f0 Author: James Page Date: Fri Jun 8 12:02:06 2018 +0100 Add 'osd blacklist' to default mon perms Ensure that the default permissions for clients include the 'osd blacklist' command; This ensures that in the event of a client crashing (due to power outage or segfault), the client and re-connect and write to any devices on reboot. This is a safe permission for all supported Ceph releases. Depends-On: I0b43dece4e1c56fb838b0147bfb75fb9906e6657 Change-Id: Ib1f1e8d7ed54528603b8b08051dafeec075a3232 Closes-Bug: 1773449 ** Changed in: charm-ceph-mon Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
Reviewed: https://review.openstack.org/573659 Committed: https://git.openstack.org/cgit/openstack/charms.ceph/commit/?id=4d8f31d0ea0f47fd71939973cba86137d5daaef4 Submitter: Zuul Branch:master commit 4d8f31d0ea0f47fd71939973cba86137d5daaef4 Author: James Page Date: Fri Jun 8 11:34:28 2018 +0100 Add 'osd blacklist' to default mon perms Ensure that the default permissions for clients include the 'osd blacklist' command; This ensures that in the event of a client crashing (due to power outage or segfault), the client and re-connect and write to any devices on reboot. Change-Id: I0b43dece4e1c56fb838b0147bfb75fb9906e6657 Closes-Bug: 1773449 ** Changed in: charms.ceph Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
Temporal solution is not use cephx on auth, and change it to none on charm, but i think this is not recommended. I tested that with QCOW2 and RAW images without any problem. Maybe can help to someone. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
** Changed in: charm-ceph-mon Assignee: (unassigned) => James Page (james-page) ** Changed in: charms.ceph Assignee: (unassigned) => James Page (james-page) ** Changed in: charm-ceph-mon Status: Triaged => In Progress ** Changed in: charms.ceph Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
** Changed in: charm-ceph-mon Assignee: (unassigned) => James Page (james-page) ** Changed in: charms.ceph Assignee: (unassigned) => James Page (james-page) ** Changed in: charm-ceph-mon Status: Triaged => In Progress ** Changed in: charms.ceph Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
OK figured this one out - the cephx keys are missing a permission which allows them to see blacklisted clients - as a result they can't deal with a hard crash: mon 'allow command "osd blacklist"' This is a charm issue after all. As a workaround you can manually update the existing client keys for nova-compute using: sudo ceph auth caps client.nova-compute mon 'allow r, allow command "osd blacklist"' osd 'allow rwx' from any mon unit. ** Changed in: nova Status: New => Invalid ** Changed in: ceph (Ubuntu) Status: New => Invalid ** Changed in: nova (Ubuntu) Status: New => Invalid ** Changed in: cloud-archive Status: New => Invalid ** Changed in: qemu (Ubuntu) Status: New => Invalid ** Also affects: charm-ceph-mon Importance: Undecided Status: New ** Also affects: charms.ceph Importance: Undecided Status: New ** Changed in: charms.ceph Status: New => Triaged ** Changed in: charm-ceph-mon Status: New => Triaged ** Changed in: charms.ceph Importance: Undecided => High ** Changed in: charm-ceph-mon Importance: Undecided => High ** Changed in: charm-ceph-mon Milestone: None => 18.08 ** Changed in: cloud-archive Assignee: Sean Feole (sfeole) => (unassigned) ** Changed in: nova (Ubuntu) Assignee: Sean Feole (sfeole) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Server, which is subscribed to qemu in Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1773449] Re: VM rbd backed block devices inconsistent after unexpected host outage
OK figured this one out - the cephx keys are missing a permission which allows them to see blacklisted clients - as a result they can't deal with a hard crash: mon 'allow command "osd blacklist"' This is a charm issue after all. As a workaround you can manually update the existing client keys for nova-compute using: sudo ceph auth caps client.nova-compute mon 'allow r, allow command "osd blacklist"' osd 'allow rwx' from any mon unit. ** Changed in: nova Status: New => Invalid ** Changed in: ceph (Ubuntu) Status: New => Invalid ** Changed in: nova (Ubuntu) Status: New => Invalid ** Changed in: cloud-archive Status: New => Invalid ** Changed in: qemu (Ubuntu) Status: New => Invalid ** Also affects: charm-ceph-mon Importance: Undecided Status: New ** Also affects: charms.ceph Importance: Undecided Status: New ** Changed in: charms.ceph Status: New => Triaged ** Changed in: charm-ceph-mon Status: New => Triaged ** Changed in: charms.ceph Importance: Undecided => High ** Changed in: charm-ceph-mon Importance: Undecided => High ** Changed in: charm-ceph-mon Milestone: None => 18.08 ** Changed in: cloud-archive Assignee: Sean Feole (sfeole) => (unassigned) ** Changed in: nova (Ubuntu) Assignee: Sean Feole (sfeole) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1773449 Title: VM rbd backed block devices inconsistent after unexpected host outage To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ceph-mon/+bug/1773449/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
