Re: [Openstack-operators] ceph rbd root disk unexpected deletion
This was Newton booting from ephemeral disk. There were no delete events in the nova api database, just an unexpected stop when the kernel oom killer got qemu. > On Mar 17, 2017, at 8:28 AM, Saverio Proto wrote: > > Hello Mike, > > what version of openstack ? > is the instance booting from ephemeral disk or booting from cinder volume ? > > When you boot from volume, that will be the root disk of your > instance. The user could have clicked on "Delete Volume on Instance > Delete". It can be selected when creating a new instance. > > Saverio > > 2017-03-13 15:47 GMT+01:00 Mike Lowe : >> Over the weekend a user reported that his instance was in a stopped state >> and could not be started, on further examination it appears that the vm had >> crashed and the strange thing is that the root disk is now gone. Has >> anybody come across anything like this before? >> >> And why on earth is it attempting deletion of the rbd device without >> deletion of the instance? >> >> 2017-03-12 10:59:07.591 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] >> rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms >> failed >> 2017-03-12 10:59:17.613 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] >> rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms >> failed >> 2017-03-12 10:59:26.143 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] >> rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms >> failed >> ___ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> smime.p7s Description: S/MIME cryptographic signature ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] ceph rbd root disk unexpected deletion
Hello Mike, what version of openstack ? is the instance booting from ephemeral disk or booting from cinder volume ? When you boot from volume, that will be the root disk of your instance. The user could have clicked on "Delete Volume on Instance Delete". It can be selected when creating a new instance. Saverio 2017-03-13 15:47 GMT+01:00 Mike Lowe : > Over the weekend a user reported that his instance was in a stopped state and > could not be started, on further examination it appears that the vm had > crashed and the strange thing is that the root disk is now gone. Has anybody > come across anything like this before? > > And why on earth is it attempting deletion of the rbd device without deletion > of the instance? > > 2017-03-12 10:59:07.591 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] > rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms > failed > 2017-03-12 10:59:17.613 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] > rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms > failed > 2017-03-12 10:59:26.143 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] > rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms > failed > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] ceph rbd root disk unexpected deletion
I recall encountering something like this once when an instance termination failed halfway through - the root disk was removed but the instance record remained in the database. In my case, it didn't spontaneously happen, but was a requested termination that blew up at some point between removing the root disk and removing the instance record from the DB. If the instance action log doesn't indicate that anyone asked for the instance to be terminated, that's a bit weird. From: joml...@iu.edu Subject: Re:[Openstack-operators] ceph rbd root disk unexpected deletion Over the weekend a user reported that his instance was in a stopped state and could not be started, on further examination it appears that the vm had crashed and the strange thing is that the root disk is now gone. Has anybody come across anything like this before? And why on earth is it attempting deletion of the rbd device without deletion of the instance? 2017-03-12 10:59:07.591 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms failed 2017-03-12 10:59:17.613 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms failed 2017-03-12 10:59:26.143 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms failed ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] ceph rbd root disk unexpected deletion
Over the weekend a user reported that his instance was in a stopped state and could not be started, on further examination it appears that the vm had crashed and the strange thing is that the root disk is now gone. Has anybody come across anything like this before? And why on earth is it attempting deletion of the rbd device without deletion of the instance? 2017-03-12 10:59:07.591 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms failed 2017-03-12 10:59:17.613 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms failed 2017-03-12 10:59:26.143 3010 WARNING nova.virt.libvirt.storage.rbd_utils [-] rbd remove 4367a2e4-d704-490d-b3a6-129b9465cd0d_disk in pool ephemeral-vms failed smime.p7s Description: S/MIME cryptographic signature ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators