Public bug reported: Description ===========
When attempting to rescue a volume-based instance using an image without the hw_rescue_device and hw_rescue_bus properties set (i.e. which doesn't support the stable rescue mode) I would expect the API call to fail leaving the state in its existing state. Instead Nova appears to go ahead with the rescue which fails (since non-stable rescues for volume backed instances aren't supported) leading to an unrecoverable, broken instance. Steps to reproduce ================== 1. Create an instance booting from a volume (in my case the instance is booting from a volume in Ceph) 2. Attempt to trigger a rescue using an image *without* the hw_rescue_device and hw_rescue_bus properties set using: $ openstack --os-compute-api-version 2.87 server rescue --image <IMAGE UUID> <INSTANCE UUID> Expected result =============== I would have expected step 2 to fail with an error along the lines of "Cannot rescue a volume-backed instance", leaving the instance unchanged. My crude skimming of the Nova source appears to try and perform this check here: https://github.com/openstack/nova/blob/f766db261634c8f95f874ba132159f148de9e8bf/nova/compute/api.py#L4514-L4517. Actual result ============= The rescue command returns normally and the instance enters the rescue state very briefly before ending up in the Error state with the message: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-43 Looking at the nova-compute logs on the hypervisor gets the following 2022-06-16 11:53:40.765 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Attempting rescue 2022-06-16 11:53:40.768 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Creating image 2022-06-16 11:53:40.943 32936 WARNING nova.compute.manager [req-a39b605e-e2db-45ce-8f7f-c92a192d9fbe 023555cbd3a2484f89984da17ae9bbfb 0c6d3af6910e429894c203af88fc96c4 - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Received unexpected event network-vif-unplugged-464bd8c6-2b41-4648-abe8-d900c88dd996 for instance with vm_state active a> 2022-06-16 11:53:50.665 32936 INFO nova.virt.libvirt.driver [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Instance destroyed successfully. 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Error launching a defined domain with XML: <domain type='kvm'> <name>instance-00015604</name> <uuid>05043c69-3533-4396-a610-98e564fa3ead</uuid> ...snip... <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <auth username='cinder'> <secret type='ceph' uuid='...'/> </auth> <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk.rescue'> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> </source> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <auth username='cinder'> <secret type='ceph' uuid='...'/> </auth> <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk'> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> </source> <target dev='sdb' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> ...snip... : libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest Traceback (most recent call last): 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest six.reraise(c, e, tb) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest raise value 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Error trying to Rescue Instance: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] Traceback (most recent call last): 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self.driver.rescue(context, instance, network_info, 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self._create_guest( 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] guest.launch(pause=pause) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] LOG.exception('Error launching a defined domain with XML: %s', 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self.force_reraise() 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] raise self.value 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] return self._domain.createWithFlags(flags) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] result = proxy_call(self._autowrap, f, *args, **kwargs) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] rv = execute(f, *args, **kwargs) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] six.reraise(c, e, tb) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] raise value 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] rv = meth(*args, **kwargs) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] 2022-06-16 11:53:52.310 32936 INFO nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Successfully reverted task state from rescuing on failure for instance. 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Exception during message handling: nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.driver.rescue(context, instance, network_info, 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self._create_guest( 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server guest.launch(pause=pause) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server LOG.exception('Error launching a defined domain with XML: %s', 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return self._domain.createWithFlags(flags) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification( 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 183, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. " 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 154, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1433, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context, 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4214, in rescue_instance 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise exception.InstanceNotRescuable( 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 2022-06-16 11:53:55.555 32936 INFO nova.compute.manager [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] VM Stopped (Lifecycle Event) 2022-06-16 11:53:55.610 32936 INFO nova.compute.manager [req-7790122f-6942-4296-bdfa-37362785baec - - - - -] [instance: 05043c69-3533-4396-a610-98e564fa3ead] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. Environment =========== Nova f766db261634c8f95f874ba132159f148de9e8bf (installed from Git by OSA) Hypervisor: Libvirt + KVM (Libvirt 6.0.0-0ubuntu8.16) Storage type: Ceph Networking: Neutron (using Linux networking, no OVS) ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1978958 Title: Attempting to rescue volume-based instance doesn't return error but leaves VM in error state Status in OpenStack Compute (nova): New Bug description: Description =========== When attempting to rescue a volume-based instance using an image without the hw_rescue_device and hw_rescue_bus properties set (i.e. which doesn't support the stable rescue mode) I would expect the API call to fail leaving the state in its existing state. Instead Nova appears to go ahead with the rescue which fails (since non-stable rescues for volume backed instances aren't supported) leading to an unrecoverable, broken instance. Steps to reproduce ================== 1. Create an instance booting from a volume (in my case the instance is booting from a volume in Ceph) 2. Attempt to trigger a rescue using an image *without* the hw_rescue_device and hw_rescue_bus properties set using: $ openstack --os-compute-api-version 2.87 server rescue --image <IMAGE UUID> <INSTANCE UUID> Expected result =============== I would have expected step 2 to fail with an error along the lines of "Cannot rescue a volume-backed instance", leaving the instance unchanged. My crude skimming of the Nova source appears to try and perform this check here: https://github.com/openstack/nova/blob/f766db261634c8f95f874ba132159f148de9e8bf/nova/compute/api.py#L4514-L4517. Actual result ============= The rescue command returns normally and the instance enters the rescue state very briefly before ending up in the Error state with the message: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-43 Looking at the nova-compute logs on the hypervisor gets the following 2022-06-16 11:53:40.765 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Attempting rescue 2022-06-16 11:53:40.768 32936 INFO nova.virt.libvirt.driver [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Creating image 2022-06-16 11:53:40.943 32936 WARNING nova.compute.manager [req-a39b605e-e2db-45ce-8f7f-c92a192d9fbe 023555cbd3a2484f89984da17ae9bbfb 0c6d3af6910e429894c203af88fc96c4 - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Received unexpected event network-vif-unplugged-464bd8c6-2b41-4648-abe8-d900c88dd996 for instance with vm_state active a> 2022-06-16 11:53:50.665 32936 INFO nova.virt.libvirt.driver [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Instance destroyed successfully. 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Error launching a defined domain with XML: <domain type='kvm'> <name>instance-00015604</name> <uuid>05043c69-3533-4396-a610-98e564fa3ead</uuid> ...snip... <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <auth username='cinder'> <secret type='ceph' uuid='...'/> </auth> <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk.rescue'> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> </source> <target dev='sda' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='network' device='disk'> <driver name='qemu' type='raw' cache='writeback' discard='unmap'/> <auth username='cinder'> <secret type='ceph' uuid='...'/> </auth> <source protocol='rbd' name='ephemeral-vms/05043c69-3533-4396-a610-98e564fa3ead_disk'> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> <host name='...snip...' port='6789'/> </source> <target dev='sdb' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> ...snip... : libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest Traceback (most recent call last): 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest six.reraise(c, e, tb) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest raise value 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.807 32936 ERROR nova.virt.libvirt.guest 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Error trying to Rescue Instance: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] Traceback (most recent call last): 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self.driver.rescue(context, instance, network_info, 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self._create_guest( 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] guest.launch(pause=pause) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] LOG.exception('Error launching a defined domain with XML: %s', 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] self.force_reraise() 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] raise self.value 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] return self._domain.createWithFlags(flags) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] result = proxy_call(self._autowrap, f, *args, **kwargs) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] rv = execute(f, *args, **kwargs) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] six.reraise(c, e, tb) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] raise value 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] rv = meth(*args, **kwargs) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:51.808 32936 ERROR nova.compute.manager [instance: 05043c69-3533-4396-a610-98e564fa3ead] 2022-06-16 11:53:52.310 32936 INFO nova.compute.manager [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] [instance: 05043c69-3533-4396-a610-98e564fa3ead] Successfully reverted task state from rescuing on failure for instance. 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server [req-bd8be6a4-2195-498e-9209-caed5c525524 ce20f804a02fc250ec44394149d69a682aed245fd50194cd917fd238c2dc52c1 b259e9c9e29949f287d63e3d05ae9a8a - default default] Exception during message handling: nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4207, in rescue_instance 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.driver.rescue(context, instance, network_info, 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 4145, in rescue 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self._create_guest( 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 7197, in _create_guest 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server guest.launch(pause=pause) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 162, in launch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server LOG.exception('Error launching a defined domain with XML: %s', 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/virt/libvirt/guest.py", line 159, in launch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return self._domain.createWithFlags(flags) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/six.py", line 719, in reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/libvirt.py", line 1265, in createWithFlags 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 71, in wrapped 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification( 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 63, in wrapped 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 183, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. " 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 154, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1433, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 211, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context, 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise self.value 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 200, in decorated_function 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0/lib/python3.8/site-packages/nova/compute/manager.py", line 4214, in rescue_instance 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server raise exception.InstanceNotRescuable( 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server nova.exception.InstanceNotRescuable: Instance 05043c69-3533-4396-a610-98e564fa3ead cannot be rescued: Driver Error: internal error: qemu unexpectedly closed the monitor: 2022-06-16T11:53:51.583795Z qemu-system-x86_64: -blockdev {"driver":"rbd","pool":"ephemeral-vms","image":"05043c69-3533-4396-a610-98e564fa3ead_disk","server":[{"host":"10.81.240.10","port":"6789"},{"host":"10.81.240.11","port":"6789"},{"host":"10.81.240.12","port":"6789"}],"user":"cinder","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-secret0","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: error reading header from 05043c69-3533-4396-a610-98e564fa3ead_disk: No such file or directory 2022-06-16 11:53:52.312 32936 ERROR oslo_messaging.rpc.server 2022-06-16 11:53:55.555 32936 INFO nova.compute.manager [-] [instance: 05043c69-3533-4396-a610-98e564fa3ead] VM Stopped (Lifecycle Event) 2022-06-16 11:53:55.610 32936 INFO nova.compute.manager [req-7790122f-6942-4296-bdfa-37362785baec - - - - -] [instance: 05043c69-3533-4396-a610-98e564fa3ead] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. Environment =========== Nova f766db261634c8f95f874ba132159f148de9e8bf (installed from Git by OSA) Hypervisor: Libvirt + KVM (Libvirt 6.0.0-0ubuntu8.16) Storage type: Ceph Networking: Neutron (using Linux networking, no OVS) To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1978958/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp