** Description changed: - Server has dependency of original base image used to Create Server or - Rebuild Server. If original image is deleted from glance, nova has - server.image.id reference to non-existent image. Rescue in stable device - rescue mode leads to driver error and Server in ERROR state. + [ Impact ] + + When rescuing an instance using stable device rescue mode (i.e. rescue + image contains `hw_rescue_bus` or `hw_rescue_device` properties), Nova's + libvirt driver attempts to look up the original base image metadata from + Glance via `instance.image_ref`. + + If the original base image (or snapshot/backup image after a rebuild) + has been deleted from Glance, the lookup raises + `nova.exception.ImageNotFound`. This leads to a driver error and puts + the instance into the unrecoverable `ERROR` state. + + The fix catches `nova.exception.ImageNotFound` during image metadata + retrieval in `nova/virt/libvirt/driver.py` and falls back to using the + stashed `instance.image_meta` already saved on the instance. + + [ Test Plan ] + + 1. Prepare test images: + $ openstack image save --file /tmp/cirros.qcow2 cirros + $ openstack image create --disk-format qcow2 --container-format bare --file /tmp/cirros.qcow2 test-base-img + $ openstack image create --disk-format qcow2 --container-format bare --file /tmp/cirros.qcow2 \ + --property hw_rescue_bus=scsi --property hw_rescue_device=cdrom test-rescue-img + + 2. Boot an instance from the base image and wait until it is ACTIVE: + $ openstack server create --image test-base-img --flavor m1.tiny --network private test-server + $ openstack server show test-server -f value -c status + + 3. Delete the base image from Glance: + $ openstack image delete test-base-img + + 4. Rescue the server with the rescue image: + $ openstack server rescue --image test-rescue-img test-server + + Verification: + - Without fix: The server rescue fails with "Driver Error: Image <UUID> could not be found." and the server transitions to ERROR state. + - With fix: The server successfully transitions to RESCUE state. Running `openstack server unrescue test-server` successfully restores the server to ACTIVE state. + + [ Where problems could occur ] + + - The modification is strictly localized to the libvirt driver's stable rescue path (`nova/virt/libvirt/driver.py`) inside the `hardware.check_hw_rescue_props()` block. + - It only triggers when `ImageNotFound` is encountered; regular rescue operations with existing Glance images follow the existing code path. + - Potential regressions could occur if the stashed `instance.image_meta` was malformed or missing attributes required for disk mapping generation, which could cause rescue failures. However, `instance.image_meta` is saved upon instance create/rebuild and contains the necessary properties. + - Standard rescue (non-stable rescue mode) and boot-from-volume instances without `image_ref` are unaffected. + + [ Other Info ] + + - Upstream commit: 714894a7ff81e23f5ac7545a6b91ea484447a988 + - Fixed upstream in OpenStack Nova 30.0.0 (Dalmatian). + + Original Bug Description Below + =========== + Server has dependency of original base image used to Create Server or Rebuild Server. If original image is deleted from glance, nova has server.image.id reference to non-existent image. Rescue in stable device rescue mode leads to driver error and Server in ERROR state. Step to reproduce: 1. Create server from glance image. 2. Delete original glance image. 3. Run rescue server with input parameter of rescue image CD with hw_rescue_device and hw_rescue_bus image properties defined. Second use case: 1. Restore Server from backup (with server Rebuild) 2. Delete backup image. 3. Run rescue in stable device mode. Result: Instance 73706a9a-e976-4024-8068-c439404ec953 cannot be rescued: Driver Error: Image be0e302d-79bd-44e1-93a5-68661e93b43a could not be found. Result2: VM in ERROR state. Expected: VM returned to original state, or in shutdown state. Expected2: I see no reason to keep dependency from original image for any operation with server. I tested all other operations with server, and they all were succesful, even if base image deleted. Libvirt driver code shows that rescue mode use only 1 property from image_meta to determine bus type for device attached. So I see no reason to raise exception if image_meta data if empty. Stack trace for openstack ZED: 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [None req-6611c9f1-63f1-497a-a1bc-51e770d2df09 a27cbf048299483592d84f13e9fe9797 b7df78883aad4ef7ba0a2778c7741462 - - 8a096fed8060480f8f906f57fca19780 8a096fed8060480f8f906f57fca19780] [instance: 73706a9a-e976-4024-8068-c439404ec953] Error trying to Rescue Instance: nova.exception.ImageNotFound: Image be0e302d-79bd-44e1-93a5-68661e93b43a could not be found. 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] Traceback (most recent call last): - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 285, in show - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] image = self._client.call(context, 2, 'get', args=(image_id,)) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 191, in call - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] result = getattr(controller, method)(*args, **kwargs) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return self._get(image_id) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return RequestIdProxy(wrapped(*args, **kwargs)) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] resp, body = self.http_client.get(url, headers=header) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return self.request(url, 'GET', **kwargs) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 380, in request - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return self._handle_response(resp) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 120, in _handle_response - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] raise exc.from_response(resp, resp.content) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 285, in show + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] image = self._client.call(context, 2, 'get', args=(image_id,)) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 191, in call + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] result = getattr(controller, method)(*args, **kwargs) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return self._get(image_id) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return RequestIdProxy(wrapped(*args, **kwargs)) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] resp, body = self.http_client.get(url, headers=header) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return self.request(url, 'GET', **kwargs) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 380, in request + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return self._handle_response(resp) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 120, in _handle_response + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] raise exc.from_response(resp, resp.content) 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] glanceclient.exc.HTTPNotFound: HTTP 404 Not Found: No image found with ID be0e302d-79bd-44e1-93a5-68661e93b43a 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] During handling of the above exception, another exception occurred: 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] Traceback (most recent call last): - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4485, in rescue_instance - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] self.driver.rescue(context, instance, network_info, - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4249, in rescue - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] image_meta = objects.ImageMeta.from_image_ref( - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/objects/image_meta.py", line 151, in from_image_ref - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] image_meta = image_api.get(context, image_ref) - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 1205, in get - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return session.show(context, image_id, - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 287, in show - 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] _reraise_translated_image_exception(image_id) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 4485, in rescue_instance + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] self.driver.rescue(context, instance, network_info, + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4249, in rescue + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] image_meta = objects.ImageMeta.from_image_ref( + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/objects/image_meta.py", line 151, in from_image_ref + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] image_meta = image_api.get(context, image_ref) + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 1205, in get + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] return session.show(context, image_id, + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] File "/usr/lib/python3/dist-packages/nova/image/glance.py", line 287, in show + 2023-01-11 07:33:28.761 1207602 ERROR nova.compute.manager [instance: 73706a9a-e976-4024-8068-c439404ec953] _reraise_translated_image_exception(image_id)
** Summary changed: - Server Rescue leads to VM ERROR state if VM base image is deleted + [SRU] Server Rescue leads to VM ERROR state if VM base image is deleted ** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2002606 Title: [SRU] Server Rescue leads to VM ERROR state if VM base image is deleted To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2002606/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
