Reviewed: https://review.opendev.org/c/openstack/nova/+/955039 Committed: https://opendev.org/openstack/nova/commit/668d9d37f8ec1e8c9729c13f4d9f0435303a22fb Submitter: "Zuul (22348)" Branch: master
commit 668d9d37f8ec1e8c9729c13f4d9f0435303a22fb Author: melanie witt <[email protected]> Date: Mon Jul 14 21:44:22 2025 +0000 libvirt: Get info with abs path, rebase with rel path When a boot-from-volume instance with Cinder backed by NFS, when attempting to delete a volume snapshot after deleting a newer volume snapshot while the instance is stopped, the delete fails when Cinder raises RemoteFSInvalidBackingFile. The root cause is Nova using an absolute file path when specifying the backing file when rebasing the qcow2 image during the deletion of the newer volume snapshot. Then when the older volume snapshot is requested to be deleted, on the Cinder side it fails because Cinder detects the absolute path from Nova. Originally, Nova used a relative file back for the backing file path but it was changed to address a past bug #1885528 where an absolute file path was needed in order to call qemu_img_info(). This changes to using an absolute path only for the qemu_img_info() call and retain the relative path for the actual 'qemu-img rebase' command invocation. The 'qemu-img rebase' command is designed to work with both relative paths and absolute paths [1]. Closes-Bug: #2119353 [1] https://qemu-project.gitlab.io/qemu/tools/qemu-img.html#cmdoption-qemu-img-commands-arg-rebase Change-Id: I6b5fe3bba49461fb10bc81e19bd16acf39e84ec6 Signed-off-by: melanie witt <[email protected]> ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2119353 Title: Volume snapshot deletion of stopped Cinder NFS backed instance fails with RemoteFSInvalidBackingFile Status in OpenStack Compute (nova): Fix Released Bug description: This bug is the same problem as a past fixed bug [1] but somehow the fix was incomplete and recently reported downstream [2]. When a boot-from-volume instance with Cinder backed by NFS, when attempting to delete a volume snapshot after deleting a newer volume snapshot while the instance is stopped, the delete fails when Cinder raises RemoteFSInvalidBackingFile: DEBUG cinder.volume.drivers.remotefs [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Deleting online snapshot 637a614e-0037-4dae-a8bf-68c28c5a1353 of volume 5032d631-b69a-4a03-a46d-526f2a026f0f {{(pid=89061) _delete_snapshot /opt/stack/cinder/cinder/volume/drivers/remotefs.py:1278}} DEBUG cinder.volume.drivers.remotefs [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] snapshot_file for this snap is: volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf {{(pid=89061) _delete_snapshot /opt/stack/cinder/cinder/volume/drivers/remotefs.py:1307}} DEBUG oslo.privsep.daemon [-] privsep: reply[b1db715e-22bb-4ff7-b8d7-d4a53cde47a8]: (4, 'qcow2') {{(pid=98228) _call_back /opt/stack/data/venv/lib/python3.12/site-packages/oslo_privsep/daemon.py:512}} DEBUG oslo_concurrency.processutils [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Running cmd (subprocess): /opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf {{(pid=89061) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:349}} : PWD=/ ; USER=root ; COMMAND=/opt/stack/data/venv/bin/cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf (sudo:session): session opened for user root(uid=0) by (uid=1001) (sudo:session): session closed for user root DEBUG oslo_concurrency.processutils [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] CMD "/opt/stack/data/venv/bin/python3.12 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 -- sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf" returned: 0 in 0.229s {{(pid=89061) execute /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/processutils.py:372}} DEBUG cinder.coordination [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Lock "cinder-nfs-5032d631-b69a-4a03-a46d-526f2a026f0f" released by "delete_snapshot" :: held 0.239s {{(pid=89061) __release /opt/stack/cinder/cinder/coordination.py:157}} INFO cinder.message.api [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Creating message record for request_id = req-f4057282-a9a5-45e2-bca8-555831b07f22 DEBUG cinder.coordination [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Lock "cinder-637a614e-0037-4dae-a8bf-68c28c5a1353-delete_snapshot" released by "delete_snapshot" :: held 0.263s {{(pid=89061) __release /opt/stack/cinder/cinder/coordination.py:157}} ERROR oslo_messaging.rpc.server [None req-f4057282-a9a5-45e2-bca8-555831b07f22 tempest-TestVolumeBootPattern-1181586744 None] Exception during message handling: cinder.exception.RemoteFSInvalidBackingFile: File /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf has invalid backing file /opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f. ERROR oslo_messaging.rpc.server Traceback (most recent call last): ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/server.py", line 174, in _process_incoming ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 219, in wrapper ERROR oslo_messaging.rpc.server skip_clean = func(self, context, snapshot, *args, **kwargs) ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.12/site-packages/decorator.py", line 235, in fun ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 239, in _synchronized ERROR oslo_messaging.rpc.server return f(*a, **k) ERROR oslo_messaging.rpc.server ^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1351, in delete_snapshot ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception(): ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise ERROR oslo_messaging.rpc.server raise self.value ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 1337, in delete_snapshot ERROR oslo_messaging.rpc.server self.driver.delete_snapshot(snapshot) ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.12/site-packages/decorator.py", line 235, in fun ERROR oslo_messaging.rpc.server return caller(func, *(extras + args), **kw) ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/coordination.py", line 239, in _synchronized ERROR oslo_messaging.rpc.server return f(*a, **k) ERROR oslo_messaging.rpc.server ^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 620, in delete_snapshot ERROR oslo_messaging.rpc.server return self._delete_snapshot(snapshot) ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 1312, in _delete_snapshot ERROR oslo_messaging.rpc.server snapshot_path_img_info = self._qemu_img_info( ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 585, in _qemu_img_info ERROR oslo_messaging.rpc.server return super(NfsDriver, self)._qemu_img_info_base( ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/remotefs.py", line 900, in _qemu_img_info_base ERROR oslo_messaging.rpc.server raise exception.RemoteFSInvalidBackingFile( ERROR oslo_messaging.rpc.server cinder.exception.RemoteFSInvalidBackingFile: File /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f.144e7b21-bc8c-466b-a793-33d06ed0eacf has invalid backing file /opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-5032d631-b69a-4a03-a46d-526f2a026f0f. ERROR oslo_messaging.rpc.server Steps to reproduce: 1. Create a volume boot instance 2. Create two (or more) volume snapshots from above volume 3. Stop instance 4. Delete the newest volume snapshot 5. Delete the older volume snapshot (fails) The root cause is Nova using an absolute file path when specifying the backing file when rebasing the qcow2 image during the deletion of the newer volume snapshot. Then when the older volume snapshot is requested to be deleted, on the Cinder side it fails because Cinder detects the absolute path from Nova. [1] https://bugs.launchpad.net/nova/+bug/1885528 [2] https://issues.redhat.com/browse/OSPRH-17787 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2119353/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
