Reviewed: https://review.opendev.org/c/openstack/nova/+/957757 Committed: https://opendev.org/openstack/nova/commit/93c0f9bc749cea39f9cd1bd9d3e5a5585f1f6cac Submitter: "Zuul (22348)" Branch: master
commit 93c0f9bc749cea39f9cd1bd9d3e5a5585f1f6cac Author: Sean Mooney <[email protected]> Date: Fri Jun 13 18:15:46 2025 +0100 restrict swap volume to cinder This change tightens the validation around the attachment update API to ensure that it can only be called if the source volume has a non empty migration status. That means it will only accept a request to swap the volume if it is the result of a cinder volume migration. This change is being made to prevent the instance domain XML from getting out of sync with the nova BDM records and cinder connection info. In the future support for direct swap volume actions can be re-added if and only if the nova libvirt driver is updated to correctly modify the domain. The libvirt driver is the only driver that supported this API outside of a cinder orchestrated swap volume. By allowing the domain XML and BDMs to get out of sync if an admin later live-migrates the VM the host path will not be modified for the destination host. Normally this results in a live migration failure which often prompts the admin to cold migrate instead. however if the source device path exists on the destination the migration will proceed. This can lead to 2 VMs using the same host block device. At best this will cause a crash or data corruption. At worst it will allow one guest to access the data of another. Prior to this change there was an explicit warning in nova API ref stating that humans should never call this API because it can lead to this situation. Now it considered a hard error due to the security implications. Closes-Bug: #2112187 Depends-on: https://review.opendev.org/c/openstack/tempest/+/957753 Change-Id: I439338bd2f27ccd65a436d18c8cbc9c3127ee612 Signed-off-by: Sean Mooney <[email protected]> ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2112187 Title: Live migration fails after migrating the volume the instance has attached Status in OpenStack Compute (nova): Fix Released Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: In Progress Status in tempest: In Progress Status in watcher: Fix Released Bug description: Using Watcher, live migrating an instance that has a volume attached fails after migrating the volume. The migration fails with libvirt.libvirtError: Cannot access storage file '/dev/sdd': No such file or directory Steps to reproduce the nova live migrate issue after volume migration: 1. Create an audit template: openstack optimize audittemplate create zm_test hardware_maintenance --strategy zone_migration 2. Create vms with volumes: set -ex # Create Image IMG=cirros-0.5.2-x86_64-disk.img URL=http://download.cirros-cloud.net/0.5.2/$IMG DISK_FORMAT=qcow2 RAW=$IMG NUMBER_OF_INSTANCES=${1:-8} # Create flavor openstack flavor show m1.tiny || \ openstack flavor create --ram 512 --vcpus 1 --disk 1 --ephemeral 1 m1.tiny curl -L -# $URL > /tmp/$IMG if type qemu-img >/dev/null 2>&1; then RAW=$(echo $IMG | sed s/img/raw/g) qemu-img convert -f qcow2 -O raw /tmp/$IMG /tmp/$RAW DISK_FORMAT=raw fi openstack image show cirros || \ openstack image create --container-format bare --disk-format $DISK_FORMAT cirros < /tmp/$RAW # Create security group and icmp/ssh rules openstack security group show basic || { openstack security group create basic openstack security group rule create basic --protocol icmp --ingress --icmp-type -1 openstack security group rule create basic --protocol tcp --ingress --dst-port 22 } # Create an instance for (( i=1; i<${NUMBER_OF_INSTANCES}; i++ )); do NAME=test_${i} VOL_NAME=tes_vol_${i} # create a volume openstack volume create --size ${i} --image cirros ${VOL_NAME} openstack server show ${NAME} || { openstack server create --flavor m1.tiny --image cirros --nic net-id=private ${NAME} --security-group basic --wait fip=$(openstack floating ip create public -f value -c floating_ip_address) openstack server add floating ip ${NAME} $fip } openstack server add volume ${NAME} ${VOL_NAME} openstack server list --long done 3. Create the audit: openstack optimize audit create -a zm_test -p storage_pools='[{"src_pool": "jgilaber- watcher-3@lvmdriver-1#lvmdriver-1", "dst_pool": "jgilaber- watcher-2@lvmdriver-1#lvmdriver-1", "src_type": "lvmdriver-1", "dst_type": "lvmdriver-1"}]' -p compute_nodes='[{"src_node": "jgilaber-watcher-3", "dst_node": "jgilaber-watcher-1"}]' -p with_attached_volume=true Note the audit id that is generated 4. Check the action plan generated openstack optimize actionplan list --audit <audit_id> Note the action plan uuid 5. Start the proposed action plan openstack optimize actionplan start <action_plan_id> As a result, the audit triggers the migration of volumes and instances 5 and 2, test_2 was migrated succesfully, but test_5 failed. List of events: (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event list test_5 +------------------------------------------+--------------------------------------+----------------+----------------------------+ | Request ID | Server ID | Action | Start Time | +------------------------------------------+--------------------------------------+----------------+----------------------------+ | req-c805b970-97cd-424d-8415-7c30f4101399 | db250a8a-f788-48d4-bd56-76fd6890c11b | live-migration | 2025-05-30T08:42:08.000000 | | req-6850d126-7e0c-4d80-85d4-16216476a842 | db250a8a-f788-48d4-bd56-76fd6890c11b | swap_volume | 2025-05-30T08:41:16.000000 | | req-24a8e19a-a43c-4249-a700-2efa0edaf92f | db250a8a-f788-48d4-bd56-76fd6890c11b | attach_volume | 2025-05-30T08:35:51.000000 | | req-05a16a88-4266-45b6-8c35-bc2c0d83a434 | db250a8a-f788-48d4-bd56-76fd6890c11b | create | 2025-05-30T08:35:29.000000 | +------------------------------------------+--------------------------------------+----------------+----------------------------+ ``` ``` (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-6850d126-7e0c-4d80-85d4-16216476a842 +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | swap_volume | | events | details=, event='compute_swap_volume', finish_time='2025-05-30T08:41:41.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:41:16.000000', traceback= | | id | req-6850d126-7e0c-4d80-85d4-16216476a842 | | message | None | | project_id | 14f6d4c19ca04da0b68cd09b986428a8 | | request_id | req-6850d126-7e0c-4d80-85d4-16216476a842 | | start_time | 2025-05-30T08:41:16.000000 | | user_id | af0c6c70cc2a46e1abe39d3ad31de18c | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-c805b970-97cd-424d-8415-7c30f4101399 +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | live-migration | | events | details=, event='compute_rollback_live_migration_at_destination', finish_time='2025-05-30T08:42:41.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:38.000000', traceback= | | | details=, event='compute_pre_live_migration', finish_time='2025-05-30T08:42:29.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:17.000000', traceback= | | | details=, event='compute_live_migration', finish_time='2025-05-30T08:42:15.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:14.000000', traceback= | | | details=, event='compute_check_can_live_migrate_source', finish_time='2025-05-30T08:42:12.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:11.000000', traceback= | | | details=, event='compute_check_can_live_migrate_destination', finish_time='2025-05-30T08:42:13.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:09.000000', traceback= | | | details=, event='conductor_live_migrate_instance', finish_time='2025-05-30T08:42:14.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:08.000000', traceback= | | id | req-c805b970-97cd-424d-8415-7c30f4101399 | | message | None | | project_id | 14f6d4c19ca04da0b68cd09b986428a8 | | request_id | req-c805b970-97cd-424d-8415-7c30f4101399 | | start_time | 2025-05-30T08:42:08.000000 | | user_id | af0c6c70cc2a46e1abe39d3ad31de18c | +------------+----------------------------------------------------------------------------------------------------------------------------- Compute-nova logs from the source host with the error and guest xml https://paste.openstack.org/show/b9u14yO4KYJS31babgqt/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2112187/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
