Public bug reported: Using Watcher, live migrating an instance that has a volume attached fails after migrating the volume. The migration fails with libvirt.libvirtError: Cannot access storage file '/dev/sdd': No such file or directory
Steps to reproduce the nova live migrate issue after volume migration: 1. Create an audit template: openstack optimize audittemplate create zm_test hardware_maintenance --strategy zone_migration 2. Create vms with volumes: set -ex # Create Image IMG=cirros-0.5.2-x86_64-disk.img URL=http://download.cirros-cloud.net/0.5.2/$IMG DISK_FORMAT=qcow2 RAW=$IMG NUMBER_OF_INSTANCES=${1:-8} # Create flavor openstack flavor show m1.tiny || \ openstack flavor create --ram 512 --vcpus 1 --disk 1 --ephemeral 1 m1.tiny curl -L -# $URL > /tmp/$IMG if type qemu-img >/dev/null 2>&1; then RAW=$(echo $IMG | sed s/img/raw/g) qemu-img convert -f qcow2 -O raw /tmp/$IMG /tmp/$RAW DISK_FORMAT=raw fi openstack image show cirros || \ openstack image create --container-format bare --disk-format $DISK_FORMAT cirros < /tmp/$RAW # Create security group and icmp/ssh rules openstack security group show basic || { openstack security group create basic openstack security group rule create basic --protocol icmp --ingress --icmp-type -1 openstack security group rule create basic --protocol tcp --ingress --dst-port 22 } # Create an instance for (( i=1; i<${NUMBER_OF_INSTANCES}; i++ )); do NAME=test_${i} VOL_NAME=tes_vol_${i} # create a volume openstack volume create --size ${i} --image cirros ${VOL_NAME} openstack server show ${NAME} || { openstack server create --flavor m1.tiny --image cirros --nic net-id=private ${NAME} --security-group basic --wait fip=$(openstack floating ip create public -f value -c floating_ip_address) openstack server add floating ip ${NAME} $fip } openstack server add volume ${NAME} ${VOL_NAME} openstack server list --long done 3. Create the audit: openstack optimize audit create -a zm_test -p storage_pools='[{"src_pool": "jgilaber- watcher-3@lvmdriver-1#lvmdriver-1", "dst_pool": "jgilaber- watcher-2@lvmdriver-1#lvmdriver-1", "src_type": "lvmdriver-1", "dst_type": "lvmdriver-1"}]' -p compute_nodes='[{"src_node": "jgilaber- watcher-3", "dst_node": "jgilaber-watcher-1"}]' -p with_attached_volume=true Note the audit id that is generated 4. Check the action plan generated openstack optimize actionplan list --audit <audit_id> Note the action plan uuid 5. Start the proposed action plan openstack optimize actionplan start <action_plan_id> As a result, the audit triggers the migration of volumes and instances 5 and 2, test_2 was migrated succesfully, but test_5 failed. List of events: (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event list test_5 +------------------------------------------+--------------------------------------+----------------+----------------------------+ | Request ID | Server ID | Action | Start Time | +------------------------------------------+--------------------------------------+----------------+----------------------------+ | req-c805b970-97cd-424d-8415-7c30f4101399 | db250a8a-f788-48d4-bd56-76fd6890c11b | live-migration | 2025-05-30T08:42:08.000000 | | req-6850d126-7e0c-4d80-85d4-16216476a842 | db250a8a-f788-48d4-bd56-76fd6890c11b | swap_volume | 2025-05-30T08:41:16.000000 | | req-24a8e19a-a43c-4249-a700-2efa0edaf92f | db250a8a-f788-48d4-bd56-76fd6890c11b | attach_volume | 2025-05-30T08:35:51.000000 | | req-05a16a88-4266-45b6-8c35-bc2c0d83a434 | db250a8a-f788-48d4-bd56-76fd6890c11b | create | 2025-05-30T08:35:29.000000 | +------------------------------------------+--------------------------------------+----------------+----------------------------+ ``` ``` (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-6850d126-7e0c-4d80-85d4-16216476a842 +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | swap_volume | | events | details=, event='compute_swap_volume', finish_time='2025-05-30T08:41:41.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:41:16.000000', traceback= | | id | req-6850d126-7e0c-4d80-85d4-16216476a842 | | message | None | | project_id | 14f6d4c19ca04da0b68cd09b986428a8 | | request_id | req-6850d126-7e0c-4d80-85d4-16216476a842 | | start_time | 2025-05-30T08:41:16.000000 | | user_id | af0c6c70cc2a46e1abe39d3ad31de18c | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-c805b970-97cd-424d-8415-7c30f4101399 +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | live-migration | | events | details=, event='compute_rollback_live_migration_at_destination', finish_time='2025-05-30T08:42:41.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:38.000000', traceback= | | | details=, event='compute_pre_live_migration', finish_time='2025-05-30T08:42:29.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:17.000000', traceback= | | | details=, event='compute_live_migration', finish_time='2025-05-30T08:42:15.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:14.000000', traceback= | | | details=, event='compute_check_can_live_migrate_source', finish_time='2025-05-30T08:42:12.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:11.000000', traceback= | | | details=, event='compute_check_can_live_migrate_destination', finish_time='2025-05-30T08:42:13.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:09.000000', traceback= | | | details=, event='conductor_live_migrate_instance', finish_time='2025-05-30T08:42:14.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:08.000000', traceback= | | id | req-c805b970-97cd-424d-8415-7c30f4101399 | | message | None | | project_id | 14f6d4c19ca04da0b68cd09b986428a8 | | request_id | req-c805b970-97cd-424d-8415-7c30f4101399 | | start_time | 2025-05-30T08:42:08.000000 | | user_id | af0c6c70cc2a46e1abe39d3ad31de18c | +------------+----------------------------------------------------------------------------------------------------------------------------- Compute-nova logs from the source host with the error and guest xml https://paste.openstack.org/show/b9u14yO4KYJS31babgqt/ ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2112187 Title: Live migration fails after migrating the volume the instance has attached Status in OpenStack Compute (nova): New Bug description: Using Watcher, live migrating an instance that has a volume attached fails after migrating the volume. The migration fails with libvirt.libvirtError: Cannot access storage file '/dev/sdd': No such file or directory Steps to reproduce the nova live migrate issue after volume migration: 1. Create an audit template: openstack optimize audittemplate create zm_test hardware_maintenance --strategy zone_migration 2. Create vms with volumes: set -ex # Create Image IMG=cirros-0.5.2-x86_64-disk.img URL=http://download.cirros-cloud.net/0.5.2/$IMG DISK_FORMAT=qcow2 RAW=$IMG NUMBER_OF_INSTANCES=${1:-8} # Create flavor openstack flavor show m1.tiny || \ openstack flavor create --ram 512 --vcpus 1 --disk 1 --ephemeral 1 m1.tiny curl -L -# $URL > /tmp/$IMG if type qemu-img >/dev/null 2>&1; then RAW=$(echo $IMG | sed s/img/raw/g) qemu-img convert -f qcow2 -O raw /tmp/$IMG /tmp/$RAW DISK_FORMAT=raw fi openstack image show cirros || \ openstack image create --container-format bare --disk-format $DISK_FORMAT cirros < /tmp/$RAW # Create security group and icmp/ssh rules openstack security group show basic || { openstack security group create basic openstack security group rule create basic --protocol icmp --ingress --icmp-type -1 openstack security group rule create basic --protocol tcp --ingress --dst-port 22 } # Create an instance for (( i=1; i<${NUMBER_OF_INSTANCES}; i++ )); do NAME=test_${i} VOL_NAME=tes_vol_${i} # create a volume openstack volume create --size ${i} --image cirros ${VOL_NAME} openstack server show ${NAME} || { openstack server create --flavor m1.tiny --image cirros --nic net-id=private ${NAME} --security-group basic --wait fip=$(openstack floating ip create public -f value -c floating_ip_address) openstack server add floating ip ${NAME} $fip } openstack server add volume ${NAME} ${VOL_NAME} openstack server list --long done 3. Create the audit: openstack optimize audit create -a zm_test -p storage_pools='[{"src_pool": "jgilaber- watcher-3@lvmdriver-1#lvmdriver-1", "dst_pool": "jgilaber- watcher-2@lvmdriver-1#lvmdriver-1", "src_type": "lvmdriver-1", "dst_type": "lvmdriver-1"}]' -p compute_nodes='[{"src_node": "jgilaber-watcher-3", "dst_node": "jgilaber-watcher-1"}]' -p with_attached_volume=true Note the audit id that is generated 4. Check the action plan generated openstack optimize actionplan list --audit <audit_id> Note the action plan uuid 5. Start the proposed action plan openstack optimize actionplan start <action_plan_id> As a result, the audit triggers the migration of volumes and instances 5 and 2, test_2 was migrated succesfully, but test_5 failed. List of events: (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event list test_5 +------------------------------------------+--------------------------------------+----------------+----------------------------+ | Request ID | Server ID | Action | Start Time | +------------------------------------------+--------------------------------------+----------------+----------------------------+ | req-c805b970-97cd-424d-8415-7c30f4101399 | db250a8a-f788-48d4-bd56-76fd6890c11b | live-migration | 2025-05-30T08:42:08.000000 | | req-6850d126-7e0c-4d80-85d4-16216476a842 | db250a8a-f788-48d4-bd56-76fd6890c11b | swap_volume | 2025-05-30T08:41:16.000000 | | req-24a8e19a-a43c-4249-a700-2efa0edaf92f | db250a8a-f788-48d4-bd56-76fd6890c11b | attach_volume | 2025-05-30T08:35:51.000000 | | req-05a16a88-4266-45b6-8c35-bc2c0d83a434 | db250a8a-f788-48d4-bd56-76fd6890c11b | create | 2025-05-30T08:35:29.000000 | +------------------------------------------+--------------------------------------+----------------+----------------------------+ ``` ``` (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-6850d126-7e0c-4d80-85d4-16216476a842 +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | swap_volume | | events | details=, event='compute_swap_volume', finish_time='2025-05-30T08:41:41.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:41:16.000000', traceback= | | id | req-6850d126-7e0c-4d80-85d4-16216476a842 | | message | None | | project_id | 14f6d4c19ca04da0b68cd09b986428a8 | | request_id | req-6850d126-7e0c-4d80-85d4-16216476a842 | | start_time | 2025-05-30T08:41:16.000000 | | user_id | af0c6c70cc2a46e1abe39d3ad31de18c | +------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- (venv) ubuntu@jgilaber-watcher-1:/opt/stack/watcher$ openstack server event show test_5 req-c805b970-97cd-424d-8415-7c30f4101399 +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | action | live-migration | | events | details=, event='compute_rollback_live_migration_at_destination', finish_time='2025-05-30T08:42:41.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:38.000000', traceback= | | | details=, event='compute_pre_live_migration', finish_time='2025-05-30T08:42:29.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:17.000000', traceback= | | | details=, event='compute_live_migration', finish_time='2025-05-30T08:42:15.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:14.000000', traceback= | | | details=, event='compute_check_can_live_migrate_source', finish_time='2025-05-30T08:42:12.000000', host='jgilaber-watcher-3', | | | host_id='381be10e2d2b6140b67be690bf8877fa69223584c6a2c4a513155085', result='Success', start_time='2025-05-30T08:42:11.000000', traceback= | | | details=, event='compute_check_can_live_migrate_destination', finish_time='2025-05-30T08:42:13.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:09.000000', traceback= | | | details=, event='conductor_live_migrate_instance', finish_time='2025-05-30T08:42:14.000000', host='jgilaber-watcher-1', | | | host_id='636609f8b2efe009a9bf62531dbfd40694ee83a1bb9d81403df7c404', result='Success', start_time='2025-05-30T08:42:08.000000', traceback= | | id | req-c805b970-97cd-424d-8415-7c30f4101399 | | message | None | | project_id | 14f6d4c19ca04da0b68cd09b986428a8 | | request_id | req-c805b970-97cd-424d-8415-7c30f4101399 | | start_time | 2025-05-30T08:42:08.000000 | | user_id | af0c6c70cc2a46e1abe39d3ad31de18c | +------------+----------------------------------------------------------------------------------------------------------------------------- Compute-nova logs from the source host with the error and guest xml https://paste.openstack.org/show/b9u14yO4KYJS31babgqt/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2112187/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
