Public bug reported: As discussed in detail on a k8s cloud-provider-openstack bug [1], it appears that Nova always times out when waiting for libvirt events during a disk detach, with the following warning appearing in logs:
``` Jul 24 15:48:56 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.block_device [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] [instance: e8d15440-1442-49c3-9356-4592cdb697d2] Attempting to driver detach volume df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from mountpoint /dev/vdb Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: WARNING nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Waiting for libvirt event about the detach of device vdb with device alias ua-df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from instance e8d15440-1442-49c3-9356-4592cdb697d2 is timed out. Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Successfully detached device vdb from instance e8d15440-1442-49c3-9356-4592cdb697d2 from the live domain config. ``` This appears to have started in 2024.1 (Caracal) and appears regardless of distro. The net of effect of this is a 20 second (by default) timeout for each disk detach. This is proving particularly problematic for CPO since our tests do a lot of attach/detach calls (for creating/deleting PVCs) and run tests in series, however, we also see the same issue in the Nova CI. [1] https://github.com/kubernetes/cloud-provider- openstack/pull/2742#issuecomment-3135940269 ** Affects: nova Importance: Undecided Status: Confirmed ** Tags: libvirt ** Changed in: nova Status: New => Confirmed ** Tags added: libvirt -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2119114 Title: Consistent timeouts waiting for libvirt event during disk detach Status in OpenStack Compute (nova): Confirmed Bug description: As discussed in detail on a k8s cloud-provider-openstack bug [1], it appears that Nova always times out when waiting for libvirt events during a disk detach, with the following warning appearing in logs: ``` Jul 24 15:48:56 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.block_device [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] [instance: e8d15440-1442-49c3-9356-4592cdb697d2] Attempting to driver detach volume df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from mountpoint /dev/vdb Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: WARNING nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Waiting for libvirt event about the detach of device vdb with device alias ua-df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from instance e8d15440-1442-49c3-9356-4592cdb697d2 is timed out. Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Successfully detached device vdb from instance e8d15440-1442-49c3-9356-4592cdb697d2 from the live domain config. ``` This appears to have started in 2024.1 (Caracal) and appears regardless of distro. The net of effect of this is a 20 second (by default) timeout for each disk detach. This is proving particularly problematic for CPO since our tests do a lot of attach/detach calls (for creating/deleting PVCs) and run tests in series, however, we also see the same issue in the Nova CI. [1] https://github.com/kubernetes/cloud-provider- openstack/pull/2742#issuecomment-3135940269 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2119114/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
