** Description changed: + [ Impact ] + + Starting with libvirt 9.5.0, the default behavior for TAP devices changed; it now expects to manage the creation and lifecycle of the TAP device itself. + OpenStack Nova’s networking drivers are designed to pre-create the TAP device before handing it off to libvirt. Because libvirt 10.0.0 tries to "own" the device Nova already created, it fails to launch the instance, resulting in VirtDriver errors and instances stuck in an ERROR state. + + The patch explicitly adds managed="no" to the interface configuration in + the libvirt domain XML. This tells libvirt to skip its management + attempt and simply use the device provided by Nova, restoring the + intended workflow. + + [ Test Plan ] + + Ubuntu does not have support for Neutron Calico (it isn't packaged) but to test and verify this issue we don't need it because it is just a matter of adding a tap device to an existing vm which we can do manually. + + To test this we can do: + + * Deploy Openstack with Neutron OVN + + * Create a guest vm with one port (the tap device will be created by + libvirt) + + # openstack server create --flavor m1.tiny --image cirros --network + test-net test-vm + + * Stop the vm + + # openstack server stop test-vm + or + # virsh shutdown instance-00000001 + + * Manually create a new tap device and add it to the vm libvirt xml + + # sudo ip tuntap add dev tap1 mode tap + # sudo ip link set tap1 up + # virsh edit instance-00000001 + + On the <interface> section replace with <target dev='tap1'/> + + * Start the vm + + # virsh start instance-00000001 + error: Failed to start domain 'instance-00000001' + error: Requested operation is not valid: The tap1 interface already exists + + * Without the patch this will cause an error but with the patch it + should work. + + * With the patch, the VM can boot successfully. + + This patch explicitly adds managed="no" to the interface configuration + in the libvirt domain XML. This tells libvirt to skip its management + attempt and simply use the device provided by Nova, restoring the + intended workflow. + + We will do this manually and edit the XML again and add managed = no. + + Now try to start the VM again; + # virsh start instance-00000001 + Domain 'instance-00000001' started + + [Where problems could occur] + + This change specifically targets the XML generation for TAP interfaces. + Since Noble requires libvirt >= 10.0.0, we are not worried about backwards compatibility with extremely old libvirt versions that might not recognize the attribute. + + [ Other Info ] + + The bug has been reported upstream: + https://bugs.launchpad.net/nova/+bug/2033681 + + This fix is already merged upstream in Nova (see: + https://review.opendev.org/c/openstack/nova/+/967570) and is required + for Nova to function correctly on any distribution using libvirt 9.5.0 + or newer, which includes Ubuntu Noble. + + [ Old description ] Description =========== Calico (out of tree) uses vif type tap. But libvirt doesn't like pre-existing tap devices https://github.com/libvirt/libvirt/commit/a2ae3d299cf from libvirt 9.5.0. This causes openstack clusters that run calico networking backend to fail during instance creation. Steps to reproduce ================== - Ubuntu does not have support for Neutron Calico (it isnt packaged) but to test and verify this issue we don't need it because it is just a matter of adding a tap device to an existing vm which we can do manually. To test this we can do: - - * Deploy openstack with neutron OVN - * Create a guest vm with one port (the tap device will be created by libvirt) - * Stop the vm - * Manually create a new tap device and add it to the vm libvirt xml - * Start the vm - * Without the patch this will cause an error but with the patch it should work Expected result =============== The VM is able to boot without any problems Actual result Other information ================= 13:34:38 < sean-k-mooney> calico is apparently still using vif type tap https://github.com/projectcalico/calico/blob/cf7fa35475eba84f5afcd7f53ac7d07dcb403202/networking- calico/networking_calico/plugins/ml2/drivers/calico/test/lib.py#L66C31-L66C34 13:35:06 < sean-k-mooney> vif type tap is not supported by our os-vif code so its usign the legacy fallback 13:35:51 < sean-k-mooney> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L595-L596 13:36:15 < sean-k-mooney> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L420-L430 13:36:48 < sean-k-mooney> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L44-L55 13:37:40 < sean-k-mooney> zer0c00l: with that said the tap was always ment to be created by libvirt so it sound like calico might have been doing things it shoudl not have been 13:38:03 < zer0c00l> sean-k-mooney: Thanks for looking into this. :( 13:38:36 < sean-k-mooney> we could proably correct this with a bug fix 13:38:52 < sean-k-mooney> jsut setting managed='no' 13:39:13 < sean-k-mooney> here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L427 13:39:54 < sean-k-mooney> the problem is that the there is no way to test this really upstream 13:40:06 < sean-k-mooney> well beyond unit/fucntional tests 13:40:12 < sean-k-mooney> but we dont have any calico ci 13:40:37 < sean-k-mooney> calico should be the only backend using vif_type=tap 13:40:52 < sean-k-mooney> but im not sure if we woudl need a config option in the workarounds section for this or not Potential patch =============== diff --git a/nova/virt/libvirt/config.py b/nova/virt/libvirt/config.py index 47e92e3..5af3ce4 100644 --- a/nova/virt/libvirt/config.py +++ b/nova/virt/libvirt/config.py @@ -1749,6 +1749,7 @@ self.device_addr = None self.mtu = None self.alias = None + self.managed = 'no' def __eq__(self, other): if not isinstance(other, LibvirtConfigGuestInterface): @@ -1851,7 +1852,7 @@ dev.append(vlan_elem) if self.target_dev is not None: - dev.append(etree.Element("target", dev=self.target_dev)) + dev.append(etree.Element("target", dev=self.target_dev, managed=self.managed)) if self.vporttype is not None: vport = etree.Element("virtualport", type=self.vporttype)
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2033681 Title: Calico still uses vif type tap and it causes failures with libvirt 9.5.0 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2033681/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
