Public bug reported:
We recently upgraded to OpenStack XENA on a Kolla deployment and we hit
this issue
Neutron is able to delete a port in use and Nova errors out when trying
to clean up the VM XML (we are talking about Windows VMs on KVM
hypervisor).
We need to move the ports from some VMs to a different subnet. We tried the
following procedure:
remove the port of a VM
create a new port on the new subnet
attach the new port to the VM
Result:
We have a VM with no network connectivity.
The port gets deleted from OpenStack, and from OVS as well, but in the VM XML
we see this:
<interface type='bridge'>
<mac address='00:16:3c:7b:2c:1c'/>
<source bridge='br-int'/>
<virtualport type='openvswitch'>
<parameters interfaceid='fb15ad83-bf28-455d-a1b1-14158203b4bf'/>
</virtualport>
<target dev='tapfb15ad83-bf'/>
<model type='virtio'/>
<mtu size='1500'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<interface type='ethernet'>
<mac address='00:16:3c:7b:2c:1c'/>
<target dev='tapba91ea48-36'/>
<model type='virtio'/>
<mtu size='1500'/>
<alias name='net1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
</interface>
The bridge part represents the old interface (the interface of the old deleted
port) and the ethernet part is the new port.
We also tried to use:
virsh detach-interface in order to remove the stale interface from the XML, the
command said it was completed successfully but the interface is still there.
We noticed that rebooting the VM cleans the XML file and the connectivity is
back (this is not the desired solution)
In the logs we see:
When we delete the old port:
2022-05-31 12:13:29.935 7 INFO nova.compute.manager
[req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Neutron deleted interface
fb15ad83-bf28-455d-a1b1-14158203b4bf; detaching it from the instance and
deleting it from the info cache
2022-05-31 12:13:30.076 7 WARNING nova.virt.libvirt.driver
[req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Detaching interface 00:16:3c:7b:2c:1c
failed because the device is no longer found on the guest.:
nova.exception.DeviceNotFound: Device 'tapfb15ad83-bf' not found.
2022-05-31 12:13:30.740 7 INFO os_vif [req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb
9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default
default] Successfully unplugged vif
VIFOpenVSwitch(active=True,address=00:16:3c:7b:2c:1c,bridge_name='br-int',has_traffic_filtering=True,id=fb15ad83-bf28-455d-a1b1-14158203b4bf,network=Network(b03631f6-6fa7-4ff3-97e6-0a3bd077fac3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tapfb15ad83-bf')
When we attach the new port:
2022-05-31 12:20:16.427 7 WARNING nova.compute.manager
[req-f42820d6-1c70-428a-9c2d-305737838bfc 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event
network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with
vm_state active and task_state None.
2022-05-31 12:20:19.188 7 WARNING nova.compute.manager
[req-305324c7-c25b-44a7-96cd-a8cc84284727 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event
network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with
vm_state active and task_state None.
We found that the following workaround works:
we use virsh detach-interface while the old port of the VM exists (before we
delete it)
then we delete the old port
after that, we attach the new port
This works as expected and the VM has network connectivity.
** Affects: neutron
Importance: Undecided
Status: New
** Tags: nova
** Summary changed:
- Neutron deletes port and nova errors out
+ Neutron deletes port and nova errors out when cleaning the VM XML
** Summary changed:
- Neutron deletes port and nova errors out when cleaning the VM XML
+ Neutron deletes port in use and nova errors out when cleaning the VM XML
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1977485
Title:
Neutron deletes port in use and nova errors out when cleaning the VM
XML
Status in neutron:
New
Bug description:
We recently upgraded to OpenStack XENA on a Kolla deployment and we
hit this issue
Neutron is able to delete a port in use and Nova errors out when
trying to clean up the VM XML (we are talking about Windows VMs on KVM
hypervisor).
We need to move the ports from some VMs to a different subnet. We tried the
following procedure:
remove the port of a VM
create a new port on the new subnet
attach the new port to the VM
Result:
We have a VM with no network connectivity.
The port gets deleted from OpenStack, and from OVS as well, but in the VM XML
we see this:
<interface type='bridge'>
<mac address='00:16:3c:7b:2c:1c'/>
<source bridge='br-int'/>
<virtualport type='openvswitch'>
<parameters interfaceid='fb15ad83-bf28-455d-a1b1-14158203b4bf'/>
</virtualport>
<target dev='tapfb15ad83-bf'/>
<model type='virtio'/>
<mtu size='1500'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<interface type='ethernet'>
<mac address='00:16:3c:7b:2c:1c'/>
<target dev='tapba91ea48-36'/>
<model type='virtio'/>
<mtu size='1500'/>
<alias name='net1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
</interface>
The bridge part represents the old interface (the interface of the old
deleted port) and the ethernet part is the new port.
We also tried to use:
virsh detach-interface in order to remove the stale interface from the XML,
the command said it was completed successfully but the interface is still there.
We noticed that rebooting the VM cleans the XML file and the connectivity is
back (this is not the desired solution)
In the logs we see:
When we delete the old port:
2022-05-31 12:13:29.935 7 INFO nova.compute.manager
[req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Neutron deleted interface
fb15ad83-bf28-455d-a1b1-14158203b4bf; detaching it from the instance and
deleting it from the info cache
2022-05-31 12:13:30.076 7 WARNING nova.virt.libvirt.driver
[req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Detaching interface 00:16:3c:7b:2c:1c
failed because the device is no longer found on the guest.:
nova.exception.DeviceNotFound: Device 'tapfb15ad83-bf' not found.
2022-05-31 12:13:30.740 7 INFO os_vif
[req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] Successfully unplugged vif
VIFOpenVSwitch(active=True,address=00:16:3c:7b:2c:1c,bridge_name='br-int',has_traffic_filtering=True,id=fb15ad83-bf28-455d-a1b1-14158203b4bf,network=Network(b03631f6-6fa7-4ff3-97e6-0a3bd077fac3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tapfb15ad83-bf')
When we attach the new port:
2022-05-31 12:20:16.427 7 WARNING nova.compute.manager
[req-f42820d6-1c70-428a-9c2d-305737838bfc 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event
network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with
vm_state active and task_state None.
2022-05-31 12:20:19.188 7 WARNING nova.compute.manager
[req-305324c7-c25b-44a7-96cd-a8cc84284727 9879764509c84ca58d054fc3b9575df6
24783cb241264363ad1b8808ba21c131 - default default] [instance:
b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event
network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with
vm_state active and task_state None.
We found that the following workaround works:
we use virsh detach-interface while the old port of the VM exists (before we
delete it)
then we delete the old port
after that, we attach the new port
This works as expected and the VM has network connectivity.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1977485/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
