[Yahoo-eng-team] [Bug 1977485] [NEW] Neutron deletes port in use and nova errors out when cleaning the VM XML
Public bug reported: We recently upgraded to OpenStack XENA on a Kolla deployment and we hit this issue Neutron is able to delete a port in use and Nova errors out when trying to clean up the VM XML (we are talking about Windows VMs on KVM hypervisor). We need to move the ports from some VMs to a different subnet. We tried the following procedure: remove the port of a VM create a new port on the new subnet attach the new port to the VM Result: We have a VM with no network connectivity. The port gets deleted from OpenStack, and from OVS as well, but in the VM XML we see this: The bridge part represents the old interface (the interface of the old deleted port) and the ethernet part is the new port. We also tried to use: virsh detach-interface in order to remove the stale interface from the XML, the command said it was completed successfully but the interface is still there. We noticed that rebooting the VM cleans the XML file and the connectivity is back (this is not the desired solution) In the logs we see: When we delete the old port: 2022-05-31 12:13:29.935 7 INFO nova.compute.manager [req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Neutron deleted interface fb15ad83-bf28-455d-a1b1-14158203b4bf; detaching it from the instance and deleting it from the info cache 2022-05-31 12:13:30.076 7 WARNING nova.virt.libvirt.driver [req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Detaching interface 00:16:3c:7b:2c:1c failed because the device is no longer found on the guest.: nova.exception.DeviceNotFound: Device 'tapfb15ad83-bf' not found. 2022-05-31 12:13:30.740 7 INFO os_vif [req-1bf9e960-fc99-453f-8bf9-cd6a76c12feb 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] Successfully unplugged vif VIFOpenVSwitch(active=True,address=00:16:3c:7b:2c:1c,bridge_name='br-int',has_traffic_filtering=True,id=fb15ad83-bf28-455d-a1b1-14158203b4bf,network=Network(b03631f6-6fa7-4ff3-97e6-0a3bd077fac3),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=True,vif_name='tapfb15ad83-bf') When we attach the new port: 2022-05-31 12:20:16.427 7 WARNING nova.compute.manager [req-f42820d6-1c70-428a-9c2d-305737838bfc 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with vm_state active and task_state None. 2022-05-31 12:20:19.188 7 WARNING nova.compute.manager [req-305324c7-c25b-44a7-96cd-a8cc84284727 9879764509c84ca58d054fc3b9575df6 24783cb241264363ad1b8808ba21c131 - default default] [instance: b6c60a66-e571-4d50-984a-101dcb29f6aa] Received unexpected event network-vif-plugged-ba91ea48-3676-4934-87ba-1ad4cf80b1bc for instance with vm_state active and task_state None. We found that the following workaround works: we use virsh detach-interface while the old port of the VM exists (before we delete it) then we delete the old port after that, we attach the new port This works as expected and the VM has network connectivity. ** Affects: neutron Importance: Undecided Status: New ** Tags: nova ** Summary changed: - Neutron deletes port and nova errors out + Neutron deletes port and nova errors out when cleaning the VM XML ** Summary changed: - Neutron deletes port and nova errors out when cleaning the VM XML + Neutron deletes port in use and nova errors out when cleaning the VM XML -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1977485 Title: Neutron deletes port in use and nova errors out when cleaning the VM XML Status in neutron: New Bug description: We recently upgraded to OpenStack XENA on a Kolla deployment and we hit this issue Neutron is able to delete a port in use and Nova errors out when trying to clean up the VM XML (we are talking about Windows VMs on KVM hypervisor). We need to move the ports from some VMs to a different subnet. We tried the following procedure: remove the port of a VM create a new port on the new subnet attach the new port to the VM Result: We have a VM with no network connectivity. The port gets deleted from OpenStack, and from OVS as well, but in the VM XML we see this: The bridge part represents the old interface (the interface of the
[Yahoo-eng-team] [Bug 1937319] Re: namespace collision on url kernel arg results in downloading iso multiple times
This bug was fixed in the package casper - 1.472 --- casper (1.472) kinetic; urgency=medium * Support iso-url= on the kernel command line as a synonym for url=, to help avoid the conflict with cloud-init over that argument. (LP: #1937319) -- Michael Hudson-Doyle Fri, 03 Jun 2022 11:25:33 +1200 ** Changed in: casper (Ubuntu) Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1937319 Title: namespace collision on url kernel arg results in downloading iso multiple times Status in cloud-init: Triaged Status in casper package in Ubuntu: Fix Released Bug description: As listed at https://discourse.ubuntu.com/t/netbooting-the-live- server-installer/14510/135 When attempting a netboot, we can end up in a situation where cloud- init downloads a full iso, only to decide that the first few bytes aren't the expected header and to ignore it. I suggest an enhancement where a small amount of data is downloaded at first, we check this data for the required #cloud-config, and then use that info to decide to continue to download the file or not. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1937319/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1977470] [NEW] [networking-bgpvpn] zed unit test failures
Public bug reported: tox -e py310 results in multiple "has no attribute 'register_common_config_options'" errors: = [0/30] Failures during discovery = --- import errors --- Failed to import test module: networking_bgpvpn.tests.unit.db.test_db Traceback (most recent call last): File "/usr/lib/python3.10/unittest/loader.py", line 436, in _find_test_path module = self._get_module_from_name(name) File "/usr/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name __import__(name) File "/home/corey/pkg/zed/upstream/networking-bgpvpn/networking_bgpvpn/tests/unit/db/test_db.py", line 20, in from networking_bgpvpn.neutron.db.bgpvpn_db import BGPVPNPluginDb File "/home/corey/pkg/zed/upstream/networking-bgpvpn/networking_bgpvpn/neutron/db/bgpvpn_db.py", line 34, in from networking_bgpvpn.neutron.extensions import bgpvpn as bgpvpn_ext File "/home/corey/pkg/zed/upstream/networking-bgpvpn/networking_bgpvpn/neutron/extensions/bgpvpn.py", line 37, in common_config.register_common_config_options() AttributeError: module 'neutron.common.config' has no attribute 'register_common_config_options' ** Affects: neutron Importance: Undecided Status: New ** Description changed: - tox -e py310 results in: + tox -e py310 results in multiple "has no attribute + 'register_common_config_options'" errors: = [0/30] - Failures during discovery - = - --- import errors --- - Failed to import test module: networking_bgpvpn.tests.unit.db.test_db - Traceback (most recent call last): - File "/usr/lib/python3.10/unittest/loader.py", line 436, in _find_test_path - module = self._get_module_from_name(name) - File "/usr/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name - __import__(name) - File "/home/corey/pkg/zed/upstream/networking-bgpvpn/networking_bgpvpn/tests/unit/db/test_db.py", line 20, in - from networking_bgpvpn.neutron.db.bgpvpn_db import BGPVPNPluginDb - File "/home/corey/pkg/zed/upstream/networking-bgpvpn/networking_bgpvpn/neutron/db/bgpvpn_db.py", line 34, in - from networking_bgpvpn.neutron.extensions import bgpvpn as bgpvpn_ext - File "/home/corey/pkg/zed/upstream/networking-bgpvpn/networking_bgpvpn/neutron/extensions/bgpvpn.py", line 37, in - common_config.register_common_config_options() - AttributeError: module 'neutron.common.config' has no attribute 'register_common_config_options'
[Yahoo-eng-team] [Bug 1977468] [NEW] Nova sanitize_hostname problematic with fqdn display_name
Public bug reported:
Current implementation of nova/utils.py in function sanitize_hostname
does a simple replace for dots in hostname.
This causes an issue with nova/compute/api.py where _populate_instance_names
sets hostname from display_name. We have multiple cases where we want display
name to reflect FQDN of the instance.
In this case server1.example.com becomes hostname server1-example-com. This
tends to create a cascading array of problems, when the hostname is not the
actual hostname, but variation of FQDN. Cloud-init does pick up the generated
name and the issues can carry to /etc/hosts (depending on configuration).
It's desirable to have FQDN as display name, as there may be instances
that have the same hostname, but different domain listed in various
views that list instances.
Tools like Ansible's openstack.cloud.server.module do not have the
ability to specify display_name and hostname individually.
It would be preferable to have an option to select the way the name is
sanitized. Aka. cut down everything after first dot (possibly with more
logic to check for valid FQDN) or to have current way of just replacing
dots '.' with dashes '-'. I don't see either as specifically correct way
of doing things, trying to deduce a hostname from display name is an
opinionated thing.
I did a dirty fix for my specific problem by splitting the hostname from
first dot and picking up the first part as hostname:
--- /tmp/utils.py.orig 2022-06-02 22:02:48.152040276 +0300
+++ /tmp/utils.py 2022-06-02 22:22:00.319168645 +0300
@@ -365,6 +365,8 @@
# Remove characters outside the Unicode range U+-U+00FF
hostname = hostname.encode('latin-1', 'ignore').decode('latin-1')
+if hostname.find('.') >= 0:
+hostname = hostname.split('.')[0]
hostname = truncate_hostname(hostname)
hostname = re.sub(r'[ _\.]', '-', hostname)
hostname = re.sub(r'[^\w.-]+', '', hostname)
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1977468
Title:
Nova sanitize_hostname problematic with fqdn display_name
Status in OpenStack Compute (nova):
New
Bug description:
Current implementation of nova/utils.py in function sanitize_hostname
does a simple replace for dots in hostname.
This causes an issue with nova/compute/api.py where _populate_instance_names
sets hostname from display_name. We have multiple cases where we want display
name to reflect FQDN of the instance.
In this case server1.example.com becomes hostname server1-example-com. This
tends to create a cascading array of problems, when the hostname is not the
actual hostname, but variation of FQDN. Cloud-init does pick up the generated
name and the issues can carry to /etc/hosts (depending on configuration).
It's desirable to have FQDN as display name, as there may be instances
that have the same hostname, but different domain listed in various
views that list instances.
Tools like Ansible's openstack.cloud.server.module do not have the
ability to specify display_name and hostname individually.
It would be preferable to have an option to select the way the name is
sanitized. Aka. cut down everything after first dot (possibly with
more logic to check for valid FQDN) or to have current way of just
replacing dots '.' with dashes '-'. I don't see either as specifically
correct way of doing things, trying to deduce a hostname from display
name is an opinionated thing.
I did a dirty fix for my specific problem by splitting the hostname
from first dot and picking up the first part as hostname:
--- /tmp/utils.py.orig 2022-06-02 22:02:48.152040276 +0300
+++ /tmp/utils.py 2022-06-02 22:22:00.319168645 +0300
@@ -365,6 +365,8 @@
# Remove characters outside the Unicode range U+-U+00FF
hostname = hostname.encode('latin-1', 'ignore').decode('latin-1')
+if hostname.find('.') >= 0:
+hostname = hostname.split('.')[0]
hostname = truncate_hostname(hostname)
hostname = re.sub(r'[ _\.]', '-', hostname)
hostname = re.sub(r'[^\w.-]+', '', hostname)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1977468/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1976526] Re: IPv6-enabled EC2 subnets always have dhcp enabled
** Bug watch added: Red Hat Bugzilla #2092459 https://bugzilla.redhat.com/show_bug.cgi?id=2092459 ** Also affects: cloud-init (Fedora) via https://bugzilla.redhat.com/show_bug.cgi?id=2092459 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1976526 Title: IPv6-enabled EC2 subnets always have dhcp enabled Status in cloud-init: New Status in cloud-init package in Fedora: Unknown Bug description: A Fedora user opened an issue[0] about an issue with cloud-init on EC2. When they launch an instance on a IPv6-enabled subnet, cloud-init sets the `ipv6.method` to `dhcp` in NetworkManager. However, AWS uses router advertisements to let instances know about the nearest router. Using the `ipv6.method: dhcp` setting prevents router advertisements from working. The instance ends up with an IPv6 address assigned, but it cannot route traffic. A workaround is to run NetworkManager commands to fix the issue: ``` nmcli con edit CONNECTION_UUID nmcli> set ipv6.method auto nmcli> save nmcli> activate ``` The instance immediately routes IPv6 traffic after making that change. The dhcp setting appears to be applied automatically for EC2 subnets with IPv6 addresses assigned[1]. I would expect `ipv6.method` to be `auto` in these situations. Thank you! [0] https://pagure.io/cloud-sig/issue/382 [1] https://github.com/canonical/cloud-init/blob/53a995e2f852d043d51ad25c1b9afbbe1edafd57/cloudinit/sources/DataSourceEc2.py#L861-L863 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1976526/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1973726] Re: [DB] "SubnetPool" exact match queries to non-indexed columns
Reviewed: https://review.opendev.org/c/openstack/neutron/+/844297 Committed: https://opendev.org/openstack/neutron/commit/5957e90575eddc864b59ea2862e4c2bcd7a1bf7a Submitter: "Zuul (22348)" Branch:master commit 5957e90575eddc864b59ea2862e4c2bcd7a1bf7a Author: yatinkarel Date: Wed Jun 1 19:34:25 2022 +0530 Create an index for subnetpools.address_scope_id The method "get_network_address_scope" filters "Subnetpool" with "address_scope_id" using an exact match. Making the column indexed will improve the query performance. Closes-Bug: #1973726 Change-Id: Ib3f8e18ba28b277d5fa02dd386ca80a5a113c247 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1973726 Title: [DB] "SubnetPool" exact match queries to non-indexed columns Status in neutron: Fix Released Bug description: In ``neutron.objects.address_scope.AddressScope.get_network_address_scope``, the OVO executes a query using exact match filters to "address_scope_id". This column is not indexed and the query could under perform. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1973726/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1976360] Re: [sqlalchemy-20] Missing DB context in networking-sfc flow classifier
This should have opened against networking-sfc, it's fixed with https://review.opendev.org/c/openstack/networking-sfc/+/844251. Will update project. ** Project changed: neutron => networking-sfc ** Changed in: networking-sfc Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1976360 Title: [sqlalchemy-20] Missing DB context in networking-sfc flow classifier Status in networking-sfc: Fix Committed Bug description: Logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ab7/842135/1/gate/neutron- tempest-plugin-sfc/ab703be/controller/logs/screen-q-svc.txt Snippet: https://paste.opendev.org/show/bYc7TKCf9X5f4yFhYu7O/ To manage notifications about this bug go to: https://bugs.launchpad.net/networking-sfc/+bug/1976360/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
