[Yahoo-eng-team] [Bug 1881424] Re: Neutron ovs agent fails on rpc_loop iteration:1
Looks like this was fixed by Rodolfo's singleton patch
** No longer affects: ovsdbapp
** Changed in: neutron
Status: New => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1881424
Title:
Neutron ovs agent fails on rpc_loop iteration:1
Status in kolla-ansible:
Invalid
Status in kolla-ansible victoria series:
Invalid
Status in neutron:
Fix Released
Bug description:
Hi Neutrinos!
This is from Kolla-Ansible CI, it started happening in Victoria on May 28.
It affects all distros deb-ubu-centos and makes the jobs fail (ovs agent is
dead).
It does *not* affect OVN though.
br-ex exists before iteration:0 and acts fine in iteration:0 but not
in iteration:1.
The only quasi-relevant change in neutron seems to be
https://review.opendev.org/721554 but still it should only affect DVR
which we are not running. Odd.
Full logs with config and debug are attached.
The relevant logs look like this (bottom of the file):
2020-05-30 15:27:59.003 7 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Agent rpc_loop -
iteration:1 started
2020-05-30 15:27:59.006 7 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] ofctl request
version=0x4,msg_type=0x12,msg_len=0x38,xid=0x15f04540,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1)
result
[OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=371283074374527098,duration_nsec=71800,duration_sec=5,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)]
_send_msg
/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py:113
2020-05-30 15:27:59.008 7 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Physical bridge br-ex was
just re-created.
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.009 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index name
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.010 7 DEBUG ovsdbapp.backend.ovs_idl
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Created index target
autocreate_indices
/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:99
2020-05-30 15:27:59.011 7 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Mapping physical network
physnet1 to bridge br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command
[req-680d0d82-c47f-4b45-86cc-53520f537f29 - - - - -] Error executing command:
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with
name=br-ex
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command Traceback
(most recent call last):
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py",
line 38, in execute
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command
self.run_idl(None)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py",
line 214, in run_idl
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command record =
self.api.lookup(self.table, self.record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py",
line 171, in lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command return
self._lookup(table, record)
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py",
line 218, in _lookup
2020-05-30 15:27:59.012 7 ERROR ovsdbapp.backend.ovs_idl.command row =
idlutils.row_by_value(self, rl.table, rl.column, record)
2020-05-30 15:27:59.012 7 ERROR
[Yahoo-eng-team] [Bug 1926399] Re: UT failing with sqlalchemy 1.4
Fixes for the bugs mentioned in comment #9 have merged. ** Changed in: cinder Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1926399 Title: UT failing with sqlalchemy 1.4 Status in Cinder: Fix Released Status in masakari: In Progress Status in neutron: Fix Released Status in OpenStack Compute (nova): In Progress Status in oslo.db: Fix Released Bug description: See job cross-neutron-py36 in test patch https://review.opendev.org/c/openstack/requirements/+/788339/ https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ac7/788339/1/check /cross-neutron-py36/ac77335/testr_results.html To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1926399/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1926426] Re: Nova is not compatible with sqlalchemy 1.4
Reviewed: https://review.opendev.org/c/openstack/nova/+/788471 Committed: https://opendev.org/openstack/nova/commit/39a617752f71cb8fe7188baee08b1c2ba48ad066 Submitter: "Zuul (22348)" Branch:master commit 39a617752f71cb8fe7188baee08b1c2ba48ad066 Author: Balazs Gibizer Date: Mon May 3 10:44:52 2021 +0200 Adapt to SQLAlchemy 1.4 This patch makes the necessary change to adapt to the SQLAlchemy 1.4 release in a way that is still compatible with the currently pinned 1.3 versions. This is related to the overall effort to bump SQLAlchemy to 1.4 https://review.opendev.org/c/openstack/requirements/+/788339 Co-Authored-By: Mike Bayer Closes-Bug: #1926426 Change-Id: I8a0ab3b91b4203ab603caac02ee5132be7890e9a ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1926426 Title: Nova is not compatible with sqlalchemy 1.4 Status in OpenStack Compute (nova): Fix Released Bug description: As testing in [1] shows we have issues with the new sqlalchemy release. Two types of error are visible: 1) sqlalchemy.exc.InvalidRequestError: Entity namespace for "count(instances.id)" has no property "deleted" during: /home/zuul/src/opendev.org/openstack/nova/nova/objects/instance.py", line 1525, in _get_counts_in_db and 2) url.database = ident AttributeError: can't set attribute during: /home/zuul/src/opendev.org/openstack/nova/nova/tests/unit/db/test_migration_utils.py", line 56, in setUp Example result: https://zuul.opendev.org/t/openstack/build/e18e86f0c074450d9a4f823937bc1c16/logs [1] https://review.opendev.org/c/openstack/requirements/+/788339/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1926426/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1927020] Re: cloudconfig not writing maas data source
I'm marking the cloud-init task invalid. After getting config and logs we confirmed that the cloud-init package in the Debian image does not contain a recent handle_maas_preseed() that Ubuntu uses and curtin/MAAS rely upon to install the MAAS datasource correctly. ** Changed in: cloud-init Status: Incomplete => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1927020 Title: cloudconfig not writing maas data source Status in cloud-init: Invalid Status in curtin: Invalid Bug description: further background https://discourse.maas.io/t/debian10-fails-on- final-reboot/4486 I'm deploying a debian 10 buster image using MAAS. No errors are reported in MAAS or curtin install but on the final boot cloud-init reports `Failed to load metadata and userdata`. checking the config for the machine in maas shows all the cloudconfig: to connect to maas but the files with oauth credentials etc dont seem to have been copied to the target. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1927020/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1927020] Re: cloudconfig not writing maas data source
I'm marking the curtin task invalid; after getting config and logs we confirmed that the cloud-init package in the Debian image does not contain a recent handle_maas_preseed() that Ubuntu uses and curtin/MAAS rely upon to install the MAAS datasource correctly. ** Changed in: curtin Status: Incomplete => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1927020 Title: cloudconfig not writing maas data source Status in cloud-init: Invalid Status in curtin: Invalid Bug description: further background https://discourse.maas.io/t/debian10-fails-on- final-reboot/4486 I'm deploying a debian 10 buster image using MAAS. No errors are reported in MAAS or curtin install but on the final boot cloud-init reports `Failed to load metadata and userdata`. checking the config for the machine in maas shows all the cloudconfig: to connect to maas but the files with oauth credentials etc dont seem to have been copied to the target. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1927020/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1894843] Re: [dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host
This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7~cloud0 --- neutron (2:12.1.1-0ubuntu7~cloud0) xenial-queens; urgency=medium . * New update for the Ubuntu Cloud Archive. . neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium . * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch (LP: #1849098). . neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium . * Do not initialize snat-ns twice (LP: #1850779) - d/p/0001-Do-not-initialize-snat-ns-twice.patch . neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium . * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843) - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch ** Changed in: cloud-archive/queens Status: Fix Committed => Fix Released ** Changed in: cloud-archive Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1894843 Title: [dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host Status in Ubuntu Cloud Archive: Fix Released Status in Ubuntu Cloud Archive queens series: Fix Released Status in Ubuntu Cloud Archive rocky series: Fix Released Status in Ubuntu Cloud Archive stein series: Fix Released Status in Ubuntu Cloud Archive train series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Released Status in Ubuntu Cloud Archive victoria series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Bionic: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Groovy: Fix Released Status in neutron source package in Hirsute: Fix Released Bug description: [Impact] When neutron schedules snat namespaces it sometimes deletes the rfp interface from qrouter namespaces which breaks external network (fip) connectivity. The fix prevents this from happening. [Test Case] * deploy Openstack (Ussuri or above) with dvr_snat enabled in compute hosts. * ensure min. 2 compute hosts * create one ext network and one private network * add private subnet to router and ext as gateway * check which compute has the snat ns (ip netns| grep snat) * create a vm on each compute host * check that qrouter ns on both computes has rfp interface * ip netns| grep qrouter; ip netns exec ip a s| grep rfp * disable and re-enable router * openstack router set --disable ; openstack router set --enable * check again * ip netns| grep qrouter; ip netns exec ip a s| grep rfp [Where problems could occur] no regression is expected, but if one occurs it would likely result in breakage with external network connectivity - Hello, In the case of dvr_snat l3 agents are deployed on hypervisors there can be race condition. The agent creates snat namespaces on each scheduled host and removes them at second step. At this second step agent removes the rfp interface from qrouter even when there is VM with floating IP on the host. When VM is deployed at the time of second step we can lost external access to VMs floating IP. The issue can be reproduced by hand: 1. Create tenant network and router with external gateway 2. Create VM with floating ip 3. Ensure that VM on the hypervisor without snat-* namespace 4. Set the router to disabled state (openstack router set --disable ) 5. Set the router to enabled state (openstack router set --enabled ) 6. The external access to VMs FIP have lost because L3 agent creates the qrouter namespace without rfp interface. Environment: 1. Neutron with ML2 OVS plugin. 2. L3 agents in dvr_snat mode on each hypervisor 3. openstack-neutron-common-15.1.1-0.2020061910.7d97420.el8ost.noarch To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1894843/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7~cloud0
---
neutron (2:12.1.1-0ubuntu7~cloud0) xenial-queens; urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium
.
* Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
- d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
(LP: #1849098).
.
neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium
.
* Do not initialize snat-ns twice (LP: #1850779)
- d/p/0001-Do-not-initialize-snat-ns-twice.patch
.
neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium
.
* Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
- d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch
** Changed in: cloud-archive/queens
Status: New => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1849098
Title:
ovs agent is stuck with OVSFWTagNotFound when dealing with unbound
port
Status in Ubuntu Cloud Archive:
New
Status in Ubuntu Cloud Archive queens series:
Fix Released
Status in neutron:
Fix Released
Status in neutron package in Ubuntu:
Fix Released
Status in neutron source package in Bionic:
Fix Released
Bug description:
[Impact]
somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.
[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation
failed"
[Where problems could occur]
while no regressions are expected, if they do occur it would be when getting
or creating vif port
[Others]
Original description.
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device
ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding
failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc
[req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for
devices {'ef34215f-e099-4fd0-935f-c9a42951d166',
'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca',
'2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd',
'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5',
'9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF
ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound:
Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py",
line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port =
self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError:
'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling
of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py",
line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR
[Yahoo-eng-team] [Bug 1928345] [NEW] "neutron_tempest_plugin.api.test_trunk*" tests failing
Public bug reported: "neutron_tempest_plugin.api.test_trunk.TrunkTestMtusJSON" and "neutron_tempest_plugin.api.test_trunk_negative.TrunkTestMtusJSON" are currently failing in the CI. This problem seems to be related to the recent change in devstack [1] that changes the default ML2 driver to OVN. [1]https://review.opendev.org/c/openstack/devstack/+/735097/ ** Affects: neutron Importance: Critical Status: New ** Changed in: neutron Importance: Undecided => Critical -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1928345 Title: "neutron_tempest_plugin.api.test_trunk*" tests failing Status in neutron: New Bug description: "neutron_tempest_plugin.api.test_trunk.TrunkTestMtusJSON" and "neutron_tempest_plugin.api.test_trunk_negative.TrunkTestMtusJSON" are currently failing in the CI. This problem seems to be related to the recent change in devstack [1] that changes the default ML2 driver to OVN. [1]https://review.opendev.org/c/openstack/devstack/+/735097/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1928345/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1798224] Re: DeprecationWarning: The behavior of .best_match for the Accept classes is currently being maintained for backward compatibility, but the method will be deprecated in
** Also affects: cinder
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1798224
Title:
DeprecationWarning: The behavior of .best_match for the Accept classes
is currently being maintained for backward compatibility, but the
method will be deprecated in the future
Status in Cinder:
In Progress
Status in OpenStack Compute (nova):
Fix Released
Bug description:
When executing 'tox -e py35', the following deprecation warning is shown.
It should be fixed.
2018-10-16 03:36:49.117553 | ubuntu-xenial | {5}
nova.tests.unit.api.openstack.compute.test_disk_config.DiskConfigTestCaseV21.test_update_server_override_auto
[0.544275s] ... ok
2018-10-16 03:36:49.117626 | ubuntu-xenial |
2018-10-16 03:36:49.117666 | ubuntu-xenial | Captured stderr:
2018-10-16 03:36:49.117703 | ubuntu-xenial |
(snipped...)
2018-10-16 03:36:49.118228 | ubuntu-xenial |
b'/home/zuul/src/git.openstack.org/openstack/nova/.tox/py35/lib/python3.5/site-packages/webob/acceptparse.py:1379:
DeprecationWarning: The behavior of .best_match for the Accept classes is
currently being maintained for backward compatibility, but the method will be
deprecated in the future, as its behavior is not specified in (and currently
does not conform to) RFC 7231.'
2018-10-16 03:36:49.118288 | ubuntu-xenial | b' DeprecationWarning,'
2018-10-16 03:36:49.118319 | ubuntu-xenial | b''
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1798224/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1928330] [NEW] [OVN] Unable to ping router from test IPV6 subnet
Public bug reported:
Steps to reproduce:
1) Create test network, IPV6 subnet in arbitrary mode, eg. SLAAC;
2) Create test router and add an interface in the test subnet;
3) Create test instance and try to ping the router - it fails.
Sample OVN trace:
# ovn-trace piotr-geneve-ipv6-stateful 'inport == "9b6d8c19-ca57-4f0d-
8af0-ee29642c6fb8" && eth.src == fa:16:3e:f5:aa:2a && eth.dst ==
fa:16:3e:6b:1b:79 && ip6.src == 2a01:49a0:112a::289 && ip6.dst ==
2a01:49a0:112a::'
0. ls_out_pre_lb (ovn-northd.c:5169): ip && outport == "7f82a5", priority 110,
uuid 4cb710e8
next;
1. ls_out_pre_acl (ovn-northd.c:5169): ip && outport == "7f82a5", priority
110, uuid 65a121ee
next;
4. ls_out_acl_hint (ovn-northd.c:5486): !ct.new && ct.est && !ct.rpl &&
ct_label.blocked == 0, priority 4, uuid 99195420
reg0[8] = 1;
reg0[10] = 1;
next;
10. ls_out_port_sec_l2 (ovn-northd.c:5118): outport == "7f82a5", priority 50,
uuid cc53fba5
output;
/* output to "7f82a5", type "patch" */
ingress(dp="test-router", inport="lrp-7f82a5")
--
0. lr_in_admission (ovn-northd.c:9556): eth.dst == fa:16:3e:6b:1b:79 && inport
== "lrp-7f82a5", priority 50, uuid ba5e360d
xreg0[0..47] = fa:16:3e:6b:1b:79;
next;
1. lr_in_lookup_neighbor (ovn-northd.c:9635): 1, priority 0, uuid 75a8a812
reg9[2] = 1;
next;
2. lr_in_learn_neighbor (ovn-northd.c:9644): reg9[2] == 1, priority 100, uuid
72706c4a
next;
3. lr_in_ip_input (ovn-northd.c:10884): ip6 && ip6.dst == 2a01:49a0:112a:: &&
!ip.later_frag, priority 70, uuid 13d497b1
icmp6 { eth.dst <-> eth.src; ip6.dst <-> ip6.src; ip.ttl = 255; icmp6.type
= 1; icmp6.code = 3; next; };
icmp6
-
eth.dst <-> eth.src;
ip6.dst <-> ip6.src;
ip.ttl = 255;
icmp6.type = 1;
icmp6.code = 3;
next;
10. lr_in_ip_routing (ovn-northd.c:8710): ip6.dst == 2a01:49a0:112a::/112,
priority 225, uuid 57af3cd5
10. lr_in_ip_routing (ovn-northd.c:8710): ip6.dst == 2a01:49a0:112a::/112,
priority 225, uuid 57af3cd5
ip.ttl--;
reg8[0..15] = 0;
xxreg0 = ip6.dst;
xxreg1 = 2a01:49a0:112a::;
eth.src = fa:16:3e:6b:1b:79;
outport = "lrp-7f82a5";
flags.loopback = 1;
next;
11. lr_in_ip_routing_ecmp (ovn-northd.c:9902): reg8[0..15] == 0, priority 150,
uuid 7e1242f0
next;
12. lr_in_policy (ovn-northd.c:10027): 1, priority 0, uuid dfb65200
reg8[0..15] = 0;
next;
13. lr_in_policy_ecmp (ovn-northd.c:10029): reg8[0..15] == 0, priority 150,
uuid a59e560f
next;
14. lr_in_arp_resolve (ovn-northd.c:10245): outport == "lrp-7f82a5" && xxreg0
== 2a01:49a0:112a::289, priority 100, uuid 1e99a10c
eth.dst = fa:16:3e:f5:aa:2a;
next;
18. lr_in_arp_request (ovn-northd.c:10652): 1, priority 0, uuid d05b7ef1
output;
egress(dp="test-router", inport="lrp-7f82a5", outport="lrp-7f82a5")
---
3. lr_out_delivery (ovn-northd.c:10700): outport == "lrp-7f82a5", priority
100, uuid 3d0b42d6
output;
/* output to "lrp-7f82a5", type "patch" */
ingress(dp="piotr-geneve-ipv6-stateful", inport="7f82a5")
-
0. ls_in_port_sec_l2 (ovn-northd.c:5023): inport == "7f82a5", priority 50,
uuid e1f011ad
next;
5. ls_in_pre_acl (ovn-northd.c:5166): ip && inport == "7f82a5", priority 110,
uuid f4b6d9ab
next;
6. ls_in_pre_lb (ovn-northd.c:5166): ip && inport == "7f82a5", priority 110,
uuid c42a6170
next;
8. ls_in_acl_hint (ovn-northd.c:5486): !ct.new && ct.est && !ct.rpl &&
ct_label.blocked == 0, priority 4, uuid 5d064a89
reg0[8] = 1;
reg0[10] = 1;
next;
23. ls_in_l2_lkup (ovn-northd.c:7615): eth.dst == fa:16:3e:f5:aa:2a, priority
50, uuid 7faa449b
outport = "9b6d8c";
output;
egress(dp="piotr-geneve-ipv6-stateful", inport="7f82a5", outport="9b6d8c")
--
1. ls_out_pre_acl (ovn-northd.c:5226): ip, priority 100, uuid 32588e71
reg0[0] = 1;
next;
2. ls_out_pre_stateful (ovn-northd.c:5411): reg0[0] == 1, priority 100, uuid
58780681
ct_next;
ct_next(ct_state=est|trk /* default (use --ct to customize) */)
---
4. ls_out_acl_hint (ovn-northd.c:5486): !ct.new && ct.est && !ct.rpl &&
ct_label.blocked == 0, priority 4, uuid 99195420
reg0[8] = 1;
reg0[10] = 1;
next;
5. ls_out_acl (ovn-northd.c:5744): reg0[10] == 1 && (outport ==
@neutron_pg_drop && ip), priority 2001, uuid ec8b7afe
ct_commit { ct_label.blocked = 1; };
** Affects: neutron
[Yahoo-eng-team] [Bug 1923592] Re: [Routed networks] Router routes to other segment CIDRs should have a gateway IP
Hello Oskari: https://review.opendev.org/c/openstack/neutron/+/90250 was never meant to work with a RPN (was implemented time before) and RPN were designed to skip any L3 routing inside Neutron, externalizing this task to the underlying network infrastructure. I'm going to propose a patch that could make legacy routers to work with RPN. But the community will decide if this is valid or not. Regards. ** Changed in: neutron Status: Won't Fix => In Progress ** Changed in: neutron Importance: Medium => Low ** Changed in: neutron Importance: Low => Wishlist -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1923592 Title: [Routed networks] Router routes to other segment CIDRs should have a gateway IP Status in neutron: In Progress Bug description: The router namespace routes for other segment CIDRs, should have the "via" parameter to the gateway IP address. The environment I'm using has HA routers; I didn't test yet with DVR or legacy routers. For example, in my env [1]: - Network: public - Subnets: (s1) aebc3fc1-d984-40aa-8405-db51df03f60d: 10.46.54.0/26, gwip: 10.46.54.62 (s2) 55a66c34-3643-49ba-ab8c-326ee76e5f35: 10.46.54.64/26, gwip: 10.46.54.126 - Router namespace: [root@controller-0 ~]# ip netns exec $r ip r default via 10.46.54.62 dev qg-7847fd00-72 proto 112 default via 10.46.54.62 dev qg-7847fd00-72 proto static 10.1.0.0/16 dev qr-d224ca69-77 proto kernel scope link src 10.1.0.1 10.46.54.0/26 dev qg-7847fd00-72 proto kernel scope link src 10.46.54.34 10.46.54.64/26 dev qg-7847fd00-72 proto 112 scope link 10.46.54.64/26 dev qg-7847fd00-72 proto static scope link 169.254.0.0/24 dev ha-91b6c056-57 proto kernel scope link src 169.254.0.233 169.254.192.0/18 dev ha-91b6c056-57 proto kernel scope link src 169.254.193.151 This router is attached to the segment of the subnet aebc3fc1-d984-40aa-8405-db51df03f60d (s1). The route to 10.46.54.64/26 (the other subnet/segment this router is NOT attached to), should have the gateway IP address of the subnet that belongs to the segment the router is attached to; in this case, as commented, (s1). [1]http://paste.openstack.org/show/804429/ Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1945061 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1923592/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1815989] Re: OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky
** Also affects: nova/victoria
Importance: Undecided
Status: New
** Also affects: nova/train
Importance: Undecided
Status: New
** Also affects: nova/ussuri
Importance: Undecided
Status: New
** Also affects: nova/wallaby
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1815989
Title:
OVS drops RARP packets by QEMU upon live-migration causes up to 40s
ping pause in Rocky
Status in neutron:
In Progress
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) train series:
New
Status in OpenStack Compute (nova) ussuri series:
New
Status in OpenStack Compute (nova) victoria series:
New
Status in OpenStack Compute (nova) wallaby series:
New
Status in os-vif:
Invalid
Bug description:
This issue is well known, and there were previous attempts to fix it,
like this one
https://bugs.launchpad.net/neutron/+bug/1414559
This issue still exists in Rocky and gets worse. In Rocky, nova compute, nova
libvirt and neutron ovs agent all run inside containers.
So far the only simply fix I have is to increase the number of RARP
packets QEMU sends after live-migration from 5 to 10. To be complete,
the nova change (not merged) proposed in the above mentioned activity
does not work.
I am creating this ticket hoping to get an up-to-date (for Rockey and
onwards) expert advise on how to fix in nova-neutron.
For the record, below are the time stamps in my test between neutron ovs
agent "activating" the VM port and rarp packets seen by tcpdump on the compute.
10 RARP packets are sent by (recompiled) QEMU, 7 are seen by tcpdump, the 2nd
last packet barely made through.
openvswitch-agent.log:
2019-02-14 19:00:13.568 73453 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-26129036-b514-4fa0-a39f-a6b21de17bb9 - - - - -] Port
57d0c265-d971-404d-922d-963c8263e6eb updated. Details: {'profile': {},
'network_qos_policy_id': None, 'qos_policy_id': None,
'allowed_address_pairs': [], 'admin_state_up': True, 'network_id':
'1bf4b8e0-9299-485b-80b0-52e18e7b9b42', 'segmentation_id': 648,
'fixed_ips': [
{'subnet_id': 'b7c09e83-f16f-4d4e-a31a-e33a922c0bac', 'ip_address':
'10.0.1.4'}
], 'device_owner': u'compute:nova', 'physical_network': u'physnet0',
'mac_address': 'fa:16:3e:de:af:47', 'device':
u'57d0c265-d971-404d-922d-963c8263e6eb', 'port_security_enabled': True,
'port_id': '57d0c265-d971-404d-922d-963c8263e6eb', 'network_type': u'vlan',
'security_groups': [u'5f2175d7-c2c1-49fd-9d05-3a8de3846b9c']}
2019-02-14 19:00:13.568 73453 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-26129036-b514-4fa0-a39f-a6b21de17bb9 - - - - -] Assigning 4 as local vlan
for net-id=1bf4b8e0-9299-485b-80b0-52e18e7b9b42
tcpdump for rarp packets:
[root@overcloud-ovscompute-overcloud-0 nova]# tcpdump -i any rarp -nev
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size
262144 bytes
19:00:10.788220 B fa:16:3e:de:af:47 ethertype Reverse ARP (0x8035), length
62: Ethernet (len 6), IPv4 (len 4), Reverse Request who-is fa:16:3e:de:af:47
tell fa:16:3e:de:af:47, length 46
19:00:11.138216 B fa:16:3e:de:af:47 ethertype Reverse ARP (0x8035), length
62: Ethernet (len 6), IPv4 (len 4), Reverse Request who-is fa:16:3e:de:af:47
tell fa:16:3e:de:af:47, length 46
19:00:11.588216 B fa:16:3e:de:af:47 ethertype Reverse ARP (0x8035), length
62: Ethernet (len 6), IPv4 (len 4), Reverse Request who-is fa:16:3e:de:af:47
tell fa:16:3e:de:af:47, length 46
19:00:12.138217 B fa:16:3e:de:af:47 ethertype Reverse ARP (0x8035), length
62: Ethernet (len 6), IPv4 (len 4), Reverse Request who-is fa:16:3e:de:af:47
tell fa:16:3e:de:af:47, length 46
19:00:12.788216 B fa:16:3e:de:af:47 ethertype Reverse ARP (0x8035), length
62: Ethernet (len 6), IPv4 (len 4), Reverse Request who-is fa:16:3e:de:af:47
tell fa:16:3e:de:af:47, length 46
19:00:13.538216 B fa:16:3e:de:af:47 ethertype Reverse ARP (0x8035), length
62: Ethernet (len 6), IPv4 (len 4), Reverse Request who-is fa:16:3e:de:af:47
tell fa:16:3e:de:af:47, length 46
19:00:14.388320 B fa:16:3e:de:af:47 ethertype Reverse ARP (0x8035), length
62: Ethernet (len 6), IPv4 (len 4), Reverse Request who-is fa:16:3e:de:af:47
tell fa:16:3e:de:af:47, length 46
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1815989/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1849098] Re: ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port
This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7
---
neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium
* Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
- d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
(LP: #1849098).
neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium
* Do not initialize snat-ns twice (LP: #1850779)
- d/p/0001-Do-not-initialize-snat-ns-twice.patch
neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium
* Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
- d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch
-- Seyeong Kim Mon, 03 May 2021 17:15:28
+0900
** Changed in: neutron (Ubuntu Bionic)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1849098
Title:
ovs agent is stuck with OVSFWTagNotFound when dealing with unbound
port
Status in Ubuntu Cloud Archive:
New
Status in Ubuntu Cloud Archive queens series:
New
Status in neutron:
Fix Released
Status in neutron package in Ubuntu:
Fix Released
Status in neutron source package in Bionic:
Fix Released
Bug description:
[Impact]
somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.
[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation
failed"
[Where problems could occur]
while no regressions are expected, if they do occur it would be when getting
or creating vif port
[Others]
Original description.
neutron-openvswitch-agent meets unbound port:
2019-10-17 11:32:21.868 135 WARNING
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-
aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device
ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding
failed
Later when applying firewall rules:
2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc
[req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for
devices {'ef34215f-e099-4fd0-935f-c9a42951d166',
'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca',
'2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd',
'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5',
'9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF
ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound:
Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py",
line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port =
self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError:
'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling
of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most
recent call last):
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File
"/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py",
line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return
int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR
[Yahoo-eng-team] [Bug 1850779] Re: [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart
This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7 --- neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch (LP: #1849098). neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium * Do not initialize snat-ns twice (LP: #1850779) - d/p/0001-Do-not-initialize-snat-ns-twice.patch neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843) - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch -- Seyeong Kim Mon, 03 May 2021 17:15:28 +0900 ** Changed in: neutron (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1850779 Title: [L3] snat-ns will be initialized twice for DVR+HA routers during agent restart Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Bionic: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Groovy: Fix Released Status in neutron source package in Hirsute: Fix Released Bug description: If the DVR+HA router has external gateway, the snat-namespace will be initialized twice during agent restart. And that initialized function will run many [1][2] external resource processing actions which will definitely increase the starting time of agent. https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39 https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108 SRU: [Impact] Longer l3-agent initialization time during restarts due to creation of snat namespace and setting corresponding sysctl twice. With this fix, the initialization phase is triggered only once. [Test Case] * deploy Openstack on bionic queens (with neutron dvr l3 ha settings and debug mode on for neutron ) and create a router (If stsstack-bundles are used, here are the commands ./generate-bundle.sh -s bionic -n bionicqueens --dvr-snat-l3ha --create-model --run ./configure # Configure creates a router with external gateway attached ) * Restart neutron-l3-agent on one of the node systemctl restart neutron-l3-agent.service * Check /var/log/neutron/neutron-l3-agent.log and wait for the logs to be settled with all initialization steps During initialization steps, following sysctl's are configured [1] [2]. Verify if the debug logs show sysctl execution statements are displayed twice after restart for snat namespace. (If the fix is applied they should be displayed only once) grep -inr snat- /var/log/neutron/neutron-l3-agent.log | grep sysctl Example log: 2718:2021-04-14 05:17:20.114 10868 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'snat-f64dded1-ef73-47b4-bcee-bb25840e9a02', 'sysctl', '-w', 'net.ipv4.ip_forward=1'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:87 [Where problems could occur] no regression is expected, but if one occurs it would likely result in longer init time and/or failure to correctly init the snat-namespace [1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_snat_ns.py#L31-L39 [2] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L91-L108 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1850779/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1894843] Re: [dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host
This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7 --- neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch (LP: #1849098). neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium * Do not initialize snat-ns twice (LP: #1850779) - d/p/0001-Do-not-initialize-snat-ns-twice.patch neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843) - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch -- Seyeong Kim Mon, 03 May 2021 17:15:28 +0900 ** Changed in: neutron (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1894843 Title: [dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host Status in Ubuntu Cloud Archive: Fix Committed Status in Ubuntu Cloud Archive queens series: Fix Committed Status in Ubuntu Cloud Archive rocky series: Fix Released Status in Ubuntu Cloud Archive stein series: Fix Released Status in Ubuntu Cloud Archive train series: Fix Released Status in Ubuntu Cloud Archive ussuri series: Fix Released Status in Ubuntu Cloud Archive victoria series: Fix Released Status in neutron: Fix Released Status in neutron package in Ubuntu: Fix Released Status in neutron source package in Bionic: Fix Released Status in neutron source package in Focal: Fix Released Status in neutron source package in Groovy: Fix Released Status in neutron source package in Hirsute: Fix Released Bug description: [Impact] When neutron schedules snat namespaces it sometimes deletes the rfp interface from qrouter namespaces which breaks external network (fip) connectivity. The fix prevents this from happening. [Test Case] * deploy Openstack (Ussuri or above) with dvr_snat enabled in compute hosts. * ensure min. 2 compute hosts * create one ext network and one private network * add private subnet to router and ext as gateway * check which compute has the snat ns (ip netns| grep snat) * create a vm on each compute host * check that qrouter ns on both computes has rfp interface * ip netns| grep qrouter; ip netns exec ip a s| grep rfp * disable and re-enable router * openstack router set --disable ; openstack router set --enable * check again * ip netns| grep qrouter; ip netns exec ip a s| grep rfp [Where problems could occur] no regression is expected, but if one occurs it would likely result in breakage with external network connectivity - Hello, In the case of dvr_snat l3 agents are deployed on hypervisors there can be race condition. The agent creates snat namespaces on each scheduled host and removes them at second step. At this second step agent removes the rfp interface from qrouter even when there is VM with floating IP on the host. When VM is deployed at the time of second step we can lost external access to VMs floating IP. The issue can be reproduced by hand: 1. Create tenant network and router with external gateway 2. Create VM with floating ip 3. Ensure that VM on the hypervisor without snat-* namespace 4. Set the router to disabled state (openstack router set --disable ) 5. Set the router to enabled state (openstack router set --enabled ) 6. The external access to VMs FIP have lost because L3 agent creates the qrouter namespace without rfp interface. Environment: 1. Neutron with ML2 OVS plugin. 2. L3 agents in dvr_snat mode on each hypervisor 3. openstack-neutron-common-15.1.1-0.2020061910.7d97420.el8ost.noarch To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1894843/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1928299] Re: centos7 train vm live migration stops network on vm for some minutes
** Also affects: neutron/train Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1928299 Title: centos7 train vm live migration stops network on vm for some minutes Status in neutron: New Status in neutron train series: New Bug description: Hello, I have upgraded my centos 7 openstack installation from Stein to Train. On train I am facing an issue with live migration: when a vm is migrated from one kvm node to another, it stops to respond to ping requests from some minutes. I had the same issue on Stein and I resolved it with a workaround suggest by Sean Mooney where legacy port binding was used. On train seems there aren't backported patches to solve the issue. I enabled debug option on neutron and here there is the dhcp-agent.log from the exact time when the live migration started: http://paste.openstack.org/show/805325/ Here there is the openvswitch-agent log from the source kvm node: http://paste.openstack.org/show/805327/ Here there is the openvswich agent log from the destination kvm node: http://paste.openstack.org/show/805329/ I am using openvswitch mechanism driver and iptables_hybrid firewall driver. Please any help will be appreciated Ignazio To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1928299/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1928299] [NEW] centos7 train vm live migration stops network on vm for some minutes
Public bug reported: Hello, I have upgraded my centos 7 openstack installation from Stein to Train. On train I am facing an issue with live migration: when a vm is migrated from one kvm node to another, it stops to respond to ping requests from some minutes. I had the same issue on Stein and I resolved it with a workaround suggest by Sean Mooney where legacy port binding was used. On train seems there aren't backported patches to solve the issue. I enabled debug option on neutron and here there is the dhcp-agent.log from the exact time when the live migration started: http://paste.openstack.org/show/805325/ Here there is the openvswitch-agent log from the source kvm node: http://paste.openstack.org/show/805327/ Here there is the openvswich agent log from the destination kvm node: http://paste.openstack.org/show/805329/ I am using openvswitch mechanism driver and iptables_hybrid firewall driver. Please any help will be appreciated Ignazio ** Affects: neutron Importance: Undecided Status: New ** Tags: train-ignazio -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1928299 Title: centos7 train vm live migration stops network on vm for some minutes Status in neutron: New Bug description: Hello, I have upgraded my centos 7 openstack installation from Stein to Train. On train I am facing an issue with live migration: when a vm is migrated from one kvm node to another, it stops to respond to ping requests from some minutes. I had the same issue on Stein and I resolved it with a workaround suggest by Sean Mooney where legacy port binding was used. On train seems there aren't backported patches to solve the issue. I enabled debug option on neutron and here there is the dhcp-agent.log from the exact time when the live migration started: http://paste.openstack.org/show/805325/ Here there is the openvswitch-agent log from the source kvm node: http://paste.openstack.org/show/805327/ Here there is the openvswich agent log from the destination kvm node: http://paste.openstack.org/show/805329/ I am using openvswitch mechanism driver and iptables_hybrid firewall driver. Please any help will be appreciated Ignazio To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1928299/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
