Reviewed: https://review.opendev.org/c/openstack/neutron/+/806938
Committed:
https://opendev.org/openstack/neutron/commit/76ee64f4a8095ba207872d5de9613f93f208fdd4
Submitter: "Zuul (22348)"
Branch: master
commit 76ee64f4a8095ba207872d5de9613f93f208fdd4
Author: Terry Wilson <twil...@redhat.com>
Date: Wed Sep 1 14:19:19 2021 +0000
Fix neutron_pg_drop-related startup issues
There is a python-ovs bug that allows transactions to be executed
before it has started monitoring the db for changes. This breaks
ovsdbapp's AddCommand behavior, which looks up a row in memory
post-commit to return it to the caller, because we process the
reply from ovsdb-server that has the UUID in it, but when we try to
look it up in memory, it isn't there since that is handled via the
monitor notifications.
Since we don't care about the return value, we can just ignore the
KeyError. In addition, the request to Idl.cond_change() to monitor
only the neutron_pg_drop table was failing for a similar reason--
it is called before the initial monitor request is sent, so
directly setting table.condition will allow the condition to be
added to the initial monitor request.
Closes-Bug: #1938766
Change-Id: I78af15970a86923fb810e903d38634b6e60a8fe4
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1938766
Title:
neutron-functional-with-uwsgi fails with "KeyError: UUID" during
create_default_drop_port_group
Status in neutron:
Fix Released
Bug description:
Key failure is KeyError in create_default_drop_port_group().
"No such file or directory" and an error during
self.mech_driver.nb_ovn.ovsdb_connection.stop() are part of the cleanup process.
--
Recently we have been seeing pretty often functional tests issues with
errors like:
ft1.13:
neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_check_for_port_security_unknown_addresstesttools.testresult.real._StringException:
traceback-1: {{{
Traceback (most recent call last):
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/base.py",
line 363, in stop
self.mech_driver.nb_ovn.ovsdb_connection.stop()
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py",
line 153, in nb_ovn
self._post_fork_event.wait()
File "/usr/lib/python3.8/threading.py", line 558, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.8/threading.py", line 302, in wait
waiter.acquire()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/semaphore.py",
line 120, in acquire
hubs.get_hub().switch()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 313, in switch
return self.greenlet.switch()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/hubs/hub.py",
line 365, in run
self.wait(sleep_time)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/hubs/poll.py",
line 80, in wait
presult = self.do_poll(seconds)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/eventlet/hubs/epolls.py",
line 31, in do_poll
return self.poll.poll(seconds)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/fixtures/_fixtures/timeout.py",
line 52, in signal_handler
raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException
}}}
traceback-2: {{{
Traceback (most recent call last):
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/base.py",
line 351, in _collect_processes_logs
self._copy_log_file("%s.log" % northd_log, dst_northd)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/base.py",
line 356, in _copy_log_file
shutil.copyfile(
File "/usr/lib/python3.8/shutil.py", line 264, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/tmps9cyr99c/ovn_northd.log'
}}}
Traceback (most recent call last):
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py",
line 41, in setUp
super(_TestMaintenanceHelper, self).setUp()
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/base.py",
line 217, in setUp
self._start_idls()
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/base.py",
line 320, in _start_idls
self.mech_driver.pre_fork_initialize(
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py",
line 254, in pre_fork_initialize
self._create_neutron_pg_drop()
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py",
line 272, in _create_neutron_pg_drop
create_default_drop_port_group(pre_ovn_nb_api)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py",
line 1241, in create_default_drop_port_group
txn.add(nb_idl.pg_add_ports(pg_name, list(ports_with_pg)))
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File
"/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py",
line 274, in transaction
yield t
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/ovsdbapp/api.py",
line 110, in transaction
del self._nested_txns_map[cur_thread_id]
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/ovsdbapp/api.py",
line 61, in __exit__
self.result = self.commit()
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
line 65, in commit
raise result.ex
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 131, in run
txn.results.put(txn.do_commit())
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
line 136, in do_commit
self.post_commit(txn)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
line 73, in post_commit
command.post_commit(txn)
File
"/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/command.py",
line 94, in post_commit
row = self.api.tables[self.table_name].rows[real_uuid]
File "/usr/lib/python3.8/collections/__init__.py", line 1010, in __getitem__
raise KeyError(key)
KeyError: UUID('c9570a47-64da-4360-9c94-12ba459344c2')
Examples of such failures:
https://bd41d5ab9bd61c5e04d6-86672578d4e6ceb498f2d932b0da6815.ssl.cf5.rackcdn.com/798961/7/check/neutron-
functional-with-uwsgi/772270c/testr_results.html
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_856/801310/2/check/neutron-
functional-with-uwsgi/856f539/testr_results.html
https://7ee9afbce159ed2d3c7a-ae7c228c39808cabcdb014f8881dacdb.ssl.cf5.rackcdn.com/800278/7/check/neutron-
functional-with-uwsgi/6332cdf/testr_results.html
https://fc20912585430bbde389-7e8720184c7a4cb82719c44bcc579362.ssl.cf5.rackcdn.com/802687/1/check/neutron-
functional-with-uwsgi/d5d2430/testr_results.html
In all cases the stacktrace is pretty the same but failing tests are
different.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1938766/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
