Public bug reported: This is a low frequency gate failure in unit tests.
Example log: http://logs.openstack.org/10/645210/4/check/openstack-tox-py37/688ffa8/job-output.txt.gz Logstash search: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22line%20171,%20in%20test_port_ip_update_revises%5C%22%20AND%20voting:1&from=864000s 2019-03-25 12:16:24.333688 | ubuntu-bionic | ============================== 2019-03-25 12:16:24.333764 | ubuntu-bionic | Failed 1 tests - output below: 2019-03-25 12:16:24.333837 | ubuntu-bionic | ============================== 2019-03-25 12:16:24.333863 | ubuntu-bionic | 2019-03-25 12:16:24.334052 | ubuntu-bionic | neutron.tests.unit.services.revisions.test_revision_plugin.TestRevisionPlugin.test_port_ip_update_revises 2019-03-25 12:16:24.334243 | ubuntu-bionic | --------------------------------------------------------------------------------------------------------- 2019-03-25 12:16:24.334271 | ubuntu-bionic | 2019-03-25 12:16:24.334326 | ubuntu-bionic | Captured traceback: 2019-03-25 12:16:24.334381 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~ 2019-03-25 12:16:24.334471 | ubuntu-bionic | b'Traceback (most recent call last):' 2019-03-25 12:16:24.334662 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/base.py", line 174, in func' 2019-03-25 12:16:24.334754 | ubuntu-bionic | b' return f(self, *args, **kwargs)' 2019-03-25 12:16:24.335103 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/services/revisions/test_revision_plugin.py", line 171, in test_port_ip_update_revises' 2019-03-25 12:16:24.335243 | ubuntu-bionic | b" response = self._update('ports', port['port']['id'], new)" 2019-03-25 12:16:24.335490 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 601, in _update' 2019-03-25 12:16:24.335642 | ubuntu-bionic | b' self.assertEqual(expected_code, res.status_int)' 2019-03-25 12:16:24.335921 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual' 2019-03-25 12:16:24.336035 | ubuntu-bionic | b' self.assertThat(observed, matcher, message)' 2019-03-25 12:16:24.336297 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat' 2019-03-25 12:16:24.336372 | ubuntu-bionic | b' raise mismatch_error' 2019-03-25 12:16:24.336486 | ubuntu-bionic | b'testtools.matchers._impl.MismatchError: 200 != 400' 2019-03-25 12:16:24.336523 | ubuntu-bionic | b'' 2019-03-25 12:16:24.336549 | ubuntu-bionic | 2019-03-25 12:16:24.336599 | ubuntu-bionic | Captured stderr: 2019-03-25 12:16:24.336650 | ubuntu-bionic | ~~~~~~~~~~~~~~~~ 2019-03-25 12:16:24.337086 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.' 2019-03-25 12:16:24.337157 | ubuntu-bionic | b' DeprecationWarning)' 2019-03-25 12:16:24.337594 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.' 2019-03-25 12:16:24.337664 | ubuntu-bionic | b' DeprecationWarning)' 2019-03-25 12:16:24.337701 | ubuntu-bionic | b'' With some extra debug logging added I managed to obtain this error message: ERROR [neutron.tests.unit.db.test_db_base_plugin_v2] XXX b\'{"NeutronError": {"type": "InvalidIpForNetwork", "message": "IP address 10.0.0.255 is not a valid IP for any of the subnets on the specified network.", "detail": ""}}\' Reading the unit test source it seems likely that a random IP+1 is occasionally the subnet broadcast address which is invalid as a fixed_ip. https://opendev.org/openstack/neutron/src/commit/1ea9326fda303b48905d7f7748d320ba8e9322aa/neutron/tests/unit/services/revisions/test_revision_plugin.py#L169 I'm going to upload an attempted fix soon. ** Affects: neutron Importance: Medium Assignee: Bence Romsics (bence-romsics) Status: In Progress ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1821948 Title: Unstable unit test uses subnet broadcast address Status in neutron: In Progress Bug description: This is a low frequency gate failure in unit tests. Example log: http://logs.openstack.org/10/645210/4/check/openstack-tox-py37/688ffa8/job-output.txt.gz Logstash search: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22line%20171,%20in%20test_port_ip_update_revises%5C%22%20AND%20voting:1&from=864000s 2019-03-25 12:16:24.333688 | ubuntu-bionic | ============================== 2019-03-25 12:16:24.333764 | ubuntu-bionic | Failed 1 tests - output below: 2019-03-25 12:16:24.333837 | ubuntu-bionic | ============================== 2019-03-25 12:16:24.333863 | ubuntu-bionic | 2019-03-25 12:16:24.334052 | ubuntu-bionic | neutron.tests.unit.services.revisions.test_revision_plugin.TestRevisionPlugin.test_port_ip_update_revises 2019-03-25 12:16:24.334243 | ubuntu-bionic | --------------------------------------------------------------------------------------------------------- 2019-03-25 12:16:24.334271 | ubuntu-bionic | 2019-03-25 12:16:24.334326 | ubuntu-bionic | Captured traceback: 2019-03-25 12:16:24.334381 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~ 2019-03-25 12:16:24.334471 | ubuntu-bionic | b'Traceback (most recent call last):' 2019-03-25 12:16:24.334662 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/base.py", line 174, in func' 2019-03-25 12:16:24.334754 | ubuntu-bionic | b' return f(self, *args, **kwargs)' 2019-03-25 12:16:24.335103 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/services/revisions/test_revision_plugin.py", line 171, in test_port_ip_update_revises' 2019-03-25 12:16:24.335243 | ubuntu-bionic | b" response = self._update('ports', port['port']['id'], new)" 2019-03-25 12:16:24.335490 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 601, in _update' 2019-03-25 12:16:24.335642 | ubuntu-bionic | b' self.assertEqual(expected_code, res.status_int)' 2019-03-25 12:16:24.335921 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual' 2019-03-25 12:16:24.336035 | ubuntu-bionic | b' self.assertThat(observed, matcher, message)' 2019-03-25 12:16:24.336297 | ubuntu-bionic | b' File "/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat' 2019-03-25 12:16:24.336372 | ubuntu-bionic | b' raise mismatch_error' 2019-03-25 12:16:24.336486 | ubuntu-bionic | b'testtools.matchers._impl.MismatchError: 200 != 400' 2019-03-25 12:16:24.336523 | ubuntu-bionic | b'' 2019-03-25 12:16:24.336549 | ubuntu-bionic | 2019-03-25 12:16:24.336599 | ubuntu-bionic | Captured stderr: 2019-03-25 12:16:24.336650 | ubuntu-bionic | ~~~~~~~~~~~~~~~~ 2019-03-25 12:16:24.337086 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.' 2019-03-25 12:16:24.337157 | ubuntu-bionic | b' DeprecationWarning)' 2019-03-25 12:16:24.337594 | ubuntu-bionic | b'/home/zuul/src/git.openstack.org/openstack/neutron/.tox/py37/lib/python3.7/site-packages/neutron_lib/context.py:154: DeprecationWarning: context.session is used with and without new enginefacade. Please update the code to use new enginefacede consistently.' 2019-03-25 12:16:24.337664 | ubuntu-bionic | b' DeprecationWarning)' 2019-03-25 12:16:24.337701 | ubuntu-bionic | b'' With some extra debug logging added I managed to obtain this error message: ERROR [neutron.tests.unit.db.test_db_base_plugin_v2] XXX b\'{"NeutronError": {"type": "InvalidIpForNetwork", "message": "IP address 10.0.0.255 is not a valid IP for any of the subnets on the specified network.", "detail": ""}}\' Reading the unit test source it seems likely that a random IP+1 is occasionally the subnet broadcast address which is invalid as a fixed_ip. https://opendev.org/openstack/neutron/src/commit/1ea9326fda303b48905d7f7748d320ba8e9322aa/neutron/tests/unit/services/revisions/test_revision_plugin.py#L169 I'm going to upload an attempted fix soon. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1821948/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
