Reviewed: https://review.opendev.org/655843
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=1a4a521fefb71aa0817770f265063a9150653743
Submitter: Zuul
Branch: master
commit 1a4a521fefb71aa0817770f265063a9150653743
Author: Stephen Finucane <sfinu...@redhat.com>
Date: Fri Apr 26 12:10:11 2019 +0100
test_rpc: Stop f****** with global state
We're occasionally seeing stacktraces like this in our tests:
Fatal Python error: Cannot recover from stack overflow.
Current thread 0x00007fe66549f740 (most recent call first):
File
"...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614
in _get
File
"...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2183
in __getattr__
File
"...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614
in _get
File
"...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2183
in __getattr__
File
"...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614
in _get
...
From a look at the oslo.config source, this seems to be occurring
because 'ConfigOpts.__cache' is somehow undefined, which results in the
'_get' method attempting to call '__getattr__' [1], which calls '_get'
[2], which calls '__getattr__' and so on.
The exact reason this is happening isn't clear, but what is clear is
that how we handle global config options in the tests that are failing
is very strange and potentially subject to race conditions. We have a
clear pattern for mocking everything and anything - the mock module -
and we should be using this here. Start doing so, reworking a lot of the
tests in the process, in order to avoid messing with oslo.config and
triggering the issue entirely.
[1]
https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2614
[2]
https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2183
Change-Id: I468cef94185a1b59f379ca527050450e03664c67
Signed-off-by: Stephen Finucane <sfinu...@redhat.com>
Closes-Bug: #1825435
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1825435
Title:
TestRPC unit tests intermittently fail with "'>' not supported between
instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal
Python error: Cannot recover from stack overflow."
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Seen here:
http://logs.openstack.org/45/649345/7/check/openstack-tox-py36/ba15c17
/job-output.txt.gz#_2019-04-18_18_12_52_121182
2019-04-18 18:12:52.121182 | ubuntu-bionic | {3}
nova.tests.unit.test_rpc.TestRPC.test_get_transport_url_null [] ... inprogress
2019-04-18 18:12:52.138024 | ubuntu-bionic | '>' not supported between
instances of 'NoneType' and 'datetime.datetime'
2019-04-18 18:12:52.353474 | ubuntu-bionic | ERROR: InvocationError for
command /home/zuul/src/git.openstack.org/openstack/nova/.tox/py36/bin/stestr
run (exited with code 1)
Which seems to kill the test and stestr.
There is also a stack overflow here:
http://logs.openstack.org/45/649345/7/check/openstack-tox-py36/ba15c17
/job-output.txt.gz#_2019-04-18_18_10_53_423952
2019-04-18 18:10:53.423952 | ubuntu-bionic | Fatal Python error: Cannot
recover from stack overflow.
2019-04-18 18:10:53.423999 | ubuntu-bionic |
2019-04-18 18:10:53.424162 | ubuntu-bionic | Current thread
0x00007f34d2bcc740 (most recent call first):
2019-04-18 18:10:53.424472 | ubuntu-bionic | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py",
line 2614 in _get
2019-04-18 18:10:53.424794 | ubuntu-bionic | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py",
line 2183 in __getattr__
...
2019-04-18 18:10:53.476982 | ubuntu-bionic | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py",
line 2183 in __getattr__
2019-04-18 18:10:53.477236 | ubuntu-bionic | ...
2019-04-18 18:10:53.477293 | ubuntu-bionic | Aborted
The stack overflow seems to be just nova since April 15:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%7C%20Fatal%20Python%20error%3A%20Cannot%20recover%20from%20stack%20overflow.%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
I don't see anything that looks related for nova changes around April
15, but maybe something that was released to upper-constraints on
April 15:
https://github.com/openstack/requirements/commit/a96ee0e258aafb2880149b3e25dd5959f7b37c09
Nothing in there looks obvious though.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1825435/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
