Public bug reported: Sometimes test_volume_swap_server_with_error functional test fails [1]. It feels like a timing issue in the test itself. But regardless if it fails or passes there is a suspicious stack trace in the log showing that something is wrong with the swap volume codepath:
2019-02-18 22:40:07,103 ERROR [nova.compute.manager] Failed to connect to volume 9c6d9c2d-7a8f-4c80-938d-3bf062b8d489 with volume at /dev/sdb Traceback (most recent call last): File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5701, in _swap_volume bdm, new_attachment_id, mountpoint) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5688, in _init_volume_connection new_cinfo['serial'] = new_volume_id TypeError: 'tuple' object does not support item assignment 2019-02-18 22:40:07,521 ERROR [oslo_messaging.rpc.server] Exception during message handling Traceback (most recent call last): File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming res = self.dispatcher.dispatch(message) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch return self._do_dispatch(endpoint, method, ctxt, args) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch result = func(ctxt, **new_args) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/exception_wrapper.py", line 79, in wrapped function_name, call_dict, binary, tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise raise value File "/home/zuul/src/git.openstack.org/openstack/nova/nova/exception_wrapper.py", line 69, in wrapped return f(self, context, *args, **kw) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/utils.py", line 1301, in decorated_function return function(self, context, *args, **kwargs) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 213, in decorated_function kwargs['instance'], e, sys.exc_info()) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise raise value File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 201, in decorated_function return function(self, context, *args, **kwargs) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5859, in swap_volume is_cinder_migration) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5757, in _swap_volume context, new_attachment_id) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise raise value File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5701, in _swap_volume bdm, new_attachment_id, mountpoint) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5688, in _init_volume_connection new_cinfo['serial'] = new_volume_id TypeError: 'tuple' object does not support item assignment [1] http://logs.openstack.org/78/637578/1/check/nova-tox-functional-py35/14da3dc/testr_results.html.gz ** Affects: nova Importance: Low Status: New ** Tags: needs-functional-test notifications testing volumes ** Tags added: needs-functional-test notifications testing volumes ** Changed in: nova Importance: Undecided => Low -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1816538 Title: test_volume_swap_server_with_error produce a suspicious stack trace Status in OpenStack Compute (nova): New Bug description: Sometimes test_volume_swap_server_with_error functional test fails [1]. It feels like a timing issue in the test itself. But regardless if it fails or passes there is a suspicious stack trace in the log showing that something is wrong with the swap volume codepath: 2019-02-18 22:40:07,103 ERROR [nova.compute.manager] Failed to connect to volume 9c6d9c2d-7a8f-4c80-938d-3bf062b8d489 with volume at /dev/sdb Traceback (most recent call last): File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5701, in _swap_volume bdm, new_attachment_id, mountpoint) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5688, in _init_volume_connection new_cinfo['serial'] = new_volume_id TypeError: 'tuple' object does not support item assignment 2019-02-18 22:40:07,521 ERROR [oslo_messaging.rpc.server] Exception during message handling Traceback (most recent call last): File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming res = self.dispatcher.dispatch(message) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch return self._do_dispatch(endpoint, method, ctxt, args) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch result = func(ctxt, **new_args) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/exception_wrapper.py", line 79, in wrapped function_name, call_dict, binary, tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise raise value File "/home/zuul/src/git.openstack.org/openstack/nova/nova/exception_wrapper.py", line 69, in wrapped return f(self, context, *args, **kw) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/utils.py", line 1301, in decorated_function return function(self, context, *args, **kwargs) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 213, in decorated_function kwargs['instance'], e, sys.exc_info()) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise raise value File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 201, in decorated_function return function(self, context, *args, **kwargs) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5859, in swap_volume is_cinder_migration) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5757, in _swap_volume context, new_attachment_id) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional-py35/lib/python3.5/site-packages/six.py", line 693, in reraise raise value File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5701, in _swap_volume bdm, new_attachment_id, mountpoint) File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 5688, in _init_volume_connection new_cinfo['serial'] = new_volume_id TypeError: 'tuple' object does not support item assignment [1] http://logs.openstack.org/78/637578/1/check/nova-tox-functional-py35/14da3dc/testr_results.html.gz To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1816538/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp