Reviewed: https://review.openstack.org/560454
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=4afa8c2b97929faaf8bbf2bbaa72235fab7d3d13
Submitter: Zuul
Branch: master
commit 4afa8c2b97929faaf8bbf2bbaa72235fab7d3d13
Author: Matt Riedemann <mriedem...@gmail.com>
Date: Wed Apr 11 10:43:34 2018 -0400
Fix race fail in test_resize_with_reschedule_then_live_migrate
The assertion in the test that the migration status is 'completed'
is flawed in that it assumes when the instance status is 'ACTIVE'
the migration is completed, which isn't True, since the instance
status gets changed before the migration is completed, but they are
very close in time so there is a race, which is how this test slipped
by. This fixes the issue by polling the migration status until it
is actually completed or we timeout.
Change-Id: I61f745667f4c003d7e3ca6f2f9a99194930ac892
Closes-Bug: #1762876
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1762876
Title:
test_resize_with_reschedule_then_live_migrate intermittently failing;
migration is not yet complete
Status in OpenStack Compute (nova):
Fix Released
Bug description:
I noticed this in a stable/pike functional test job run:
http://logs.openstack.org/46/560146/2/check/nova-tox-
functional/4a9d1fd/job-output.txt.gz#_2018-04-10_21_37_20_943583
2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr:
2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
2018-04-10 21:37:20.945029 | ubuntu-xenial | Traceback (most recent call
last):
2018-04-10 21:37:20.945231 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py",
line 457, in fire_timers
2018-04-10 21:37:20.945268 | ubuntu-xenial | timer()
2018-04-10 21:37:20.945467 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py",
line 58, in __call__
2018-04-10 21:37:20.945513 | ubuntu-xenial | cb(*args, **kw)
2018-04-10 21:37:20.945598 | ubuntu-xenial | File "nova/utils.py", line
1030, in context_wrapper
2018-04-10 21:37:20.945650 | ubuntu-xenial | func(*args, **kwargs)
2018-04-10 21:37:20.945756 | ubuntu-xenial | File
"nova/compute/manager.py", line 5620, in dispatch_live_migration
2018-04-10 21:37:20.945839 | ubuntu-xenial |
self._do_live_migration(*args, **kwargs)
2018-04-10 21:37:20.945939 | ubuntu-xenial | File
"nova/compute/manager.py", line 5599, in _do_live_migration
2018-04-10 21:37:20.945993 | ubuntu-xenial | clean_task_state=True)
2018-04-10 21:37:20.946194 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2018-04-10 21:37:20.946246 | ubuntu-xenial | self.force_reraise()
2018-04-10 21:37:20.946452 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2018-04-10 21:37:20.946532 | ubuntu-xenial | six.reraise(self.type_,
self.value, self.tb)
2018-04-10 21:37:20.946679 | ubuntu-xenial | File
"nova/compute/manager.py", line 5588, in _do_live_migration
2018-04-10 21:37:20.946764 | ubuntu-xenial | block_migration,
migrate_data)
2018-04-10 21:37:20.946856 | ubuntu-xenial | File "nova/virt/fake.py",
line 497, in live_migration
2018-04-10 21:37:20.946901 | ubuntu-xenial | migrate_data)
2018-04-10 21:37:20.947003 | ubuntu-xenial | File
"nova/exception_wrapper.py", line 76, in wrapped
2018-04-10 21:37:20.947069 | ubuntu-xenial | function_name,
call_dict, binary)
2018-04-10 21:37:20.947270 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2018-04-10 21:37:20.947322 | ubuntu-xenial | self.force_reraise()
2018-04-10 21:37:20.947536 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2018-04-10 21:37:20.947619 | ubuntu-xenial | six.reraise(self.type_,
self.value, self.tb)
2018-04-10 21:37:20.947707 | ubuntu-xenial | File
"nova/exception_wrapper.py", line 67, in wrapped
2018-04-10 21:37:20.947779 | ubuntu-xenial | return f(self, context,
*args, **kw)
2018-04-10 21:37:20.947878 | ubuntu-xenial | File
"nova/compute/manager.py", line 218, in decorated_function
2018-04-10 21:37:20.947950 | ubuntu-xenial | kwargs['instance'], e,
sys.exc_info())
2018-04-10 21:37:20.948160 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 220, in __exit__
2018-04-10 21:37:20.948214 | ubuntu-xenial | self.force_reraise()
2018-04-10 21:37:20.948433 | ubuntu-xenial | File
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise
2018-04-10 21:37:20.948510 | ubuntu-xenial | six.reraise(self.type_,
self.value, self.tb)
2018-04-10 21:37:20.948607 | ubuntu-xenial | File
"nova/compute/manager.py", line 206, in decorated_function
2018-04-10 21:37:20.948687 | ubuntu-xenial | return function(self,
context, *args, **kwargs)
2018-04-10 21:37:20.948802 | ubuntu-xenial | File
"nova/compute/manager.py", line 5824, in _post_live_migration
2018-04-10 21:37:20.948856 | ubuntu-xenial | instance, source_node)
2018-04-10 21:37:20.949000 | ubuntu-xenial | File
"nova/compute/resource_tracker.py", line 1294, in
delete_allocation_for_migrated_instance
2018-04-10 21:37:20.949111 | ubuntu-xenial |
self._delete_allocation_for_moved_instance(instance, node, 'migrated')
2018-04-10 21:37:20.949244 | ubuntu-xenial | File
"nova/compute/resource_tracker.py", line 1302, in
_delete_allocation_for_moved_instance
2018-04-10 21:37:20.949319 | ubuntu-xenial | cn_uuid =
self.compute_nodes[node].uuid
2018-04-10 21:37:20.949363 | ubuntu-xenial | KeyError: u'host3'
But I see it's also happening on master:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1762876/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
