Reviewed: https://review.opendev.org/675704
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=f578146f372386e1889561cba33e95495e66ce97
Submitter: Zuul
Branch: master
commit f578146f372386e1889561cba33e95495e66ce97
Author: Matt Riedemann <mriedem...@gmail.com>
Date: Fri Aug 9 17:17:45 2019 -0400
rt: only map compute node if we created it
If ComputeNode.create() fails, the update_available_resource
periodic will not try to create it again because it will be
mapped in the compute_nodes dict and _init_compute_node will
return early but trying to save changes to that ComputeNode
object later will fail because there is no id on the object,
since we failed to create it in the DB.
This simply reverses the logic such that we only map the
compute node if we successfully created it.
Some existing _init_compute_node testing had to be changed
since it relied on the order of when the ComputeNode object
is created and put into the compute_nodes dict in order
to pass the object along to some much lower-level PCI
tracker code, which was arguably mocking too deep for a unit
test. That is changed to avoid the low-level mocking and
assertions and just assert that _setup_pci_tracker is called
as expected.
Change-Id: I9fa1d509a3de405d6246fb8670612c65c10cc93b
Closes-Bug: #1839674
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1839674
Title:
ResourceTracker.compute_nodes won't try to create a ComputeNode a
second time if the first create() fails
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) ocata series:
Triaged
Status in OpenStack Compute (nova) pike series:
Triaged
Status in OpenStack Compute (nova) queens series:
Triaged
Status in OpenStack Compute (nova) rocky series:
In Progress
Status in OpenStack Compute (nova) stein series:
In Progress
Bug description:
I found this while writing a functional recreate test for bug 1839560.
As of this change in Ocata:
https://github.com/openstack/nova/commit/1c967593fbb0ab8b9dc8b0b509e388591d32f537
The ResourceTracker.compute_nodes dict will store the ComputeNode
object *before* trying to create it:
https://github.com/openstack/nova/blob/6b7d0caad86fe32ffc49a8672de1eb7258f3b919/nova/compute/resource_tracker.py#L570-L571
The problem is if ComputeNode.create() fails for whatever reason, the
next run through update_available_resource won't try to create the
ComputeNode again because of this:
https://github.com/openstack/nova/blob/6b7d0caad86fe32ffc49a8672de1eb7258f3b919/nova/compute/resource_tracker.py#L546
And eventually you get errors like this:
b'2019-08-09 17:02:59,356 ERROR [nova.compute.manager] Error updating
resources for node node2.'
b'Traceback (most recent call last):'
b' File "/home/osboxes/git/nova/nova/compute/manager.py", line 8250, in
_update_available_resource_for_node'
b' startup=startup)'
b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line
715, in update_available_resource'
b' self._update_available_resource(context, resources,
startup=startup)'
b' File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_concurrency/lockutils.py",
line 328, in inner'
b' return f(*args, **kwargs)'
b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line
796, in _update_available_resource'
b' self._update(context, cn, startup=startup)'
b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line
1052, in _update'
b' self.old_resources[nodename] = old_compute'
b' File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py",
line 220, in __exit__'
b' self.force_reraise()'
b' File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py",
line 196, in force_reraise'
b' six.reraise(self.type_, self.value, self.tb)'
b' File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/six.py",
line 693, in reraise'
b' raise value'
b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line
1046, in _update'
b' compute_node.save()'
b' File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_versionedobjects/base.py",
line 226, in wrapper'
b' return fn(self, *args, **kwargs)'
b' File "/home/osboxes/git/nova/nova/objects/compute_node.py", line 352,
in save'
b' db_compute = db.compute_node_update(self._context, self.id,
updates)'
b' File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_versionedobjects/base.py",
line 67, in getter'
b' self.obj_load_attr(name)'
b' File
"/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_versionedobjects/base.py",
line 603, in obj_load_attr'
b' _("Cannot load \'%s\' in the base class") % attrname)'
b"NotImplementedError: Cannot load 'id' in the base class"
We should only map the ComputeNode when we've successfully created it.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1839674/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
