** Also affects: nova/queens
Importance: Undecided
Status: New
** Changed in: nova/queens
Status: New => In Progress
** Changed in: nova/queens
Importance: Undecided => Medium
** Changed in: nova/queens
Assignee: (unassigned) => huanhongda (hongda)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1791075
Title:
update_available_resource periodic does not take into account all
evacuation states
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) queens series:
In Progress
Status in OpenStack Compute (nova) rocky series:
Fix Committed
Bug description:
Current _update_usage_from_migrations code takes into account only
REBUILDING task state, while not handling properly rebuilding spawn
and rebuilding volume attachments. This can cause issues with numa
topologies or pci devices if several instances are being evacuated and
some of them begin evacuation prior to update_available_resource
periodic pass and others immediately after, causing latter ones to
claim e.g. already pinned cpus.
Here is an example traceback that appears in nova-compute log after
the instance was evacuated:
2018-06-27T16:16:59.181573+02:00 compute-0-8.domain.tld nova-compute[19571]:
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
[req-79bc5f9f-9d5e-4f55-ad56-8351930afcb3 - - - - -] Error updating resources
for node compute-0-8.domain.tld.
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager Traceback (most
recent call last):
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6533, in
update_available_resource_for_node
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
rt.update_available_resource(context, periodic=True)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 594,
in update_available_resource
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
self._update_available_resource(context, resources, periodic=periodic)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in
inner
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager return f(*args,
**kwargs)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 661,
in _update_available_resource
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
self._update_usage_from_instances(context, instances)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 1035,
in _update_usage_from_instances
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
self._update_usage_from_instance(context, instance)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 1001,
in _update_usage_from_instance
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
self._update_usage(instance, sign=sign)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 834,
in _update_usage
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
self.compute_node, usage, free)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/virt/hardware.py", line 1491, in
get_host_numa_usage_from_instance
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
host_numa_topology, instance_numa_topology, free=free))
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/virt/hardware.py", line 1356, in
numa_usage_from_instances
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
newcell.pin_cpus(pinned_cpus)
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager File
"/usr/lib/python2.7/dist-packages/nova/objects/numa.py", line 85, in pin_cpus
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
pinned=list(self.pinned_cpus))
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager CPUPinningInvalid:
Cannot pin/unpin cpus [10, 34] from the following pinned set [9, 10, 34, 33]
2018-06-27 16:16:59.163 19571 ERROR nova.compute.manager
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1791075/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
