Public bug reported:
CPU pinning support was implemented as part of this blueprint:
http://specs.openstack.org/openstack/nova-specs/specs/juno/approved
/virt-driver-cpu-pinning.html
However, CPU pinning support is broken in some libvirt versions
(summarized below), resulting in exceptions when attempting to schedule
instances with the 'hw:cpu_policy' flavor key.
We should add a libvirt version test against known broken versions and
use that to determine whether or not to support the flavor keys.
This is somewhat related to #1422775 ("nova libvirt driver assumes qemu
support for NUMA pinning").
---
# Testing Configuration
Testing was conducted in a container which provided a single-node,
Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with
devstack). The yum-provided libvirt and its dependencies were removed
and libvirt and libvirt-python were built and installed from source.
# Results
The results are as follows:
versions status
-------- ------
1.2.9 ok
1.2.9.1 ok
1.2.9.2 fail
1.2.10 fail
1.2.11 ok
1.2.12 ok
v1.2.9.2 is broken by this (backported) patch:
https://www.redhat.com/archives/libvir-
list/2014-November/msg00275.html
This can be seen as commit
e226772 (qemu: fix domain startup failing with 'strict' mode in
numatune)
v1.2.10 inherits is broken at checkout but can be fixed by applying
these three patches (yes, one of these broke v1.2.9.2 - the irony is not
lost on me):
[0/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00274.html
- [1/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00273.html
- [2/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00276.html
- [3/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html
# Error logs
v1.2.9.2 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2301, in
_build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2171, in
_build_and_run_instance
flavor=flavor)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in
_create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in
_create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82,
in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in
_create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in
doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in
proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in
execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in
tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in
createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
dom=self)
libvirtError: Failed to create controller cpu for group: No such file or
directory
v1.2.10 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2342, in
_build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2215, in
_build_and_run_instance
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2356, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4375, in
_create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4306, in
_create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,
in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4296, in
_create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in
doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in
proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in
execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in
tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1037, in
createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
dom=self)
libvirtError: Unable to write to
'/sys/fs/cgroup/cpuset/system.slice/docker.service/machine.slice/machine-qemu\x2dinstance\x2d0000000a.scope/cpuset.mems':
Device or resource busy
** Affects: nova
Importance: Undecided
Status: New
** Tags: libvirt
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1438226
Title:
nova libvirt driver assumes libvirt support for CPU pinning
Status in OpenStack Compute (Nova):
New
Bug description:
CPU pinning support was implemented as part of this blueprint:
http://specs.openstack.org/openstack/nova-
specs/specs/juno/approved/virt-driver-cpu-pinning.html
However, CPU pinning support is broken in some libvirt versions
(summarized below), resulting in exceptions when attempting to
schedule instances with the 'hw:cpu_policy' flavor key.
We should add a libvirt version test against known broken versions and
use that to determine whether or not to support the flavor keys.
This is somewhat related to #1422775 ("nova libvirt driver assumes
qemu support for NUMA pinning").
---
# Testing Configuration
Testing was conducted in a container which provided a single-node,
Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built
with devstack). The yum-provided libvirt and its dependencies were
removed and libvirt and libvirt-python were built and installed from
source.
# Results
The results are as follows:
versions status
-------- ------
1.2.9 ok
1.2.9.1 ok
1.2.9.2 fail
1.2.10 fail
1.2.11 ok
1.2.12 ok
v1.2.9.2 is broken by this (backported) patch:
https://www.redhat.com/archives/libvir-
list/2014-November/msg00275.html
This can be seen as commit
e226772 (qemu: fix domain startup failing with 'strict' mode in
numatune)
v1.2.10 inherits is broken at checkout but can be fixed by applying
these three patches (yes, one of these broke v1.2.9.2 - the irony is
not lost on me):
[0/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00274.html
- [1/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00273.html
- [2/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00276.html
- [3/3]
https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html
# Error logs
v1.2.9.2 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2301, in
_build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2171, in
_build_and_run_instance
flavor=flavor)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in
_create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in
_create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in
_create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in
doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in
proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in
execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in
tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in
createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
failed', dom=self)
libvirtError: Failed to create controller cpu for group: No such file or
directory
v1.2.10 produces the following exception:
Traceback (most recent call last):
File "/opt/stack/nova/nova/compute/manager.py", line 2342, in
_build_resources
yield resources
File "/opt/stack/nova/nova/compute/manager.py", line 2215, in
_build_and_run_instance
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2356, in spawn
block_device_info=block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4375, in
_create_domain_and_network
power_on=power_on)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4306, in
_create_domain
LOG.error(err)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
85, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4296, in
_create_domain
domain.createWithFlags(launch_flags)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in
doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in
proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in
execute
six.reraise(c, e, tb)
File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in
tworker
rv = meth(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1037, in
createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
failed', dom=self)
libvirtError: Unable to write to
'/sys/fs/cgroup/cpuset/system.slice/docker.service/machine.slice/machine-qemu\x2dinstance\x2d0000000a.scope/cpuset.mems':
Device or resource busy
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1438226/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
