Reviewed: https://review.openstack.org/262341
Committed:
https://git.openstack.org/cgit/openstack/nova/commit/?id=f93d808bc3ba1c0a97ec46f345fad9c0d00865f8
Submitter: Jenkins
Branch: master
commit f93d808bc3ba1c0a97ec46f345fad9c0d00865f8
Author: David Edery <david.ed...@alcatel-lucent.com>
Date: Wed Dec 30 02:05:19 2015 +0200
Fix create snapshot failure on VMs with SRIOV
One use-case of guest VM network protection using SRIOV ports is having
two direct ports connected to a VM, each one is related to a network that
is connected to a different physical NIC on the host (e.g. phyNet1 on
eth0 and phyNet2 on eth1). In this use-case, due to some physical NICs
limitations it's advised to configure both ports with the same MAC
address (or else outgoing or incoming traffic will not reach its
destination in case of fail-over).
Snapshot creation on such VMs fails since libvirt's detach-interface
doesn't know which interface of the two to detach and fails. This is why
I've changed the call inside _detach_sriov_ports from (the equivalent
of) detach-interface to _detach_pci_devices with the source-device
pci address of the SRIOV VF (always present in the context of SRIOV
of course).
This fix was tested on a real environment containing the above type of
VMs. test_driver.test_detach_sriov_ports was slightly modified so
that the VIF from which data is sent to _detach_pci_devices will
contain the correct SRIOV values (pci_slot, vlan and hw_veb VIF type)
Change-Id: If3edc1965c01a077eb61984a442e0d778d870d75
Closes-Bug: #1529785
** Changed in: nova
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1529785
Title:
Create snapshot fails on VM containing multiple SRIOV vNICs configured
with the same MAC
Status in OpenStack Compute (nova):
Fix Released
Bug description:
Use case:
SRIOV protection on the guest VM with Intel NICs on the host requires (at one
scenario) 2 SRIOV ports attached to different physical NICs (e.g. phyNet1 &
phyNet2) configured with the same MAC address (and as a result, same IP).
Problem:
Create snapshot on such VM fails
Reason:
(from nova-compute.log)
2015-12-28 12:03:37.594 4616 ERROR oslo_messaging.rpc.dispatcher
[req-57a8c147-b945-43e8-9915-5a52d3b7deb9 18800eca7d674a60995039349089e8da
0b5c257568424704854f8d10342edf80 - - -] Exception during message handling:
operation failed: multiple devices matching mac address fa:16:3e:e1:7f:3f found
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher Traceback
(most recent call last):
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142,
in _dispatch_and_reply
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
executor_callback))
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186,
in _dispatch
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
executor_callback)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130,
in _do_dispatch
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher result =
func(ctxt, **new_args)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6917, in
snapshot_instance
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher return
self.manager.snapshot_instance(ctxt, image_id, instance)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher payload)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 341, in
decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 312, in
decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in
decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
kwargs['instance'], e, sys.exc_info())
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 357, in
decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 417, in
decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
instance=instance)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 407, in
decorated_function
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher *args,
**kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3297, in
snapshot_instance
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
task_states.IMAGE_SNAPSHOT)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3327, in
_snapshot_instance
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
update_task_state)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1422, in
snapshot
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
self._detach_sriov_ports(context, instance, virt_dom)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3121, in
_detach_sriov_ports
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
libvirt.VIR_DOMAIN_AFFECT_LIVE)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher result =
proxy_call(self._autowrap, f, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher rv =
execute(f, *args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
six.reraise(c, e, tb)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher rv =
meth(*args, **kwargs)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1193, in detachDeviceFlags
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher if ret
== -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
libvirtError: operation failed: multiple devices matching mac address
fa:16:3e:e1:7f:3f found
2015-12-28 12:03:37.594 4616 TRACE oslo_messaging.rpc.dispatcher
Observation:
When trying to manually call "virsh detach-interface instance-0000005b
hostdev --mac fa:16:3e:e1:7f:3f" (MAC addr being the MAC addr of both SRIOV
ports) the result is:
error: Domain has multiple interfaces matching MAC address fa:16:3e:e1:7f:3f.
You must use detach-device and specify the device pci address to remove it.
Solution:
In driver.py call the equivalent of "virsh detach-device <pci addr>" instead
of: guest.detach_device(cfg, live=True) - cfg being the
<interface>...</interface> XML
(e.g.:
<interface type="hostdev" managed="yes">
<mac address="fa:16:3e:e1:7f:3f"/>
<source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x1e"
function="0x6"/>
</source>
<vlan>
<tag id="2145"/>
</vlan>
</interface>)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1529785/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
