** Changed in: nova Importance: Undecided => Medium ** Also affects: nova/newton Importance: Undecided Status: New
** Also affects: nova/ocata Importance: Undecided Status: New ** Changed in: nova/newton Status: New => In Progress ** Changed in: nova/ocata Status: New => In Progress ** Changed in: nova/newton Importance: Undecided => Medium ** Changed in: nova/ocata Importance: Undecided => Medium ** Changed in: nova/newton Assignee: (unassigned) => Lee Yarwood (lyarwood) ** Changed in: nova/ocata Assignee: (unassigned) => Lee Yarwood (lyarwood) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1685185 Title: disconnect_volume not called when rebase failures are encountered during swap_volume Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: In Progress Status in OpenStack Compute (nova) ocata series: In Progress Bug description: Description =========== At present disconnect_volume is not called when rebase failures are encountered during swap_volume. This results in the new volume remaining connected to the compute host prior to terminate_connection then being called. This can in turn lead to left over devices remaining on the compute host for some volume backends such as LVM/iSCSI. Steps to reproduce ================== Downstream, the easiest way to reproduce this is via https://bugzilla.redhat.com/show_bug.cgi?id=1401755 : # sudo setenforce 1 # nova update-volume ${instance_uuid} \ ${attached_NFS_volume_id} \ ${unattached_iSCSI_volume_id} Upstream, I've been unable to get devstack to even work correctly with SELinux in enforcing mode so I've been unable to reproduce the rollback this way. Expected result =============== New volume disconnected from compute host. Actual result ============= New volume remains connected to compute host. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ N/A 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? Libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? NFS + LVM/iSCSI 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== Example Newton trace : 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher [req-f31f6110-e880-41e8-be1a-2c41ea7fd9ce 75fbc7a6db34480091d2a53e2e20b695 62e53e5e804e49a9890928a5a4846f60 - - -] Exception during message handling: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/disk/by-id/dm-uuid-mpath-360014053aac4f90daef4d76baa773169': Permission denied 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher payload) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5033, in swap_volume 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher resize_to) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4980, in _swap_volume 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher self.volume_api.unreserve_volume(context, new_volume_id) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4958, in _swap_volume 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher resize_to) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1434, in swap_volume 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher self._swap_volume(guest, disk_dev, conf.source_path, resize_to) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1391, in _swap_volume 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher dev.rebase(new_path, copy=True, reuse_ext=True) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 546, in rebase 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher self._disk, base, self.REBASE_DEFAULT_BANDWIDTH, flags=flags) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher rv = execute(f, *args, **kwargs) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher six.reraise(c, e, tb) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher rv = meth(*args, **kwargs) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/libvirt.py", line 930, in blockRebase 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self) 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/disk/by-id/dm-uuid-mpath-360014053aac4f90daef4d76baa773169': Permission denied 2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1685185/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp