Nothing special, Purity 4.1.5. These are not brand new array’s we have had them for a while. (FAS450) 8gb FC switches (Force 10) All hosts zoned, initiators manually created in the pure or configured by cinder does not seem to matter. The weird thing is everything seems to be working fine except the migration. Provisioning a new instance on the pure no problem, move a volume when not attached and boot on the pure, no probem. Migrate from the pure to the compellent, no problem. If you need more details on the Pure config let me know.
-Steve From: Remo Mattei [mailto:r...@italy1.com] Sent: Tuesday, May 29, 2018 9:55 PM To: Steven D. Searles <ssear...@zimcom.net> Cc: openstack@lists.openstack.org Subject: Re: [Openstack] Cinder volume live migration issue. Pure Storage Cinder-Libvirt What’s the config on PURE? On May 29, 2018, at 12:03 PM, Steven D. Searles <ssear...@zimcom.net<mailto:ssear...@zimcom.net>> wrote: Hello everyone, I am seeing a strange issue with cinder block live migration and libvirt and looking for some assistance. Environment: Openstack Pike OS: Ubuntu 16.04LTS Cinder FC Driver: Pure Storage Cinder FC Driver: Dell Compellent libvirtd (libvirt) 3.6.0 Cinder-volume 2:11.1.0-0ubuntu1~cloud0 I am trying to migrate some volumes from our older Dell Compellent SC storage arrays to new Pure Storage arrays. I can create new volumes on both array’s via cinder. I can create a volume and boot it on the Pure AFA’s. I can live migrate volumes from the Pure array to the Compellent array but not from the Compellent to the Pure. The volume is created and data copied to it, it is then deleted by Cinder after the failure is logged. I receive the following error when going from the Compellent to the Pure Array. I can migrate in both directions if the volume is not attached to an instance. Any ideas? 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Failure rebasing volume /dev/disk/by-path/pci-0000:03:00.0-fc-0x524a937cddfa5902-lun-2 on vda.: libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver Traceback (most recent call last): 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1345, in _swap_volume 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver dev.abort_job(pivot=True) 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 751, in abort_job 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver self._guest._domain.blockJobAbort(self._disk, flags=flags) 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver result = proxy_call(self._autowrap, f, *args, **kwargs) 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv = execute(f, *args, **kwargs) 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver six.reraise(c, e, tb) 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver rv = meth(*args, **kwargs) 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/libvirt.py", line 766, in blockJobAbort 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self) 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver libvirtError: Requested operation is not valid: pivot of disk 'vda' requires an active copy job 2018-05-29 13:08:27.618 3198 ERROR nova.virt.libvirt.driver 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Failed to swap volume bb59023e-d463-44e9-8b1a-a9af495d3d4f for ecc059d0-79f9-402e-aea8-8c99004f221d: VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] Traceback (most recent call last): 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] mountpoint, resize_to) 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] instance) 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] self.force_reraise() 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] six.reraise(self.type_, self.value, self.tb) 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] self._swap_volume(guest, disk_dev, conf, resize_to) 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] raise exception.VolumeRebaseFailed(reason=six.text_type(exc)) 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job 2018-05-29 13:08:28.719 3198 ERROR nova.compute.manager [instance: f834fc03-7e2d-41f4-9307-a1bded3abb29] 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server [req-8ab3bc34-d810-4584-8f31-4dd611ed7b98 385c60230b3f49da930dda4d089eda6b 723aa12337a44f818b6d1e1a59f16e49 - default default] Exception during message handling: VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server function_name, call_dict, binary) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 218, in decorated_function 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 206, in decorated_function 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5255, in swap_volume 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server is_cinder_migration) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5154, in _swap_volume 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server context, new_attachment_id) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5114, in _swap_volume 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server mountpoint, resize_to) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1408, in swap_volume 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server instance) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1404, in swap_volume 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server self._swap_volume(guest, disk_dev, conf, resize_to) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1351, in _swap_volume 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server raise exception.VolumeRebaseFailed(reason=six.text_type(exc)) 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server VolumeRebaseFailed: Volume rebase failed: Requested operation is not valid: pivot of disk 'vda' requires an active copy job 2018-05-29 13:08:30.417 3198 ERROR oslo_messaging.rpc.server Any help would be greatly appreciated. Thanks! — Steve _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org<mailto:openstack@lists.openstack.org> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [https://bowtie.mailbutler.io/tracking/hit/2FC22EFC-C1AD-4242-8330-00BBDA659936/86971503-CAA6-4663-A993-F4396925CE1B/t.gif]
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack