** Also affects: cinder Importance: Undecided Status: New ** Also affects: cinder/kilo Importance: Undecided Status: New
** Changed in: cinder/kilo Milestone: None => 2015.1.3 ** Changed in: cinder/kilo Status: New => Fix Released ** Changed in: cinder Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1382440 Title: Detaching multipath volume doesn't work properly when using different targets with same portal for each multipath device Status in Cinder: Invalid Status in Cinder kilo series: Fix Released Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) icehouse series: Fix Released Status in OpenStack Compute (nova) juno series: Fix Released Status in os-brick: Fix Released Bug description: Overview: On Icehouse(2014.1.2) with "iscsi_use_multipath=true", detaching iSCSI multipath volume doesn't work properly. When we use different targets(IQNs) associated with same portal for each different multipath device, all of the targets will be deleted via disconnect_volume(). This problem is not yet fixed in upstream. However, the attached patch fixes this problem. Steps to Reproduce: We can easily reproduce this issue without any special storage system in the following Steps: 1. configure "iscsi_use_multipath=True" in nova.conf on compute node. 2. configure "volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver" in cinder.conf on cinder node. 2. create an instance. 3. create 3 volumes and attach them to the instance. 4. detach one of these volumes. 5. check "multipath -ll" and "iscsiadm --mode session". Detail: This problem was introduced with the following patch which modified attaching and detaching volume operations for different targets associated with different portals for the same multipath device. commit 429ac4dedd617f8c1f7c88dd8ece6b7d2f2accd0 Author: Xing Yang <xing.y...@emc.com> Date: Date: Mon Jan 6 17:27:28 2014 -0500 Fixed a problem in iSCSI multipath We found out that: > # Do a discovery to find all targets. > # Targets for multiple paths for the same multipath device > # may not be the same. > out = self._run_iscsiadm_bare(['-m', > 'discovery', > '-t', > 'sendtargets', > '-p', > iscsi_properties['target_portal']], > check_exit_code=[0, 255])[0] \ > or "" > > ips_iqns = self._get_target_portals_from_iscsiadm_output(out) ... > # If no other multipath device attached has the same iqn > # as the current device > if not in_use: > # disconnect if no other multipath devices with same iqn > self._disconnect_mpath(iscsi_properties, ips_iqns) > return > elif multipath_device not in devices: > # delete the devices associated w/ the unused multipath > self._delete_mpath(iscsi_properties, multipath_device, ips_iqns) When we use different targets(IQNs) associated with same portal for each different multipath device, the ips_iqns has all targets in compute node from the result of "iscsiadm -m discovery -t sendtargets -p <the same portal>". Then, the _delete_mpath() deletes all of the targets in the ips_iqns via /sys/block/sdX/device/delete. For example, we create an instance and attach 3 volumes to the instance: # iscsiadm --mode session tcp: [17] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-5c526ffa-ba88-4fe2-a570-9e35c4880d12 tcp: [18] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b4495e7e-b611-4406-8cce-4681ac1e36de tcp: [19] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b2c01f6a-5723-40e7-9f21-f6b728021b0e # multipath -ll 33000000300000001 dm-7 IET,VIRTUAL-DISK size=4.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 23:0:0:1 sdd 8:48 active ready running 33000000100000001 dm-5 IET,VIRTUAL-DISK size=2.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 21:0:0:1 sdb 8:16 active ready running 33000000200000001 dm-6 IET,VIRTUAL-DISK size=3.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 22:0:0:1 sdc 8:32 active ready running Then we detach one of these volumes: # nova volume-detach 95f959cd-d180-4063-ae03-9d21dbd7cc50 5c526ffa- ba88-4fe2-a570-9e35c4880d12 As a result of detaching the volume, the compute node remains 3 iSCSI sessions and the instance fails to access the attached multipath devices: # iscsiadm --mode session tcp: [17] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-5c526ffa-ba88-4fe2-a570-9e35c4880d12 tcp: [18] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b4495e7e-b611-4406-8cce-4681ac1e36de tcp: [19] 192.168.0.55:3260,1 iqn.2010-10.org.openstack:volume-b2c01f6a-5723-40e7-9f21-f6b728021b0e # multipath -ll 33000000300000001 dm-7 , size=4.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=enabled `- #:#:#:# - #:# failed faulty running 33000000200000001 dm-6 , size=3.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=enabled `- #:#:#:# - #:# failed faulty running To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1382440/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp