** Description changed: + [Impact] + iSCSI volume detach does not correctly remove the multipath device - descriptors + descriptors. + + The multipath devices are left on the compute node and multipath tools + will occaisionally send IOs to known multipath devices. + + [Test Case] tested environment: nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300. - I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully. 3rd volume also successfully detached but ended up with failed multipaths. + I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully. 3rd volume also successfully detached but ended up with failed multipaths. Here is the terminal log for last volume detach. openstack@W1DEV103:~/devstack$ cinder list +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | - ID - | Status | Name | Size | Volume Type | Bootable | - Attached to - | + ID + | Status | Name | Size | Volume Type | Bootable | + Attached to + | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 | - None - | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e | + None + | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l Fri Sep 19 21:38:13 JST 2014 360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=-1 status=active | |- 4:0:0:42 sdb 8:16 active undef running | |- 5:0:0:42 sdd 8:48 active undef running | |- 6:0:0:42 sdf 8:80 active undef running | `- 7:0:0:42 sdh 8:112 active undef running `-+- policy='round-robin 0' prio=-1 status=enabled |- 11:0:0:42 sdp 8:240 active undef running |- 8:0:0:42 sdj 8:144 active undef running |- 9:0:0:42 sdl 8:176 active undef running `- 10:0:0:42 sdn 8:208 active undef running openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session Fri Sep 19 21:38:19 JST 2014 tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7 tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7 tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4 tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5 tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6 tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4 tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5 tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6 openstack@W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e 56a63288-5cc0-4f5c-9197-cde731172dd8 openstack@W1DEV103:~/devstack$ openstack@W1DEV103:~/devstack$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | - ID - | Status | Name | Size | Volume Type | Bootable | - Attached to - | + ID + | Status | Name | Size | Volume Type | Bootable | + Attached to + | +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 | - None - | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e| + None + | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e| +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ openstack@W1DEV103:~/devstack$ openstack@W1DEV103:~/devstack$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | - ID - | Status | Name | Size | Volume Type | Bootable | Attached to | + ID + | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 | - None - | false | + None + | false | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l Fri Sep 19 21:39:23 JST 2014 360060160cf0036002d1475f6e73fe411 dm-2 , size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=-1 status=active | |- #:#:#:# - #:# active undef running | |- #:#:#:# - #:# active undef running | |- #:#:#:# - #:# active undef running | `- #:#:#:# - #:# active undef running `-+- policy='round-robin 0' prio=-1 status=enabled |- #:#:#:# - #:# active undef running |- #:#:#:# - #:# active undef running |- #:#:#:# - #:# active undef running `- #:#:#:# - #:# active undef running openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session Fri Sep 19 21:39:27 JST 2014 iscsiadm: No active sessions. Then I manually removed the multipaths, openstack@W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411 openstack@W1CN103:/etc/iscsi$ sudo multipath -l openstack@W1CN103:/etc/iscsi$ - I think the problem is in, + I think the problem is in, virt/libvirt/volume.py:LibvirtISCSIVolumeDriver - def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device): + def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device): End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return. - self._remove_multipath_device_descriptor(multipath_device) - return - Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one), this method return without calling _remove_multipath_device_descriptor due to following if statement. - - if not in_use: - # disconnect if no other multipath devices with same iqn - self._disconnect_mpath(iscsi_properties, ips_iqns) - return + self._remove_multipath_device_descriptor(multipath_device) + return + Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one), this method return without calling _remove_multipath_device_descriptor due to following if statement. + + if not in_use: + # disconnect if no other multipath devices with same iqn + self._disconnect_mpath(iscsi_properties, ips_iqns) + return It just disconnect them but not remove them. One of the reasons why we have to remove them is, https://bugs.launchpad.net/nova/+bug/1223975 IMO, we should call _remove_multipath_device_descriptor in above if statement before return. + + [Regression Potential] + + - Low: the change is relatively minor and allows the code to also remove + the device when the last path is removed. Should a regression occur, it + should be limited to iscsi-multipath device detachment, which is a small + portion of installations.
** Patch added: "kilo cloud archive debdiff" https://bugs.launchpad.net/nova/+bug/1374999/+attachment/4623679/+files/kilo-lp1374999.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1374999 Title: iSCSI volume detach does not correctly remove the multipath device descriptors To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1374999/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
