Public bug reported: [Environment]
OpenStack Kilo Trusty 14.04.4 [Description] if the attached multipath devices doesn't have same iqn like regular lvm+iscsi backend, in_use will be false. In that case,_disconnect_volume_multipath_iscsi() returns without calling _remove_multipath_device_descriptor(). [Reproduction] 1) Enable cinder LVM ISCSI on /etc/cinder/cinder.conf volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver 2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute nodes: iscsi_use_multipath = True 3) Create 3 cinder volumes $ cinder create 1 $ cinder create 1 $ cinder create 1 $ cinder list ubuntu@niedbalski2-bastion:~/specs/1374999$ cinder list +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ | 10844be6-8f86-414f-a10e-e1a31e2ba6e7 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | | 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | | 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ 4) Attach them to nova $ nova volume-attach instance_id 10844be6-8f86-414f-a10e-e1a31e2ba6e7 $ nova volume-attach instance_id 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 $ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f 5) Check on the nova-compute unit for the current multipath/session status tcp: [1] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-10844be6-8f86-414f-a10e-e1a31e2ba6e7 tcp: [2] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-1648d24c-0d65-4377-9fa5-6d3aeb8b1291 tcp: [3] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f Multipath: root@juju-1374999-machine-10:/home/ubuntu# multipath -ll 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 8:0:0:1 sdg 8:96 active ready running 33000000200000001 dm-1 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 9:0:0:1 sda 8:0 active ready running 6) Detach the current volumes. First. ubuntu@niedbalski2-bastion:~/specs/1374999$ nova volume-detach b0a14447-5740-408a-b96f-a1e904b229e5 10844be6-8f86-414f-a10e- e1a31e2ba6e7 ubuntu@niedbalski2-bastion:~/specs/1374999$ juju ssh 10 "sudo multipath -ll" 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 9:0:0:1 sda 8:0 active ready running Second raises the faulty state 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running Third, raises the faulty state also sudo: unable to resolve host juju-1374999-machine-10 33000000300000001 dm-2 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running Manual fix. root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000300000001 root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000100000001 root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000200000001 ** Affects: nova Importance: Undecided Assignee: Jorge Niedbalski (niedbalski) Status: In Progress ** Affects: ubuntu Importance: Undecided Status: New ** Changed in: nova Status: New => In Progress ** Changed in: nova Assignee: (unassigned) => Jorge Niedbalski (niedbalski) ** Description changed: [Environment] OpenStack Kilo Trusty 14.04.4 [Description] if the attached multipath devices doesn't have same iqn like regular lvm+iscsi backend, in_use will be false. In that case,_disconnect_volume_multipath_iscsi() returns without calling _remove_multipath_device_descriptor(). [Reproduction] 1) Enable cinder LVM ISCSI on /etc/cinder/cinder.conf volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver - - 2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute nodes: + 2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute + nodes: iscsi_use_multipath = True - - 3) Create 3 cinder volumes + 3) Create 3 cinder volumes $ cinder create 1 $ cinder create 1 $ cinder create 1 $ cinder list ubuntu@niedbalski2-bastion:~/specs/1374999$ cinder list +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ | 10844be6-8f86-414f-a10e-e1a31e2ba6e7 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | | 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | | 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ - 4) Attach them to nova $ nova volume-attach instance_id 10844be6-8f86-414f-a10e-e1a31e2ba6e7 $ nova volume-attach instance_id 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 - $ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f + $ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f 5) Check on the nova-compute unit for the current multipath/session status tcp: [1] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-10844be6-8f86-414f-a10e-e1a31e2ba6e7 tcp: [2] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-1648d24c-0d65-4377-9fa5-6d3aeb8b1291 tcp: [3] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f Multipath: root@juju-1374999-machine-10:/home/ubuntu# multipath -ll 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active - `- 10:0:0:1 sdb 8:16 active ready running + `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active - `- 8:0:0:1 sdg 8:96 active ready running + `- 8:0:0:1 sdg 8:96 active ready running 33000000200000001 dm-1 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active - `- 9:0:0:1 sda 8:0 active ready running + `- 9:0:0:1 sda 8:0 active ready running 6) Detach the current volumes. First. ubuntu@niedbalski2-bastion:~/specs/1374999$ nova volume-detach b0a14447-5740-408a-b96f-a1e904b229e5 10844be6-8f86-414f-a10e- e1a31e2ba6e7 - ubuntu@niedbalski2-bastion:~/specs/1374999$ juju ssh 10 "sudo multipath -ll" 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active - `- 10:0:0:1 sdb 8:16 active ready running + `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active - `- #:#:#:# - #:# active faulty running + `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active - `- 9:0:0:1 sda 8:0 active ready running + `- 9:0:0:1 sda 8:0 active ready running Second raises the faulty state 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active - `- 10:0:0:1 sdb 8:16 active ready running + `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active - `- #:#:#:# - #:# active faulty running + `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active - `- #:#:#:# - #:# active faulty running + `- #:#:#:# - #:# active faulty running Third, raises the faulty state also sudo: unable to resolve host juju-1374999-machine-10 33000000300000001 dm-2 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active - `- #:#:#:# - #:# active faulty running + `- #:#:#:# - #:# active faulty running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active - `- #:#:#:# - #:# active faulty running + `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active - `- #:#:#:# - #:# active faulty running + `- #:#:#:# - #:# active faulty running + + + Manual fix. + + root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000300000001 + root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000100000001 + root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000200000001 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1452032 Title: Device descriptor not removed with different iqn and multipath enabled. Status in OpenStack Compute (Nova): In Progress Status in Ubuntu: New Bug description: [Environment] OpenStack Kilo Trusty 14.04.4 [Description] if the attached multipath devices doesn't have same iqn like regular lvm+iscsi backend, in_use will be false. In that case,_disconnect_volume_multipath_iscsi() returns without calling _remove_multipath_device_descriptor(). [Reproduction] 1) Enable cinder LVM ISCSI on /etc/cinder/cinder.conf volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver 2) Enable iscsi_use_multipath on /etc/nova/nova.conf on your compute nodes: iscsi_use_multipath = True 3) Create 3 cinder volumes $ cinder create 1 $ cinder create 1 $ cinder create 1 $ cinder list ubuntu@niedbalski2-bastion:~/specs/1374999$ cinder list +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ | 10844be6-8f86-414f-a10e-e1a31e2ba6e7 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | | 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | | 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f | in-use | None | 1 | None | false | b0a14447-5740-408a-b96f-a1e904b229e5 | +--------------------------------------+----------+--------------+------+-------------+----------+--------------------------------------+ 4) Attach them to nova $ nova volume-attach instance_id 10844be6-8f86-414f-a10e-e1a31e2ba6e7 $ nova volume-attach instance_id 1648d24c-0d65-4377-9fa5-6d3aeb8b1291 $ nova volume-attach instance_id 53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f 5) Check on the nova-compute unit for the current multipath/session status tcp: [1] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-10844be6-8f86-414f-a10e-e1a31e2ba6e7 tcp: [2] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-1648d24c-0d65-4377-9fa5-6d3aeb8b1291 tcp: [3] 10.5.1.43:3260,1 iqn.2010-10.org.openstack:volume-53d6bb4e-2ca2-45ab-9ed1-887b1df2ff8f Multipath: root@juju-1374999-machine-10:/home/ubuntu# multipath -ll 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 8:0:0:1 sdg 8:96 active ready running 33000000200000001 dm-1 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 9:0:0:1 sda 8:0 active ready running 6) Detach the current volumes. First. ubuntu@niedbalski2-bastion:~/specs/1374999$ nova volume-detach b0a14447-5740-408a-b96f-a1e904b229e5 10844be6-8f86-414f-a10e- e1a31e2ba6e7 ubuntu@niedbalski2-bastion:~/specs/1374999$ juju ssh 10 "sudo multipath -ll" 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 9:0:0:1 sda 8:0 active ready running Second raises the faulty state 33000000300000001 dm-2 IET,VIRTUAL-DISK size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active `- 10:0:0:1 sdb 8:16 active ready running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running Third, raises the faulty state also sudo: unable to resolve host juju-1374999-machine-10 33000000300000001 dm-2 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000100000001 dm-0 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running 33000000200000001 dm-1 , size=1.0G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=0 status=active `- #:#:#:# - #:# active faulty running Manual fix. root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000300000001 root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000100000001 root@juju-1374999-machine-10:/home/ubuntu# multipath -F 33000000200000001 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1452032/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp