[Yahoo-eng-team] [Bug 1429684] Re: Nova and Brick can log each other out of iscsi sessions
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]
** Changed in: nova
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1429684
Title:
Nova and Brick can log each other out of iscsi sessions
Status in Cinder:
Invalid
Status in OpenStack Compute (nova):
Expired
Bug description:
Brick and nova are not synchronized with the same connect_volume lock.
This can cause nova or cinder to logout of an iscsi portal when the
other one is attempting to use it. if nova and cinder are on the same
node.
This may seem like a rare situation but commonly occurs in our CI
system as we perform many operations involving both Nova and Brick
concurrently. Likely when attaching/detaching to an instance while
attaching to the node directly for image operations.
In the below case, cinder logged out of the iscsi session while nova
was retrying rescans attempting to detect the new LUN.
Cinder volume logs:
2015-03-07 17:27:14.288 28940 DEBUG oslo_concurrency.processutils [-]
Running cmd (subprocess): sudo cinder-rootwrap
/etc/cinder/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --logout execute /usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:199
2015-03-07 17:27:14.875 28940 DEBUG oslo_concurrency.processutils [-]
CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf iscsiadm -m node
-T iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815
-p 10.250.119.127:3260 --logout" returned: 0 in 0.588s execute
/usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:225
2015-03-07 17:27:14.876 28940 DEBUG cinder.brick.initiator.connector
[-] iscsiadm ('--logout',): stdout=Logging out of session [sid: 1,
target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
Logout of [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260] successful.
Nova compute logs:
2015-03-07 17:27:12.617 DEBUG nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] iscsiadm ('--rescan',):
stdout=Rescanning session [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
stderr= _run_iscsiadm
/opt/stack/new/nova/nova/virt/libvirt/volume.py:364
2015-03-07 17:27:12.617 WARNING nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] ISCSI volume not yet found at:
vdb. Will rescan & retry. Try number: 0
2015-03-07 17:27:12.618 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] Running cmd (subprocess): sudo
nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --rescan execute /usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:199
2015-03-07 17:27:13.503 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] CMD "sudo nova-rootwrap
/etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --rescan" returned: 0 in 0.885s execute
/usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:225
2015-03-07 17:27:13.504 DEBUG nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] iscsiadm ('--rescan',):
stdout=Rescanning session [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
stderr= _run_iscsiadm
/opt/stack/new/nova/nova/virt/libvirt/volume.py:364
2015-03-07 17:27:14.504 WARNING nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] ISCSI volume not yet found at:
vdb. Will rescan & retry. Try number: 1
2015-03-07 17:27:14.505 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] Running cmd (subprocess): sudo
nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --rescan execute
[Yahoo-eng-team] [Bug 1429684] Re: Nova and Brick can log each other out of iscsi sessions
I don't believe this is an issue anymore, other than the caveat
discussed elsewhere that nova and cinder need to use the same lock_dir
if run on the same host. Please feel free to reopen if this is not the
case.
** Changed in: cinder
Status: Triaged => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1429684
Title:
Nova and Brick can log each other out of iscsi sessions
Status in Cinder:
Invalid
Status in OpenStack Compute (nova):
Incomplete
Bug description:
Brick and nova are not synchronized with the same connect_volume lock.
This can cause nova or cinder to logout of an iscsi portal when the
other one is attempting to use it. if nova and cinder are on the same
node.
This may seem like a rare situation but commonly occurs in our CI
system as we perform many operations involving both Nova and Brick
concurrently. Likely when attaching/detaching to an instance while
attaching to the node directly for image operations.
In the below case, cinder logged out of the iscsi session while nova
was retrying rescans attempting to detect the new LUN.
Cinder volume logs:
2015-03-07 17:27:14.288 28940 DEBUG oslo_concurrency.processutils [-]
Running cmd (subprocess): sudo cinder-rootwrap
/etc/cinder/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --logout execute /usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:199
2015-03-07 17:27:14.875 28940 DEBUG oslo_concurrency.processutils [-]
CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf iscsiadm -m node
-T iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815
-p 10.250.119.127:3260 --logout" returned: 0 in 0.588s execute
/usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:225
2015-03-07 17:27:14.876 28940 DEBUG cinder.brick.initiator.connector
[-] iscsiadm ('--logout',): stdout=Logging out of session [sid: 1,
target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
Logout of [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260] successful.
Nova compute logs:
2015-03-07 17:27:12.617 DEBUG nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] iscsiadm ('--rescan',):
stdout=Rescanning session [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
stderr= _run_iscsiadm
/opt/stack/new/nova/nova/virt/libvirt/volume.py:364
2015-03-07 17:27:12.617 WARNING nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] ISCSI volume not yet found at:
vdb. Will rescan & retry. Try number: 0
2015-03-07 17:27:12.618 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] Running cmd (subprocess): sudo
nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --rescan execute /usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:199
2015-03-07 17:27:13.503 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] CMD "sudo nova-rootwrap
/etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --rescan" returned: 0 in 0.885s execute
/usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:225
2015-03-07 17:27:13.504 DEBUG nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] iscsiadm ('--rescan',):
stdout=Rescanning session [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
stderr= _run_iscsiadm
/opt/stack/new/nova/nova/virt/libvirt/volume.py:364
2015-03-07 17:27:14.504 WARNING nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] ISCSI volume not yet found at:
vdb. Will rescan & retry. Try number: 1
2015-03-07 17:27:14.505 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] Running cmd (subprocess): sudo
nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T
[Yahoo-eng-team] [Bug 1429684] Re: Nova and Brick can log each other out of iscsi sessions
Nova does not use brick at the moment, right? What needs to change in
Nova?
** Also affects: cinder
Importance: Undecided
Status: New
** Changed in: cinder
Status: New = Incomplete
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1429684
Title:
Nova and Brick can log each other out of iscsi sessions
Status in Cinder:
Incomplete
Status in OpenStack Compute (Nova):
New
Bug description:
Brick and nova are not synchronized with the same connect_volume lock.
This can cause nova or cinder to logout of an iscsi portal when the
other one is attempting to use it. if nova and cinder are on the same
node.
This may seem like a rare situation but commonly occurs in our CI
system as we perform many operations involving both Nova and Brick
concurrently. Likely when attaching/detaching to an instance while
attaching to the node directly for image operations.
In the below case, cinder logged out of the iscsi session while nova
was retrying rescans attempting to detect the new LUN.
Cinder volume logs:
2015-03-07 17:27:14.288 28940 DEBUG oslo_concurrency.processutils [-]
Running cmd (subprocess): sudo cinder-rootwrap
/etc/cinder/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --logout execute /usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:199
2015-03-07 17:27:14.875 28940 DEBUG oslo_concurrency.processutils [-]
CMD sudo cinder-rootwrap /etc/cinder/rootwrap.conf iscsiadm -m node
-T iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815
-p 10.250.119.127:3260 --logout returned: 0 in 0.588s execute
/usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:225
2015-03-07 17:27:14.876 28940 DEBUG cinder.brick.initiator.connector
[-] iscsiadm ('--logout',): stdout=Logging out of session [sid: 1,
target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
Logout of [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260] successful.
Nova compute logs:
2015-03-07 17:27:12.617 DEBUG nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] iscsiadm ('--rescan',):
stdout=Rescanning session [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
stderr= _run_iscsiadm
/opt/stack/new/nova/nova/virt/libvirt/volume.py:364
2015-03-07 17:27:12.617 WARNING nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] ISCSI volume not yet found at:
vdb. Will rescan retry. Try number: 0
2015-03-07 17:27:12.618 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] Running cmd (subprocess): sudo
nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --rescan execute /usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:199
2015-03-07 17:27:13.503 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] CMD sudo nova-rootwrap
/etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
10.250.119.127:3260 --rescan returned: 0 in 0.885s execute
/usr/local/lib/python2.7/dist-
packages/oslo_concurrency/processutils.py:225
2015-03-07 17:27:13.504 DEBUG nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] iscsiadm ('--rescan',):
stdout=Rescanning session [sid: 1, target:
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815,
portal: 10.250.119.127,3260]
stderr= _run_iscsiadm
/opt/stack/new/nova/nova/virt/libvirt/volume.py:364
2015-03-07 17:27:14.504 WARNING nova.virt.libvirt.volume [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] ISCSI volume not yet found at:
vdb. Will rescan retry. Try number: 1
2015-03-07 17:27:14.505 DEBUG oslo_concurrency.processutils [req-
55f33c70-ec85-4041-aaf6-205f74abf979
VolumesV1SnapshotTestJSON-1966398854
VolumesV1SnapshotTestJSON-1188982339] Running cmd (subprocess): sudo
nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.1992-08.com.netapp:sn.60a8eb0cc4bb11e4b041123478563412:vs.40815 -p
