** Also affects: os-brick
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1530214
Title:
Tempest failures due to iSCSI DB failure on compute node
Status in OpenStack Compute (nova):
New
Status in os-brick:
New
Bug description:
Noticed a couple of these today in the SolidFire CI system. These are
initiator side errors in Nova. Excerpt from log is below, but additional logs
can also be viewed here:
http://54.164.167.86/solidfire-ci-logs/refs-changes-67-244867-12/logs/screen-n-cpu.log.txt
2015-12-30 20:53:50.829 ERROR nova.virt.block_device
[req-0f1602ac-0604-4051-8d65-81ede0976dc5
tempest-DeleteServersTestJSON-1052804341
tempest-DeleteServersTestJSON-308638552] [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] Driver failed to attach volume
23049317-f6c2-40b2-9306-0b838848f911 at /dev/vdb
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] Traceback (most recent call last):
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/nova/nova/virt/block_device.py", line 288, in attach
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] device_type=self['device_type'],
encryption=encryption)
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1114, in attach_volume
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] self._connect_volume(connection_info,
disk_info)
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 1064, in _connect_volume
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c]
vol_driver.connect_volume(connection_info, disk_info)
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/nova/nova/virt/libvirt/volume/iscsi.py", line 84, in connect_volume
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] device_info =
self.connector.connect_volume(connection_info['data'])
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line
271, in inner
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] return f(*args, **kwargs)
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/os-brick/os_brick/initiator/connector.py", line 763, in
connect_volume
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] connection_properties)
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/os-brick/os_brick/initiator/connector.py", line 591, in
_get_potential_volume_paths
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] if
self._connect_to_iscsi_portal(props):
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/os-brick/os_brick/initiator/connector.py", line 1044, in
_connect_to_iscsi_portal
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c]
self._run_iscsiadm(connection_properties, ())
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/opt/stack/os-brick/os_brick/initiator/connector.py", line 948, in
_run_iscsiadm
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] delay_on_retry=delay_on_retry)
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] File
"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line
312, in execute
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] cmd=sanitized_cmd)
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] ProcessExecutionError: Unexpected error
while running command.
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] Command: sudo nova-rootwrap
/etc/nova/rootwrap.conf iscsiadm -m node -T
iqn.2010-01.com.solidfire:l74j.uuid-23049317-f6c2-40b2-9306-0b838848f911.374375
-p 10.10.64.3:3260
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] Exit code: 6
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] Stdout: u''
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c] Stderr: u'iscsiadm: Could not execute
operation on all records: encountered iSCSI database failure\n'
2015-12-30 20:53:50.829 13456 ERROR nova.virt.block_device [instance:
d1526309-a49f-46b3-9a29-7d7ae285be9c]
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1530214/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
