** Also affects: libvirt (Ubuntu) Importance: Undecided Status: New
-- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1673483 Title: libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails) Status in OpenStack Compute (nova): Confirmed Status in libvirt package in Ubuntu: New Bug description: http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron- full-ubuntu-xenial/479c7bf/console.html#_2017-03-16_01_31_33_688563 The test teardown times out waiting for the volume to be detached from the server because the volume detach fails due to a libvirt i/o error during detach: http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron- full-ubuntu- xenial/479c7bf/logs/screen-n-cpu.txt.gz?#_2017-03-16_01_01_32_897 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [req-ac93339e-9eb1-4104-8743-5e5626253ab8 tempest-AttachVolumeShelveTestJSON-116820638 tempest-AttachVolumeShelveTestJSON-116820638] [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Failed to detach volume dc49318a-e51f-4812-97a3-64bf93e522eb from /dev/vdc 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Traceback (most recent call last): 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/opt/stack/new/nova/nova/compute/manager.py", line 4833, in _driver_detach_volume 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] encryption=encryption) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1334, in detach_volume 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] live=live) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 414, in detach_device_with_retry 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] _try_detach_device(conf, persistent, live) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 408, in _try_detach_device 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] raise exception.DeviceNotFound(device=device) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] self.force_reraise() 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] six.reraise(self.type_, self.value, self.tb) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 393, in _try_detach_device 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] self.detach_device(conf, persistent=persistent, live=live) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 450, in detach_device 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] self._domain.detachDeviceFlags(device_xml, flags=flags) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] result = proxy_call(self._autowrap, f, *args, **kwargs) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] rv = execute(f, *args, **kwargs) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] six.reraise(c, e, tb) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] rv = meth(*args, **kwargs) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1192, in detachDeviceFlags 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] libvirtError: End of file while reading data: Input/output error 2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] It's not clear if we're trying the detach twice, and the first time it's OK and then the 2nd fails with the i/o error, which is not in the list of expected libvirt errors we handle in this case. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1673483/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp