** Also affects: nova/rocky
   Importance: Undecided
       Status: New

** Changed in: nova/queens
       Status: New => Triaged

** Changed in: nova/rocky
       Status: New => Triaged

** Changed in: nova
     Assignee: Stephen Finucane (stephenfinucane) => Lee Yarwood (lyarwood)

** Changed in: nova/queens
   Importance: Undecided => Medium

** Changed in: nova/rocky
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1784353

Title:
  Rescheduled boot from volume instances fail due to the premature
  removal of their attachments

Status in OpenStack Compute (nova):
  In Progress
Status in OpenStack Compute (nova) queens series:
  Triaged
Status in OpenStack Compute (nova) rocky series:
  Triaged

Bug description:
  Description
  ===========
  This is caused by the cleanup code within the compute layer 
(_shutdown_instance) removing all volume attachments associated with an 
instance with no attempt being made to recreate these ahead of the instance 
being rescheduled.

  Steps to reproduce
  ==================
  - Attempt to boot an instance with volumes attached.
  - Ensure spawn() fails, for example by stopping the l2 network agent services 
on the compute host.

  Expected result
  ===============
  The instance is reschedule to another compute host and boots correctly.

  Actual result
  =============
  The instance fails to boot on all hosts that is rescheduled to due to a 
missing volume attachment.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     bf497cc47497d3a5603bf60de652054ac5ae1993

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM, however this shouldn't matter.

  3. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  4. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1] Traceback (most recent call last):  
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1579, in 
_prep_block_device
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     
wait_func=self._await_block_device_map_created)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 837, in 
attach_block_devices
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     _log_and_attach(device)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 834, in 
_log_and_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     bdm.attach(*attach_args, 
**attach_kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in 
wrapped
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     ret_val = method(obj, context, *args, 
**kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 617, in 
attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     virt_driver, do_driver_attach)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in 
inner
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     return f(*args, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 614, in 
_do_locked_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     self._do_attach(*args, **_kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 599, in 
_do_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     do_driver_attach)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 513, in 
_volume_attach
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     
self['mount_device'])['connection_info']
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     res = method(self, ctx, *args, 
**kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 418, in wrapper
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     attachment_id=attachment_id))
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 450, in _reraise
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     six.reraise(type(desired_exc), 
desired_exc, sys.exc_info()[2])
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     res = method(self, ctx, 
attachment_id, *args, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 824, in 
attachment_update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     'code': getattr(ex, 'code', None)})
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     self.force_reraise()
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in 
force_reraise
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     six.reraise(self.type_, self.value, 
self.tb)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 814, in 
attachment_update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     attachment_id, _connector)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 67, in 
update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     resp = self._update('/attachments/%s' 
% id, body)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/cinderclient/base.py", line 344, in _update
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     resp, body = self.api.client.put(url, 
body=body, **kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 206, in put
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     return self._cs_request(url, 'PUT', 
**kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in 
_cs_request
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     return self.request(url, method, 
**kwargs)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]   File 
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]     raise exceptions.from_response(resp, 
body)
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1] VolumeAttachmentNotFound: Volume 
attachment 11 [details]d518a9-16d4-4ccb-9487-ec2b35834945 could not be found.
      2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance: 
d48c9894-2ba2-4752-bae5-36c437933ff1]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1784353/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to