** Also affects: nova/rocky
Importance: Undecided
Status: New
** Changed in: nova/queens
Status: New => Triaged
** Changed in: nova/rocky
Status: New => Triaged
** Changed in: nova
Assignee: Stephen Finucane (stephenfinucane) => Lee Yarwood (lyarwood)
** Changed in: nova/queens
Importance: Undecided => Medium
** Changed in: nova/rocky
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1784353
Title:
Rescheduled boot from volume instances fail due to the premature
removal of their attachments
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) queens series:
Triaged
Status in OpenStack Compute (nova) rocky series:
Triaged
Bug description:
Description
===========
This is caused by the cleanup code within the compute layer
(_shutdown_instance) removing all volume attachments associated with an
instance with no attempt being made to recreate these ahead of the instance
being rescheduled.
Steps to reproduce
==================
- Attempt to boot an instance with volumes attached.
- Ensure spawn() fails, for example by stopping the l2 network agent services
on the compute host.
Expected result
===============
The instance is reschedule to another compute host and boots correctly.
Actual result
=============
The instance fails to boot on all hosts that is rescheduled to due to a
missing volume attachment.
Environment
===========
1. Exact version of OpenStack you are running. See the following
list for all releases: http://docs.openstack.org/releases/
bf497cc47497d3a5603bf60de652054ac5ae1993
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
Libvirt + KVM, however this shouldn't matter.
3. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
N/A
4. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==============
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] Traceback (most recent call last):
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1579, in
_prep_block_device
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1]
wait_func=self._await_block_device_map_created)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 837, in
attach_block_devices
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] _log_and_attach(device)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 834, in
_log_and_attach
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] bdm.attach(*attach_args,
**attach_kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in
wrapped
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] ret_val = method(obj, context, *args,
**kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 617, in
attach
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] virt_driver, do_driver_attach)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in
inner
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] return f(*args, **kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 614, in
_do_locked_attach
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] self._do_attach(*args, **_kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 599, in
_do_attach
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] do_driver_attach)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 513, in
_volume_attach
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1]
self['mount_device'])['connection_info']
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] res = method(self, ctx, *args,
**kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 418, in wrapper
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] attachment_id=attachment_id))
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 450, in _reraise
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] six.reraise(type(desired_exc),
desired_exc, sys.exc_info()[2])
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] res = method(self, ctx,
attachment_id, *args, **kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 824, in
attachment_update
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] 'code': getattr(ex, 'code', None)})
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] self.force_reraise()
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] six.reraise(self.type_, self.value,
self.tb)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 814, in
attachment_update
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] attachment_id, _connector)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 67, in
update
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] resp = self._update('/attachments/%s'
% id, body)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/cinderclient/base.py", line 344, in _update
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] resp, body = self.api.client.put(url,
body=body, **kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 206, in put
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] return self._cs_request(url, 'PUT',
**kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 191, in
_cs_request
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] return self.request(url, method,
**kwargs)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] File
"/usr/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] raise exceptions.from_response(resp,
body)
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1] VolumeAttachmentNotFound: Volume
attachment 11 [details]d518a9-16d4-4ccb-9487-ec2b35834945 could not be found.
2018-07-04 15:19:43.191 1 ERROR nova.compute.manager [instance:
d48c9894-2ba2-4752-bae5-36c437933ff1]
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1784353/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
