Public bug reported:
Hello!
The issue is annoying me and my users for long time, so I decided to report the
problem with detaching volumes from instances. When an exception occurs, the
detaching is stuck in detaching state without retrying detach procedure.
For example, below is the exception:
2020-07-23 09:36:37.964 37220 ERROR nova.volume.cinder
[req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c
b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Delete attachment failed
for attachment 35653c6c-12c5-4d10-a3d4-508b45e2b1d6. Error: Gat
eway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException:
Gateway Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
[req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c
b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Exception during message
handling: cinderclient.exceptions.ClientException: Gatew
ay Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in
dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in
_do_dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 79, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
function_name, call_dict, binary, tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
f(self, context, *args, **kw)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1372, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 219, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6198, in
detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
do_detach_volume(context, volume_id, instance, attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in
inner
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6196, in
do_detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id=attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6147, in
_detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 461, in detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id, destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 442, in
_do_detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
volume_api.attachment_delete(context, self['attachment_id'])
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 397, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
method(self, ctx, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 448, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
method(self, ctx, attachment_id, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 889, in
attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server 'code':
getattr(ex, 'code', None)})
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 882, in
attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/api_versions.py", line 423, in
substitution
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
method.func(obj, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/v3/attachments.py", line 44, in
delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._delete("/attachments/%s" % base.getid(attachment))
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/base.py", line 309, in _delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server resp, body =
self.api.client.delete(url)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 223, in delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._cs_request(url, 'DELETE', **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 205, in
_cs_request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self.request(url, method, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise
exceptions.from_response(resp, body)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
After that the volume can't be detached without reset-state to "in-use".
I suggest to include some timeout or retrying in attach/detach
procedures.
** Affects: nova
Importance: Undecided
Status: New
** Tags: cinder nova-compute
** Description changed:
Hello!
- The issue is annoying me and my users for long time, so I decided to reported
the problem with detaching volumes from instances. When exception occurs,
detaching is stuck in detaching state without retrying detach procedure.
+ The issue is annoying me and my users for long time, so I decided to report
the problem with detaching volumes from instances. When an exception occurs,
the detaching is stuck in detaching state without retrying detach procedure.
For example, below is the exception:
2020-07-23 09:36:37.964 37220 ERROR nova.volume.cinder
[req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c
b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Delete attachment failed
for attachment 35653c6c-12c5-4d10-a3d4-508b45e2b1d6. Error: Gat
eway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException:
Gateway Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
[req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c
b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Exception during message
handling: cinderclient.exceptions.ClientException: Gatew
ay Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in
dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in
_do_dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 79, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
function_name, call_dict, binary, tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
f(self, context, *args, **kw)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1372, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 219, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6198, in
detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
do_detach_volume(context, volume_id, instance, attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in
inner
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6196, in
do_detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id=attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6147, in
_detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 461, in detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id, destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 442, in
_do_detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
volume_api.attachment_delete(context, self['attachment_id'])
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 397, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
method(self, ctx, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 448, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
method(self, ctx, attachment_id, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 889, in
attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server 'code':
getattr(ex, 'code', None)})
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 882, in
attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/api_versions.py", line 423, in
substitution
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
method.func(obj, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/v3/attachments.py", line 44, in
delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._delete("/attachments/%s" % base.getid(attachment))
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/base.py", line 309, in _delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server resp, body
= self.api.client.delete(url)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 223, in delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._cs_request(url, 'DELETE', **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 205, in
_cs_request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self.request(url, method, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise
exceptions.from_response(resp, body)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
- 2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
+ 2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
-
- After that the volume can't be detached without reset-state to "in-use". I
suggest to include some timeout or retrying in attach/detach procedures.
+ After that the volume can't be detached without reset-state to "in-use".
+ I suggest to include some timeout or retrying in attach/detach
+ procedures.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1888665
Title:
Cinder volume stuck in detaching on error exception
Status in OpenStack Compute (nova):
New
Bug description:
Hello!
The issue is annoying me and my users for long time, so I decided to report
the problem with detaching volumes from instances. When an exception occurs,
the detaching is stuck in detaching state without retrying detach procedure.
For example, below is the exception:
2020-07-23 09:36:37.964 37220 ERROR nova.volume.cinder
[req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c
b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Delete attachment failed
for attachment 35653c6c-12c5-4d10-a3d4-508b45e2b1d6. Error: Gat
eway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException:
Gateway Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
[req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c
b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Exception during message
handling: cinderclient.exceptions.ClientException: Gatew
ay Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server Traceback (most
recent call last):
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in
_process_incoming
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in
dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in
_do_dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server result =
func(ctxt, **new_args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 79, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
function_name, call_dict, binary, tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
f(self, context, *args, **kw)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1372, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 219, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
kwargs['instance'], e, sys.exc_info())
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in
decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6198, in
detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
do_detach_volume(context, volume_id, instance, attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in
inner
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
f(*args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6196, in
do_detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id=attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6147, in
_detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 461, in detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id, destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 442, in
_do_detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
volume_api.attachment_delete(context, self['attachment_id'])
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 397, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
method(self, ctx, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 448, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server res =
method(self, ctx, attachment_id, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 889, in
attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server 'code':
getattr(ex, 'code', None)})
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 882, in
attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/api_versions.py", line 423, in
substitution
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
method.func(obj, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/v3/attachments.py", line 44, in
delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._delete("/attachments/%s" % base.getid(attachment))
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/base.py", line 309, in _delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server resp, body
= self.api.client.delete(url)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 223, in delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self._cs_request(url, 'DELETE', **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 205, in
_cs_request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server return
self.request(url, method, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server File
"/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server raise
exceptions.from_response(resp, body)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
After that the volume can't be detached without reset-state to "in-
use". I suggest to include some timeout or retrying in attach/detach
procedures.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1888665/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
