** Changed in: glance
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1558398
Title:
instance snapshot with large root disk fails when glance api using ssl
Status in Glance:
Invalid
Bug description:
in Kilo, using glance ssl with or without a load-balancer in between
will cause instance snapshot of an instance with a large root disk
(>~60 GB) to fail. Smaller root disks work fine. The flavor test was
specifically 4 vcpu, 16GB ram, 160GB root disk, 0 ephemeral and 0
swap. This is when using local file storage for glance and you see
pretty much 64GB worth of data hit the glance server disk before the
file disappears off the server and the connection has issues.
Glance will only say that the client closed the connection before all the
data could be sent.
WARNING glance.api.v1.upload_utils [req-a5279f16-1be8-4cbf-8625-491e25bcf5c7
766dcc03c6e0454296b449105c71db8c 1a63bbf53e8949e7928ac9e2381a0c04 - - -] Client
disconnected before sending all data to backend
This occurs when trying through horizon, openstack client, or even a
curl to the glance api. If you use glance purely with http this issue
does not occur.
When you use a load-balancer in between for ssl, (nginx in particular in my
case) it will show the below error when the connection dies. Note this setup
had glance endpoints ssl and nova doing https for glance to loadbalancer, but
backend glance was http:
2016/03/15 16:28:23 [info] 23#23: *4346 SSL_read() failed (SSL:
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record
mac) while sending request to upstream, client: 10.240.118.7, server:
vip-cream-1.lss.emc.com, request: "PUT
/v1/images/5d4d9e36-1d66-45da-8a4e-0c1aede75cd7 HTTP/1.1", upstream:
"http://10.240.118.5:9292/v1/images/5d4d9e36-1d66-45da-8a4e-0c1aede75cd7";,
host: "10.240.118.24:9292"
and nova will show this error:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 142, in _dispatch_and_reply
executor_callback))
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 186, in _dispatch
executor_callback)
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 130, in _do_dispatch
result = func(ctxt, **new_args)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
6933, in snapshot_instance
return self.manager.snapshot_instance(ctxt, image_id, instance)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in
wrapped
payload)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,
in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in
wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 333,
in decorated_function
LOG.warning(msg, e, instance_uuid=instance_uuid)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,
in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 304,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361,
in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,
in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409,
in decorated_function
instance=instance)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85,
in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399,
in decorated_function
*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
3291, in snapshot_instance
task_states.IMAGE_SNAPSHOT)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
3321, in _snapshot_instance
update_task_state)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
1447, in snapshot
image_file)
File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in
update
purge_props=purge_props)
File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 397, in
update
_reraise_translated_image_exception(image_id)
File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 395, in
update
image_id, **image_meta)
File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 218, in
call
return getattr(client.images, method)(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line
360, in update
resp, body = self.client.put(url, headers=hdrs, data=image_data)
File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line
268, in put
return self._request('PUT', url, **kwargs)
File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line
209, in _request
**kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in
request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in
send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 388, in
send
low_conn.send(hex(len(i))[2:].encode('utf-8'))
File "/usr/lib64/python2.7/httplib.py", line 820, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py",
line 101, in sendall
tail = self.send(data)
File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py",
line 81, in write
return self.fd.write(data)
File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 950, in send
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 862, in
_raise_ssl_error
raise SysCallError(errno, errorcode[errno])
SysCallError: (104, 'ECONNRESET')
When you don't use a loadbalancer and have nova point to glance directly and
have glance utilize ssl in the config, you see this type of error from nova:
2016-03-16 17:39:27.097 523 ERROR oslo_messaging.rpc.dispatcher
[req-84622019-ef9e-45a6-97a7-a3a7c8c45661 f9e76346bd0344a68a98887ce27d5a46
0070df81861b47a9a805643c8864c607 - - -] Exception during message handling: (32,
'EPIPE')
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher Traceback
(most recent call last):
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142,
in _dispatch_and_reply
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
executor_callback))
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186,
in _dispatch
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
executor_callback)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130,
in _do_dispatch
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher result =
func(ctxt, **new_args)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6933, in
snapshot_instance
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher return
self.manager.snapshot_instance(ctxt, image_id, instance)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher payload)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 333, in
decorated_function
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
LOG.warning(msg, e, instance_uuid=instance_uuid)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 304, in
decorated_function
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 361, in
decorated_function
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
kwargs['instance'], e, sys.exc_info())
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 349, in
decorated_function
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 409, in
decorated_function
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
instance=instance)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in
decorated_function
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher *args,
**kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3291, in
snapshot_instance
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
task_states.IMAGE_SNAPSHOT)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3321, in
_snapshot_instance
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
update_task_state)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1447, in
snapshot
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
image_file)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/image/api.py", line 130, in update
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
purge_props=purge_props)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/image/glance.py", line 397, in update
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
_reraise_translated_image_exception(image_id)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/image/glance.py", line 395, in update
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher image_id,
**image_meta)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/image/glance.py", line 218, in call
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher return
getattr(client.images, method)(*args, **kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 360, in
update
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher resp,
body = self.client.put(url, headers=hdrs, data=image_data)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 268, in put
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher return
self._request('PUT', url, **kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 209, in
_request
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher **kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher resp =
self.send(prep, **send_kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher r =
adapter.send(request, **kwargs)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/requests/adapters.py", line 390, in send
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
low_conn.send(i)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib64/python2.7/httplib.py", line 820, in send
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
self.sock.sendall(data)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 101, in
sendall
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher tail =
self.send(data)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 81, in
write
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher return
self.fd.write(data)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 950, in send
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher
self._raise_ssl_error(self._ssl, result)
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 862, in _raise_ssl_error
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher raise
SysCallError(errno, errorcode[errno])
2016-03-16 17:39:27.097 523 TRACE oslo_messaging.rpc.dispatcher SysCallError:
(32, 'EPIPE')
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1558398/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
