** Also affects: glance/juno Importance: Undecided Status: New ** Changed in: glance/juno Status: New => Fix Committed
** Changed in: glance/juno Milestone: None => 2014.2.1 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1198566 Title: expired image location url cause glance client errors Status in OpenStack Image Registry and Delivery Service (Glance): Fix Committed Status in Glance juno series: Fix Committed Bug description: We have a multi-node Openstack cluster running Folsom 2012.2.3 on Ubuntu 12.04. A few days ago we added a new compute node, and found that we were unable to launch new instances from a pre-existing Ubuntu Server 12.04 LTS image stored in glance. Each spawning attempt would deposit a glance client exception (shown below) in the compute node's nova-compute.log. After quite a lot of investigation, I found that the original --location URL (used during "glance image-create") of the Ubuntu Server image had gone out of date. This was evidently causing a BadStoreUri exception on the glance server during instance spawning, resulting in a 500 error being returned to our new compute node's glance client. I was able to resolve the problem by re-importing an Ubuntu Server 12.04 LTS image from a working mirror. Improved error logging would have saved us hours of troubleshooting. 2013-06-27 21:19:24 ERROR nova.compute.manager [req-f8d7c23a-e8ad-4059-bea4-4fc588a6afe0 9d8968d3f17f4697aaf923c14651ce7b e5fb3c6db0db4e9c86d0301005e2e5bb] [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] Instance failed to spawn 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] Traceback (most recent call last): 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 756, in _spawn 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] block_device_info) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] temp_level, payload) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] self.gen.next() 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] return f(*args, **kw) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1099, in spawn 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] admin_pass=admin_password) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1365, in _create_image 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] project_id=instance['project_id']) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 131, in cache 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] *args, **kwargs) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 178, in create_image 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] prepare_template(target=base, *args, **kwargs) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 795, in inner 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] retval = f(*args, **kwargs) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 122, in call_if_not_exists 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] fetch_func(target=target, *args, **kwargs) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in fetch_image 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] images.fetch_to_raw(context, image_id, target, user_id, project_id) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in fetch_to_raw 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] fetch(context, image_href, path_tmp, user_id, project_id) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 82, in fetch 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] image_service.download(context, image_id, image_file) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 222, in download 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] _reraise_translated_image_exception(image_id) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 220, in download 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] image_chunks = self._client.call(context, 1, 'data', image_id) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 138, in call 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] return getattr(client.images, method)(*args, **kwargs) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 108, in data 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] resp, body = self.api.raw_request('GET', '/v1/images/%s' % image_id) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 191, in raw_request 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] return self._http_request(url, method, **kwargs) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 158, in _http_request 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] raise exc.from_response(resp) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] HTTPInternalServerError: HTTPInternalServerError (HTTP 500) 2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1198566/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp