** Changed in: openstack-ansible Status: In Progress => Fix Committed
** Changed in: openstack-ansible Status: Fix Committed => Fix Released ** Changed in: glance Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1582279 Title: Glance (swift) download broken after Kilo to Liberty Upgrade Status in Glance: Invalid Status in openstack-ansible: Fix Released Bug description: After upgrading a Kilo environment to Liberty I noticed the issue that I can not download all glance images backed by swift when they were using Keystone v2 API and getting a 401 wrapped into a 404 swift error: 2016-05-13 17:45:51.253 4708 ERROR swiftclient [req-4a041b13-2081-45b4 -a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404) 2016-05-13 17:45:51.253 4708 ERROR swiftclient Traceback (most recent call last): 2016-05-13 17:45:51.253 4708 ERROR swiftclient File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1413, in _retry 2016-05-13 17:45:51.253 4708 ERROR swiftclient self.url, self.token = self.get_auth() 2016-05-13 17:45:51.253 4708 ERROR swiftclient File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 1367, in get_auth 2016-05-13 17:45:51.253 4708 ERROR swiftclient timeout=self.timeout) 2016-05-13 17:45:51.253 4708 ERROR swiftclient File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 490, in get_auth 2016-05-13 17:45:51.253 4708 ERROR swiftclient auth_version=auth_version) 2016-05-13 17:45:51.253 4708 ERROR swiftclient File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py", line 418, in get_auth_keystone 2016-05-13 17:45:51.253 4708 ERROR swiftclient raise ClientException('Authorization Failure. %s' % err) 2016-05-13 17:45:51.253 4708 ERROR swiftclient ClientException: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404) 2016-05-13 17:45:51.253 4708 ERROR swiftclient 2016-05-13 17:45:51.254 4708 WARNING glance.location [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Get image 95576f28-afed-4b63-93b4-1d07928930da data failed: Authorization Failure. Authorization failed: The resource could not be found. (HTTP 404) (Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404). 2016-05-13 17:45:51.254 4708 ERROR glance.location [req- 4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Glance tried all active locations to get data for image 95576f28-afed-4b63-93b4-1d07928930da but all have failed. 2016-05-13 17:45:51.256 4708 INFO eventlet.wsgi.server [req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239 7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Traceback (most recent call last): File "/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/eventlet/wsgi.py", line On further debugging I noticed that the swift client tried to retrieve a token from a Keystone v3 URL on a v2 API endpoint: POST /v2.0/auth/tokens HTTP/1.1 Host: 1.2.3.4:5000 Content-Length: 254 Accept-Encoding: gzip, deflate Accept: application/json User-Agent: python-keystoneclient Connection: keep-alive Content-Type: application/json {"auth": {"scope": {"project": {"domain": {"id": "default"}, "name": "service"}}, "identity": {"password": {"user": {"domain": {"id": "default"}, "password": "xxxxx" , "name": "glance"}}, "methods": ["password"]}}} HTTP/1.1 404 Not Found Date: Fri, 13 May 2016 22:54:23 GMT Server: Apache Vary: X-Auth-Token x-openstack-request-id: req-22a9f19a-e72c-4f22-87d3-c3f25fb78a9f Content-Length: 93 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}} which seems to be related to the config change swift_store_auth_version from 2 in Kilo to 3 in Liberty. Current Glance config default_store = swift stores = swift,http,cinder swift_store_auth_version = 3 swift_store_auth_address = http://1.2.3.4:5000/v3 swift_store_auth_insecure = False swift_store_user = service:glance swift_store_key = xxxxx swift_store_region = RegionOne swift_store_container = glance_images swift_store_endpoint_type = internalURL To overcome this issue I updated all glance image_locations to point to a V3 keystone API endpoint: update image_locations set value=replace(value, '5000/v2.0/glance_images', '5000/v3/glance_images') where value like 'swift%5000/v2.0/glance_images%' and status='active'; After that all image downloads started working. As a additional information, Keystone was running in a stock configuration, Fernet with SQL identity back end. Interestingly I did another upgrade to Liberty trying to reproduce this issue but was not successful yet. I have yet to determine what influenced the swift client to cause this behavior. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1582279/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp