** Changed in: openstack-ansible
Status: In Progress => Fix Committed
** Changed in: openstack-ansible
Status: Fix Committed => Fix Released
** Changed in: glance
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1582279
Title:
Glance (swift) download broken after Kilo to Liberty Upgrade
Status in Glance:
Invalid
Status in openstack-ansible:
Fix Released
Bug description:
After upgrading a Kilo environment to Liberty I noticed the issue that
I can not download all glance images backed by swift when they were
using Keystone v2 API and getting a 401 wrapped into a 404 swift
error:
2016-05-13 17:45:51.253 4708 ERROR swiftclient [req-4a041b13-2081-45b4
-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Authorization Failure.
Authorization failed: The resource could not be found. (HTTP 404)
(Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)
2016-05-13 17:45:51.253 4708 ERROR swiftclient Traceback (most recent call
last):
2016-05-13 17:45:51.253 4708 ERROR swiftclient File
"/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py",
line 1413, in _retry
2016-05-13 17:45:51.253 4708 ERROR swiftclient self.url, self.token =
self.get_auth()
2016-05-13 17:45:51.253 4708 ERROR swiftclient File
"/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py",
line 1367, in get_auth
2016-05-13 17:45:51.253 4708 ERROR swiftclient timeout=self.timeout)
2016-05-13 17:45:51.253 4708 ERROR swiftclient File
"/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py",
line 490, in get_auth
2016-05-13 17:45:51.253 4708 ERROR swiftclient auth_version=auth_version)
2016-05-13 17:45:51.253 4708 ERROR swiftclient File
"/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/swiftclient/client.py",
line 418, in get_auth_keystone
2016-05-13 17:45:51.253 4708 ERROR swiftclient raise
ClientException('Authorization Failure. %s' % err)
2016-05-13 17:45:51.253 4708 ERROR swiftclient ClientException: Authorization
Failure. Authorization failed: The resource could not be found. (HTTP 404)
(Request-ID: req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404)
2016-05-13 17:45:51.253 4708 ERROR swiftclient
2016-05-13 17:45:51.254 4708 WARNING glance.location
[req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Get image
95576f28-afed-4b63-93b4-1d07928930da data failed: Authorization Failure.
Authorization failed: The resource could not be found. (HTTP 404) (Request-ID:
req-dfa80296-9d5c-487f-bbd4-64c91cf819cf) (HTTP 404).
2016-05-13 17:45:51.254 4708 ERROR glance.location [req-
4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Glance tried all active
locations to get data for image 95576f28-afed-4b63-93b4-1d07928930da
but all have failed.
2016-05-13 17:45:51.256 4708 INFO eventlet.wsgi.server
[req-4a041b13-2081-45b4-a4dd-f2b1473f9be3 a451fa41b56848a9be6a16a7b4dfe239
7a1ca9f7cc4e4b13ac0ed2957f1e8c32 - - -] Traceback (most recent call last):
File
"/openstack/venvs/glance-12.0.13/lib/python2.7/site-packages/eventlet/wsgi.py",
line
On further debugging I noticed that the swift client tried to retrieve
a token from a Keystone v3 URL on a v2 API endpoint:
POST /v2.0/auth/tokens HTTP/1.1
Host: 1.2.3.4:5000
Content-Length: 254
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-keystoneclient
Connection: keep-alive
Content-Type: application/json
{"auth": {"scope": {"project": {"domain": {"id": "default"}, "name":
"service"}}, "identity": {"password": {"user": {"domain": {"id": "default"},
"password": "xxxxx"
, "name": "glance"}}, "methods": ["password"]}}}
HTTP/1.1 404 Not Found
Date: Fri, 13 May 2016 22:54:23 GMT
Server: Apache
Vary: X-Auth-Token
x-openstack-request-id: req-22a9f19a-e72c-4f22-87d3-c3f25fb78a9f
Content-Length: 93
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/json
{"error": {"message": "The resource could not be found.", "code": 404,
"title": "Not Found"}}
which seems to be related to the config change
swift_store_auth_version from 2 in Kilo to 3 in Liberty.
Current Glance config
default_store = swift
stores = swift,http,cinder
swift_store_auth_version = 3
swift_store_auth_address = http://1.2.3.4:5000/v3
swift_store_auth_insecure = False
swift_store_user = service:glance
swift_store_key = xxxxx
swift_store_region = RegionOne
swift_store_container = glance_images
swift_store_endpoint_type = internalURL
To overcome this issue I updated all glance image_locations to point
to a V3 keystone API endpoint:
update image_locations set value=replace(value,
'5000/v2.0/glance_images', '5000/v3/glance_images') where value like
'swift%5000/v2.0/glance_images%' and status='active';
After that all image downloads started working.
As a additional information, Keystone was running in a stock
configuration, Fernet with SQL identity back end.
Interestingly I did another upgrade to Liberty trying to reproduce this issue
but was not successful yet.
I have yet to determine what influenced the swift client to cause this
behavior.
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1582279/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
