Public bug reported: Upgrading a fairly standard Folsom setup to Grizzly using our packages. After the updated config files are put in place and database has been migrated, nova's EC2 API fails to authenticate requests with keystone. The OSAPI end point works just fine.
On the client: $ euca-describe-instances Unauthorized: Failure communicating with keystone On the keystone server, with debug enabled: 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ******************** 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x2d64250 200 OK>} 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] REQUEST_METHOD = POST 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] PATH_INFO = /ec2tokens 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] CONTENT_LENGTH = 436 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] eventlet.posthooks = [] 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] RAW_PATH_INFO = /v2.0/ec2tokens 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] REMOTE_ADDR = 192.168.20.50 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x2d56050> 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.url_scheme = http 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] webob._body_file = (<LimitedLengthFile(<eventlet.wsgi.Input object at 0x2d56050>, maxlen=436)>, <eventlet.wsgi.Input object at 0x2d56050>) 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SERVER_PORT = 5000 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.input = <_io.BytesIO object at 0x2d5c1d0> 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': None, 'is_admin': False} 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] HTTP_HOST = test-09.os.magners.qa.lexington:5000 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.multithread = True 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] openstack.params = {u'ec2Credentials': {u'access': u'8e283c4e394247fbbabe5474cdbda9e4', u'host': u'test-02.os.magners.qa.lexington:8773', u'verb': u'POST', u'params': {u'SignatureVersion': u'2', u'AWSAccessKeyId': u'8e283c4e394247fbbabe5474cdbda9e4', u'Timestamp': u'2013-03-21T22:53:36Z', u'SignatureMethod': u'HmacSHA256', u'Version': u'2010-08-31', u'Action': u'DescribeInstances'}, u'signature': u'gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=', u'path': u'/services/Cloud/'}} 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0) 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] SERVER_NAME = 192.168.20.57 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.run_once = False 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x7f0bbbf08270> 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] webob.is_body_seekable = True 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] CONTENT_TYPE = application/json 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ******************** 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] {"ec2Credentials": {"access": "8e283c4e394247fbbabe5474cdbda9e4", "host": "test-02.os.magners.qa.lexington:8773", "verb": "POST", "params": {"SignatureVersion": "2", "AWSAccessKeyId": "8e283c4e394247fbbabe5474cdbda9e4", "Timestamp": "2013-03-21T22:53:36Z", "SignatureMethod": "HmacSHA256", "Version": "2010-08-31", "Action": "DescribeInstances"}, "signature": "gEJ42tacqfUrqvgpj2ouqg3T+aAiwzu6c5LWMrRQDEA=", "path": "/services/Cloud/"}} 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] arg_dict: {} 2013-03-21 18:53:36 WARNING [keystone.common.wsgi] Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ******************** 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] Content-Type = application/json 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] Content-Length = 120 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ******************** 2013-03-21 18:53:36 DEBUG [keystone.common.wsgi] {"error": {"message": "Could not find: credential-8e283c4e394247fbbabe5474cdbda9e4", "code": 404, "title": "Not Found"}} 2013-03-21 18:53:36 INFO [access] 192.168.20.50 - - [21/Mar/2013:22:53:36 +0000] "POST http://test-09.os.magners.qa.lexington:5000/v2.0/ec2tokens HTTP/1.0" 404 120 2013-03-21 18:53:36 DEBUG [eventlet.wsgi.server] 192.168.20.50 - - [21/Mar/2013 18:53:36] "POST /v2.0/ec2tokens HTTP/1.1" 404 256 0.010670 The nova-api-ec2.log: 2013-03-21 18:54:28.516 13600 ERROR nova.api.ec2 [-] Unauthorized: Failure communicating with keystone 2013-03-21 18:54:28.516 13600 INFO nova.api.ec2 [-] 0.14199s 192.168.20.1 POST /services/Cloud/ None:None 400 [Boto/2.3.0 (linux2)] application/x-www-form-urlencoded text/xml 2013-03-21 18:54:28.517 13600 INFO nova.ec2.wsgi.server [-] 192.168.20.1 "POST /services/Cloud/ HTTP/1.1" status: 400 len: 327 time: 0.0149109 ** Affects: keystone Importance: Undecided Status: New ** Affects: nova Importance: Undecided Status: New ** Affects: keystone (Ubuntu) Importance: Undecided Status: New ** Affects: nova (Ubuntu) Importance: Undecided Status: New ** Tags: upgrade ** Also affects: nova Importance: Undecided Status: New ** Also affects: keystone (Ubuntu) Importance: Undecided Status: New ** Also affects: keystone Importance: Undecided Status: New ** Tags added: upgrade -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1158563 Title: After grizzly upgrade, EC2 API requests fail:Could not find: credential To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1158563/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs