For what it's worth - if the concept described in comment #5 was broken, we'd see a bunch of issues and failures in the gate with projects that deploy keystone in HA configurations by default (e.g. the openstack- ansible community does this with some of their gate tests).
I'm going to mark this as invalid for now. Please feel free to continue using this report for discussion or questions. You can also swing by #openstack-keystone on Freenode and I'd be happy to help explain fernet key rotation in greater detail. ** Changed in: keystone Status: Incomplete => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1702230 Title: fernet token fails with keystone HA Status in OpenStack Identity (keystone): Invalid Bug description: I have newton release in my environment with keystone provider fernet on Centos 7. When I am trying to upload image to glance it is failing with below message. glance-api.log: 2017-07-04 02:03:28.771 8105 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "This is not a recognized Fernet token gAAAAABZWy-wVlXrPlfa6_6PsyXky45ejM06Yt04UTLN6I51-CDT-kio83aIM00Xd6XL0bRzdwY8-Ks1L8SJD-xsGKyf-XUtm5TzskxpmhPXi0vDBYnM7pH2MnopcHW3RYH7YEUnqLIHGUVoBS5MGxgmSsgv0w20onikCu7xD-kDtR1gDOdryPU=", "code": 404, "title": "Not Found"}} 2017-07-04 02:03:28.772 8105 WARNING keystonemiddleware.auth_token [-] Authorization failed for token Below is the debug message I am getting. # openstack image list --debug START with options: [u'image', u'list', u'--debug'] options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', auth_type='', auth_url='http://192.168.27.23:35357/v3', authorization_code='', cacert=None, cert='', client_id='', client_secret='***', cloud='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='', key='', log_file=None, old_profile=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_identity_api_version='3', os_image_api_version='2', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='Default', project_id='', project_name='admin', protocol='', redirect_ uri='', region_name='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='Default', user_id='', username='admin', verbose_level=3, verify=None) Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': 'gaian', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_sou rce': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} defaults: {u'auth_type': 'password', u'status': u'active', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', u'metering_api_version': u'2', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'cacert': None, u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} cloud cfg: {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', u'orchestration_api_version': u'1', u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.27.23:35357/v3', 'password': '***', 'project_domain_name': 'Default'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_ve rsion': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'interface': None, u'disable_vendor_agent': {}} compute API version 2, cmd group openstack.compute.v2 network API version 2, cmd group openstack.network.v2 image API version 2, cmd group openstack.image.v2 volume API version 2, cmd group openstack.volume.v2 identity API version 3, cmd group openstack.identity.v3 object_store API version 1, cmd group openstack.object_store.v1 neutronclient API version 2, cmd group openstack.neutronclient.v2 Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_sourc e': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} Auth plugin password selected auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'http://192.168.27.23:35357/v3', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'user_domain_name': 'Default', 'project_name': 'admin', 'project_domain_name': 'Default'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': '3', u'volume_api_version': u'2', 'username': 'admin', 'cert': None, u'secgroup_sourc e': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}} command: image list -> openstackclient.image.v2.image.ListImage Using auth plugin: password Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'http://192.168.27.23:35357/v3', 'password': '***', 'project_domain_name': 'Default'} Get auth_ref REQ: curl -g -i -X GET http://192.168.27.23:35357/v3 -H "Accept: application/json" -H "User-Agent: osc-lib keystoneauth1/2.12.2 python-requests/2.11.1 CPython/2.7.5" Starting new HTTP connection (1): 192.168.27.23 "GET /v3 HTTP/1.1" 200 253 RESP: [200] Date: Tue, 04 Jul 2017 06:03:27 GMT Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token x-openstack-request-id: req-14636b59-b27e-48c4-8bb3-609136821eab Content-Length: 253 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json RESP BODY: {"version": {"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://192.168.27.23:35357/v3/", "rel": "self"}]}} Making authentication request to http://192.168.27.23:35357/v3/auth/tokens "POST /v3/auth/tokens HTTP/1.1" 201 1629 {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8cb83746f16f4c0c86d578aef08d2909", "name": "admin"}], "expires_at": "2017-07-04T07:03:28.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cde189882de44539afb4247aa656acf1", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.27.23:35357/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6fcd48c87a884180a118c35ab84e0671"}, {"url": "http://192.168.27.23:5000/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "76ab1355510f4d20a20e3987511223c8"}, {"url": "http://192.168.27.23:35357/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "d905017f8a9f4917acc576da8f6b8717"}], "type": "identity", "id": "61d50fd50a7f4d68aa2f7c95e51f4b51", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.27.23:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionO ne", "id": "021fba1895ba423aa2693b9033184b87"}, {"url": "http://192.168.27.23:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "86a5a30b748c4506b3fa763a58d6199e"}, {"url": "http://192.168.27.23:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "91f25b85569445d6b7c69cff4e6b6b55"}], "type": "image", "id": "e7b681e057134e398a581e01ad99841d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "5796fd4f78134259865d83effb65521e", "name": "admin"}, "audit_ids": ["zQ6hqEQMQ4mghdQkGeAHCQ"], "issued_at": "2017-07-04T06:03:28.000000Z"}} run(Namespace(columns=[], formatter='table', limit=None, long=False, marker=None, max_width=0, noindent=False, page_size=None, private=False, property=None, public=False, quote_mode='nonnumeric', shared=False, sort=None)) Instantiating image client: <class 'glanceclient.v2.client.Client'> Making authentication request to http://192.168.27.23:35357/v3/auth/tokens "POST /v3/auth/tokens HTTP/1.1" 201 1629 {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8cb83746f16f4c0c86d578aef08d2909", "name": "admin"}], "expires_at": "2017-07-04T07:03:28.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cde189882de44539afb4247aa656acf1", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.27.23:35357/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6fcd48c87a884180a118c35ab84e0671"}, {"url": "http://192.168.27.23:5000/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "76ab1355510f4d20a20e3987511223c8"}, {"url": "http://192.168.27.23:35357/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "d905017f8a9f4917acc576da8f6b8717"}], "type": "identity", "id": "61d50fd50a7f4d68aa2f7c95e51f4b51", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.27.23:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionO ne", "id": "021fba1895ba423aa2693b9033184b87"}, {"url": "http://192.168.27.23:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "86a5a30b748c4506b3fa763a58d6199e"}, {"url": "http://192.168.27.23:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "91f25b85569445d6b7c69cff4e6b6b55"}], "type": "image", "id": "e7b681e057134e398a581e01ad99841d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "5796fd4f78134259865d83effb65521e", "name": "admin"}, "audit_ids": ["zCiqSllWQwuU87EU1eUmsA"], "issued_at": "2017-07-04T06:03:28.000000Z"}} Instantiating image api: <class 'openstackclient.api.image_v2.APIv2'> REQ: curl -g -i -X GET http://192.168.27.23:9292/v2/images -H "User-Agent: osc-lib keystoneauth1/2.12.2 python-requests/2.11.1 CPython/2.7.5" -H "X-Auth-Token: {SHA1}bd691c601e36cb572f7dca23c370cac02cc3dbfa" Starting new HTTP connection (1): 192.168.27.23 "GET /v2/images HTTP/1.1" 401 253 RESP: [401] Content-Length: 253 Content-Type: text/plain; charset=UTF-8 Www-Authenticate: Keystone uri='http://192.168.27.23:5000' Date: Tue, 04 Jul 2017 06:03:28 GMT Connection: keep-alive RESP BODY: 401 Unauthorized This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required. Making authentication request to http://192.168.27.23:35357/v3/auth/tokens "POST /v3/auth/tokens HTTP/1.1" 201 1629 {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "8cb83746f16f4c0c86d578aef08d2909", "name": "admin"}], "expires_at": "2017-07-04T07:03:29.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "cde189882de44539afb4247aa656acf1", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://192.168.27.23:35357/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "6fcd48c87a884180a118c35ab84e0671"}, {"url": "http://192.168.27.23:5000/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "76ab1355510f4d20a20e3987511223c8"}, {"url": "http://192.168.27.23:35357/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "d905017f8a9f4917acc576da8f6b8717"}], "type": "identity", "id": "61d50fd50a7f4d68aa2f7c95e51f4b51", "name": "keystone"}, {"endpoints": [{"url": "http://192.168.27.23:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionO ne", "id": "021fba1895ba423aa2693b9033184b87"}, {"url": "http://192.168.27.23:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "86a5a30b748c4506b3fa763a58d6199e"}, {"url": "http://192.168.27.23:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "91f25b85569445d6b7c69cff4e6b6b55"}], "type": "image", "id": "e7b681e057134e398a581e01ad99841d", "name": "glance"}], "user": {"domain": {"id": "default", "name": "Default"}, "id": "5796fd4f78134259865d83effb65521e", "name": "admin"}, "audit_ids": ["PRP42NBYQ7iU8RcSOc1ybQ"], "issued_at": "2017-07-04T06:03:29.000000Z"}} "GET /v2/images HTTP/1.1" 401 253 RESP: [401] Content-Length: 253 Content-Type: text/plain; charset=UTF-8 Www-Authenticate: Keystone uri='http://192.168.27.23:5000' Date: Tue, 04 Jul 2017 06:03:29 GMT Connection: keep-alive RESP BODY: 401 Unauthorized This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required. Request returned failure status: 401 Unauthorized (HTTP 401) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/display.py", line 100, in run column_names, data = self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/openstackclient/image/v2/image.py", line 518, in take_action data = image_client.api.image_list(**kwargs) File "/usr/lib/python2.7/site-packages/openstackclient/api/image_v2.py", line 74, in image_list return self.list(url, **filter)['images'] File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 198, in list params=params, File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 84, in _request return session.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request resp = super(TimingSession, self).request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner return wrapped(*args, **kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 570, in request raise exceptions.from_response(resp, method, url) Unauthorized: Unauthorized (HTTP 401) clean_up ListImage: Unauthorized (HTTP 401) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run ret_val = super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 267, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand ret_value = super(OpenStackShell, self).run_subcommand(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/display.py", line 100, in run column_names, data = self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/openstackclient/image/v2/image.py", line 518, in take_action data = image_client.api.image_list(**kwargs) File "/usr/lib/python2.7/site-packages/openstackclient/api/image_v2.py", line 74, in image_list return self.list(url, **filter)['images'] File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 198, in list params=params, File "/usr/lib/python2.7/site-packages/openstackclient/api/api.py", line 84, in _request return session.request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/osc_lib/session.py", line 40, in request resp = super(TimingSession, self).request(url, method, **kwargs) File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner return wrapped(*args, **kwargs) File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 570, in request raise exceptions.from_response(resp, method, url) Unauthorized: Unauthorized (HTTP 401) END return value: 1 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1702230/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp