Re: [openstack-dev] [keystone] keystoneauth version auto discovery for internal endpoints in queens
On 05/11/2018 03:37 PM, Vlad Gusev wrote: Hello. We faced a bug in keystoneauth, which haven't existed before Queens. Sorry about that. In our OpenStack deployments we use urls like http://controller:5000/v3 for internal and admin endpoints and urls like https://api.example.org/identity/v3 for public endpoints. Thank you for using suburl deployment for your public interface and not the silly ports!!! We set option public_endpoint in [default] section of the keystone.conf/nova.conf/cinder.conf/glance.conf/neutron.conf. For example, for keystone it is 'public_endpoint=https://api.example.org/identity/'. Since keystoneauth 3.2.0 or commit https://github.com/openstack/keystoneauth/commit/8b8ff830e89923ca6862362a5d16e496a0c0093c all internal client requests to the internal endpoints (for example, openstack server list from controller node) fail with 404 error, because it tries to do auto discovery at the http://controller:5000/v3. It gets {"href": "https://api.example.org/identity/v3/";, "rel": "self"} because of the public_endpoint option, and then in function _combine_relative_url() (keystoneauth1/discover.py:405) keystoneauth combines http://controller:5000/ with the path from public href. So after auto discovery attempt it goes to the wrong path http://controller:5000/identity/v3/ Ok. I'm going to argue that there are bugs on both the server side AND in keystoneauth. I believe I know how to fix the keystoneauth one- but let me describe why I think the server is broken as well, and then we can figure out how to fix that. I'm going to describe it in slightly excruciating detail, just to make sure we're all on the same page about mechanics that may be going on behind the scenes. The user has said: I want the internal interface of v3 of the identity service First, the identity service has to be found in the catalog. Looking in the catalog, we find this: { "endpoints": [ { "id": "4deb4d0504a044a395d4480741ba628c", "interface": "public", "region": "RegionOne", "url": "https://api.example.com/identity"; }, { "id": "012322eeedcd459edabb4933021112bc", "interface": "internal", "region": "RegionOne", "url": "http://controller:5000/v3"; } ], "name": "keystone", "type": "identity" }, We've found the entry for 'identity' service, and looking at the endpoints we see that the internal endpoint is: http://controller:5000/v3 The next step is version discovery, because the user wants version 3 of the api. (I'm skipping possible optimizations that can be applied on purpose) To do version discovery, one does a GET on the endpoint found in the catalog, so GET http://controller:5000/v3. That returns: { "versions": { "values": [ { "status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v3+json" } ], "id": "v3.6", "links": [ { "href": "https://api.example.com/identity/v3/";, "rel": "self" } ] } ] } } Here is the server-side bug. A GET on the discovery document on the internal endpoint returned an endpoint for the public interface. That is incorrect information. GET http://controller:5000/v3 should return either: { "versions": { "values": [ { "status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v3+json" } ], "id": "v3.6", "links": [ { "href": "http://controller:5000/v3/";, "rel": "self" } ] } ] } } or { "versions": { "values": [ { "status": "stable", "updated": "2016-04-04T00:00:00Z", "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.identity-v3+json" } ], "id": "v3.6", "links": [ { "href": "/v3/", "rel": "self" } ] } ] } } That's because the discovery documents are maps to what the user wants. The user needs to be able to follow them automatically. NOW - there is also a keystoneauth bug in play here that combined with this server-side bug have produced the issue you have. That is in the way we do the catalog / discovery URL join. First of all - we do the catalog / discovery URL join because of a frequently occuring deployment bug in the other direction. That is, it is an EXTREMELY common misconfiguration for the discovery url to return the internal url (this is what happens
Re: [openstack-dev] [keystone] keystoneauth version auto discovery for internal endpoints in queens
Typically speaking if we broke a behavior via a change in KeystoneAuth (not some behavior change in openstackclient or the way osc processes requests), we are in the wrong and we will need to go back through and fix the previous behavior. I'll spend some time going through this to verify if this really is a KSA change bug or something else. If it is in-fact a KSA (keystoneauth) bug, we'll work to restore the previous behavior(s) as reasonably quickly as possible. Cheers, --Morgan On Fri, May 11, 2018 at 1:37 PM, Vlad Gusev wrote: > Hello. > > We faced a bug in keystoneauth, which haven't existed before Queens. > > In our OpenStack deployments we use urls like http://controller:5000/v3 for > internal and admin endpoints and urls like > https://api.example.org/identity/v3 for public endpoints. We set option > public_endpoint in [default] section of the > keystone.conf/nova.conf/cinder.conf/glance.conf/neutron.conf. For example, > for keystone it is 'public_endpoint=https://api.example.org/identity/'. > > Since keystoneauth 3.2.0 or commit > https://github.com/openstack/keystoneauth/commit/8b8ff830e89923ca6862362a5d16e496a0c0093c > all internal client requests to the internal endpoints (for example, > openstack server list from controller node) fail with 404 error, because it > tries to do auto discovery at the http://controller:5000/v3. It gets > {"href": "https://api.example.org/identity/v3/";, "rel": "self"} because of > the public_endpoint option, and then in function _combine_relative_url() > (keystoneauth1/discover.py:405) keystoneauth combines > http://controller:5000/ with the path from public href. So after auto > discovery attempt it goes to the wrong path > http://controller:5000/identity/v3/ > > Before this commit openstackclient made auth request to the > https://api.example.org/identity/v3/auth/tokens (and it worked, because in > our deployment internal services and console clients can access this public > url). At best, we expect openstackclient always go to the > http://controller:5000/v3/ > > This problem partially could be solved by explicitly passing public > --os-auth-url https://api.example.org/identity/identity/v3 to the console > clients even if we want to use internal endpoints. > > I found similiar bug in launchpad, but it haven't received any attention: > https://bugs.launchpad.net/keystoneauth/+bug/1733052 > > What could be done with this behavior of keystoneauth auto discovery? > > - Vlad > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [keystone] keystoneauth version auto discovery for internal endpoints in queens
Hello. We faced a bug in keystoneauth, which haven't existed before Queens. In our OpenStack deployments we use urls like http://controller:5000/v3 for internal and admin endpoints and urls like https://api.example.org/identity/v3 for public endpoints. We set option public_endpoint in [default] section of the keystone.conf/nova.conf/cinder.conf/glance.conf/neutron.conf. For example, for keystone it is 'public_endpoint=https://api.example.org/identity/'. Since keystoneauth 3.2.0 or commit https://github.com/openstack/keystoneauth/commit/8b8ff830e89923ca6862362a5d16e496a0c0093c all internal client requests to the internal endpoints (for example, openstack server list from controller node) fail with 404 error, because it tries to do auto discovery at the http://controller:5000/v3. It gets {"href": "https://api.example.org/identity/v3/";, "rel": "self"} because of the public_endpoint option, and then in function _combine_relative_url() (keystoneauth1/discover.py:405) keystoneauth combines http://controller:5000/ with the path from public href. So after auto discovery attempt it goes to the wrong path http://controller:5000/identity/v3/ Before this commit openstackclient made auth request to the https://api.example.org/identity/v3/auth/tokens (and it worked, because in our deployment internal services and console clients can access this public url). At best, we expect openstackclient always go to the http://controller:5000/v3/ This problem partially could be solved by explicitly passing public --os-auth-url https://api.example.org/identity/identity/v3 to the console clients even if we want to use internal endpoints. I found similiar bug in launchpad, but it haven't received any attention: https://bugs.launchpad.net/keystoneauth/+bug/1733052 What could be done with this behavior of keystoneauth auto discovery? - Vlad __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev