Re: [openstack-dev] [keystone] keystoneauth version auto discovery for internal endpoints in queens

2018-05-12 Thread Monty Taylor 

On 05/11/2018 03:37 PM, Vlad Gusev wrote:

Hello.

We faced a bug in keystoneauth, which haven't existed before Queens.


Sorry about that.

In our OpenStack deployments we use urls like http://controller:5000/v3 
for internal and admin endpoints and urls like 
https://api.example.org/identity/v3 for public endpoints.


Thank you for using suburl deployment for your public interface and not 
the silly ports!!!


We set option 
public_endpoint in [default] section of the 
keystone.conf/nova.conf/cinder.conf/glance.conf/neutron.conf. For 
example, for keystone it is 
'public_endpoint=https://api.example.org/identity/'.


Since keystoneauth 3.2.0 or commit 
https://github.com/openstack/keystoneauth/commit/8b8ff830e89923ca6862362a5d16e496a0c0093c 
all internal client requests to the internal endpoints (for example, 
openstack server list from controller node) fail with 404 error, because 
it tries to do auto discovery at the http://controller:5000/v3. It gets 
{"href": "https://api.example.org/identity/v3/;, "rel": "self"} because 
of the public_endpoint option, and then in function 
_combine_relative_url() (keystoneauth1/discover.py:405) keystoneauth 
combines http://controller:5000/ with the path from public href. So 
after auto discovery attempt it goes to the wrong path 
http://controller:5000/identity/v3/


Ok. I'm going to argue that there are bugs on both the server side AND 
in keystoneauth. I believe I know how to fix the keystoneauth one- but 
let me describe why I think the server is broken as well, and then we 
can figure out how to fix that. I'm going to describe it in slightly 
excruciating detail, just to make sure we're all on the same page about 
mechanics that may be going on behind the scenes.


The user has said:

  I want the internal interface of v3 of the identity service

First, the identity service has to be found in the catalog. Looking in 
the catalog, we find this:


  {
"endpoints": [
  {
"id": "4deb4d0504a044a395d4480741ba628c",
"interface": "public",
"region": "RegionOne",
"url": "https://api.example.com/identity;
  },
  {
"id": "012322eeedcd459edabb4933021112bc",
"interface": "internal",
"region": "RegionOne",
"url": "http://controller:5000/v3;
  }
],
"name": "keystone",
"type": "identity"
  },

We've found the entry for 'identity' service, and looking at the 
endpoints we see that the internal endpoint is:


  http://controller:5000/v3

The next step is version discovery, because the user wants version 3 of 
the api. (I'm skipping possible optimizations that can be applied on 
purpose)


To do version discovery, one does a GET on the endpoint found in the 
catalog, so GET http://controller:5000/v3. That returns:


{
  "versions": {
"values": [
  {
"status": "stable",
"updated": "2016-04-04T00:00:00Z",
"media-types": [
  {
"base": "application/json",
"type": "application/vnd.openstack.identity-v3+json"
  }
],
"id": "v3.6",
"links": [
  {
"href": "https://api.example.com/identity/v3/;,
"rel": "self"
  }
]
  }
]
  }
}

Here is the server-side bug. A GET on the discovery document on the 
internal endpoint returned an endpoint for the public interface. That is 
incorrect information. GET http://controller:5000/v3 should return either:


{
  "versions": {
"values": [
  {
"status": "stable",
"updated": "2016-04-04T00:00:00Z",
"media-types": [
  {
"base": "application/json",
"type": "application/vnd.openstack.identity-v3+json"
  }
],
"id": "v3.6",
"links": [
  {
"href": "http://controller:5000/v3/;,
"rel": "self"
  }
]
  }
]
  }
}

or

{
  "versions": {
"values": [
  {
"status": "stable",
"updated": "2016-04-04T00:00:00Z",
"media-types": [
  {
"base": "application/json",
"type": "application/vnd.openstack.identity-v3+json"
  }
],
"id": "v3.6",
"links": [
  {
"href": "/v3/",
"rel": "self"
  }
]
  }
]
  }
}

That's because the discovery documents are maps to what the user wants. 
The user needs to be able to follow them automatically.


NOW - there is also a keystoneauth bug in play here that combined with 
this server-side bug have produced the issue you have.


That is in the way we do the catalog / discovery URL join.

First of all - we do the catalog / discovery URL join because of a 
frequently occuring deployment bug in the other direction. That is, it 
is an EXTREMELY common misconfiguration for the discovery url to return 
the internal url (this is what happens if

Re: [openstack-dev] [keystone] keystoneauth version auto discovery for internal endpoints in queens

2018-05-11 Thread Morgan Fainberg 
Typically speaking if we broke a behavior via a change in KeystoneAuth
(not some behavior change in openstackclient or the way osc processes
requests), we are in the wrong and we will need to go back through and
fix the previous behavior.

I'll spend some time going through this to verify if this really is a
KSA change bug or something else. If it is in-fact a KSA
(keystoneauth) bug, we'll work to restore the previous behavior(s) as
reasonably quickly as possible.

Cheers,
--Morgan

On Fri, May 11, 2018 at 1:37 PM, Vlad Gusev  wrote:
> Hello.
>
> We faced a bug in keystoneauth, which haven't existed before Queens.
>
> In our OpenStack deployments we use urls like http://controller:5000/v3 for
> internal and admin endpoints and urls like
> https://api.example.org/identity/v3 for public endpoints. We set option
> public_endpoint in [default] section of the
> keystone.conf/nova.conf/cinder.conf/glance.conf/neutron.conf. For example,
> for keystone it is 'public_endpoint=https://api.example.org/identity/'.
>
> Since keystoneauth 3.2.0 or commit
> https://github.com/openstack/keystoneauth/commit/8b8ff830e89923ca6862362a5d16e496a0c0093c
> all internal client requests to the internal endpoints (for example,
> openstack server list from controller node) fail with 404 error, because it
> tries to do auto discovery at the http://controller:5000/v3. It gets
> {"href": "https://api.example.org/identity/v3/;, "rel": "self"} because of
> the public_endpoint option, and then in function _combine_relative_url()
> (keystoneauth1/discover.py:405) keystoneauth combines
> http://controller:5000/ with the path from public href. So after auto
> discovery attempt it goes to the wrong path
> http://controller:5000/identity/v3/
>
> Before this commit openstackclient made auth request to the
> https://api.example.org/identity/v3/auth/tokens (and it worked, because in
> our deployment internal services and console clients can access this public
> url). At best, we expect openstackclient always go to the
> http://controller:5000/v3/
>
> This problem partially could be solved by explicitly passing public
> --os-auth-url https://api.example.org/identity/identity/v3 to the console
> clients even if we want to use internal endpoints.
>
> I found similiar bug in launchpad, but it haven't received any attention:
> https://bugs.launchpad.net/keystoneauth/+bug/1733052
>
> What could be done with this behavior of keystoneauth auto discovery?
>
> - Vlad
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev