On Wed, Oct 19, 2016 at 12:59 PM, Jay Pipes <jaypi...@gmail.com> wrote:
On 10/19/2016 05:32 PM, Brian Curtin wrote:
I'm currently facing what looks more and more like an impossible
problem in determining the root of each service on a given cloud. It
is apparently a free-for-all in how endpoints can be structured, and I
think we're out of ways to approach it that catch all of the ways that
all people can think of.
In openstacksdk, we can no longer use the service catalog for
determining each service's endpoints. Among other things, this is due
to a combination of some versions of some services not actually being
listed, and with things heading the direction of version-less services
anyway. Recently we changed to using the service catalog as a pointer
to where services live and then try to find the root of that service
by stripping the path down and making some extra requests on startup
to find what's offered. Despite a few initial snags, this now works
reasonably well in a majority of cases.
We have seen endpoints structured in the following ways:
A. subdomains, e.g., https://service.cloud.com/v2
B. paths, e.g., https://cloud.com/service/v2 (sometimes there are
more paths in between the root and /service/)
C. service-specific ports, e.g., https://cloud.com:1234/v2
D. both A and B plus ports
Within all of these, we can find the root of the given service just
fine. We split the path and build successively longer paths starting
from the root. In the above examples, we need to hit the path just
short of the /v2, so in B it actually takes two requests as we'd make
one to cloud.com which fails, but then a second one to
cloud.com/service gives us what we need.
However, another case came up: the root of all endpoints is itself
another service. That makes it look like this:
E. https://cloud.com:9999/service/v2
F. https://cloud.com:9999/otherservice
In this case, https://cloud.com:9999 is keystone, so trying to get E's
base by going from the root and outward will give me a versions
response I can parse properly, but it points to keystone. We then end
up building requests for 'service' that go to keystone endpoints and
end up failing. We're doing this using itertools.accumulate on the
path fragments, so you might think 'just throw it through
`reversed()`' and go the other way. If we do that, we'll also get a
versions response that we can parse, but it's the v2 specific info,
not all available versions.
So now that we can't reliably go from the left, and we definitely
can't go from the right, how about the middle?
This sounds ridiculous, and if it sounds familiar it's because they
devise a "middle out" algorithm on the show Silicon Valley, but in
most cases it'd actually work. In E above, it'd be fine. However,
depending on the number of path fragments and which direction we chose
to move first, we'd sometimes hit either a version-specific response
or another service's response, so it's not reliable.
Ultimately, I would like to know how something like this can be solved.
1. Is there any reliable, functional, and accurate programmatic way to
get the versions and endpoints that all services on a cloud offer?
The Keystone service catalog should be the thing that provides the endpoints
for all services in the cloud. Within each service, determining the
(micro)version of the API is unfortunately going to be a per-service
endeavour. For some APIs, a microversion header is returned, others don't
have microversions. The microversion header is unfortunately not
standardized for all APIs that use microversions, though a number of us
would like to see a single:
OpenStack-API-Version: <service-type> <microversion>, ...
header supported. This is the header supported in the new placement REST
API, for what it's worth.
I get the microversion part, and we support that (for some degree of
support), but this is about the higher level major versions. The
example that started this was Keystone only listing a v2 endpoint in
the service catalog, at least on devstack. I need to be able to hit v3
APIs when a user wants to do v3 things, regardless of which version
they auth to, so the way to get it was to get the root and go from
there. That both versions weren't listed was initially confusing to
me, but that's where the suggestion of "go to the root and get
everything" started out.
The service catalog holding providing all of the available endpoints
made sense to me from what I understood in the past, but two things
are for sure about this: it doesn't work that way, and I've been told
several times that it's not going to work that way even in cases where
it is apparently working. I don't have sources to cite, but it's come
up a few times that the goal is one entry per service and you talk to
the service to find out all of its details - major versions, micro
versions, etc.
2. Are there any guidelines, rules, expectations, or other
documentation on how services can be installed and their endpoints
structured that are helpful to people build apps that use them, not in
those trying to install and operate them? I've looked around a few
times and found nothing useful. A lot of what I've found has
referenced suggestions for operators setting them up behind various
load balancing tools.
I presume you are referring to the "internal" vs "public" endpoint stuff? If
so, my preference has been that such "internal vs. external" routing should
be handled via the Keystone service catalog returning a set of endpoints
depending on the source (or X-forwarded-for) IP. So, requests from
"internal" networks (for whatever definition of "internal" you want) return
a set of endpoint URLs reflecting the "internal" endpoints.
This is more about structuring of the URLs in terms of forming a root,
port, paths, etc, like the examples A-F. Are those all of the forms
that could be expected? Are there others? Are any of those not
actually supported? So far it's been a lot of guesswork that started
with being built to work with devstack (so service-per-port), but then
people have come up and said the assumption we made doesn't work with
their cloud, so we've broadened support to catch the various cases.
That's still ongoing as cases are coming up.
I'm going to guess that operators are allowed to structure endpoints
in whatever way suits their needs and that there's no "your endpoints
should be formed like this..." document. If that's how it is, that's
fine, but that'll mean something—service catalog, an API, a new
project, whatever—is going to have to give me the roots of each
service.
