On Mon, May 01, 2017 at 05:00:17AM -0700, Flavio Percoco wrote: > On 28/04/17 11:19 -0500, Eric Fried wrote: > > If it's *just* glance we're making an exception for, I prefer #1 (don't > > deprecate/remove [glance]api_servers). It's way less code & > > infrastructure, and it discourages others from jumping on the > > multiple-endpoints bandwagon. If we provide endpoint_override_list > > (handwave), people will think it's okay to use it. > > > > Anyone aware of any other services that use multiple endpoints? > > Probably a bit late but yeah, I think this makes sense. I'm not aware of other > projects that have list of api_servers.
I thought it was just nova too, but it turns out cinder has the same exact option as nova: (I hit this in my devstack patch trying to get glance deployed as a wsgi app) https://github.com/openstack/cinder/blob/d47eda3a3ba9971330b27beeeb471e2bc94575ca/cinder/common/config.py#L51-L55 Although from what I can tell you don't have to set it and it will fallback to using the catalog, assuming you configured the catalog info for cinder: https://github.com/openstack/cinder/blob/19d07a1f394c905c23f109c1888c019da830b49e/cinder/image/glance.py#L117-L129 -Matt Treinish > > > On 04/28/2017 10:46 AM, Mike Dorman wrote: > > > Maybe we are talking about two different things here? I’m a bit confused. > > > > > > Our Glance config in nova.conf on HV’s looks like this: > > > > > > [glance] > > > api_servers=http://glance1:9292,http://glance2:9292,http://glance3:9292,http://glance4:9292 > > > glance_api_insecure=True > > > glance_num_retries=4 > > > glance_protocol=http > > > FWIW, this feature is being used as intended. I'm sure there are ways to > achieve > this using external tools like haproxy/nginx but that adds an extra burden to > OPs that is probably not necessary since this functionality is already there. > > Flavio > > > > So we do provide the full URLs, and there is SSL support. Right? I am > > > fairly certain we tested this to ensure that if one URL fails, nova goes > > > on to retry the next one. That failure does not get bubbled up to the > > > user (which is ultimately the goal.) > > > > > > I don’t disagree with you that the client side choose-a-server-at-random > > > is not a great load balancer. (But isn’t this roughly the same thing > > > that oslo-messaging does when we give it a list of RMQ servers?) For us > > > it’s more about the failure handling if one is down than it is about > > > actually equally distributing the load. > > > > > > In my mind options One and Two are the same, since today we are already > > > providing full URLs and not only server names. At the end of the day, I > > > don’t feel like there is a compelling argument here to remove this > > > functionality (that people are actively making use of.) > > > > > > To be clear, I, and I think others, are fine with nova by default getting > > > the Glance endpoint from Keystone. And that in Keystone there should > > > exist only one Glance endpoint. What I’d like to see remain is the > > > ability to override that for nova-compute and to target more than one > > > Glance URL for purposes of fail over. > > > > > > Thanks, > > > Mike > > > > > > > > > > > > > > > On 4/28/17, 8:20 AM, "Monty Taylor" <mord...@inaugust.com> wrote: > > > > > > Thank you both for your feedback - that's really helpful. > > > > > > Let me say a few more words about what we're trying to accomplish here > > > overall so that maybe we can figure out what the right way forward is. > > > (it may be keeping the glance api servers setting, but let me at least > > > make the case real quick) > > > > > > From a 10,000 foot view, the thing we're trying to do is to get > > > nova's > > > consumption of all of the OpenStack services it uses to be less > > > special. > > > > > > The clouds have catalogs which list information about the services - > > > public, admin and internal endpoints and whatnot - and then we're > > > asking > > > admins to not only register that information with the catalog, but to > > > also put it into the nova.conf. That means that any updating of that > > > info needs to be an API call to keystone and also a change to > > > nova.conf. > > > If we, on the other hand, use the catalog, then nova can pick up > > > changes > > > in real time as they're rolled out to the cloud - and there is > > > hopefully > > > a sane set of defaults we could choose (based on operator feedback > > > like > > > what you've given) so that in most cases you don't have to tell nova > > > where to find glance _at_all_ becuase the cloud already knows where it > > > is. (nova would know to look in the catalog for the interal interface > > > of > > > the image service - for instance - there's no need to ask an operator > > > to > > > add to the config "what is the service_type of the image service we > > > should talk to" :) ) > > > > > > Now - glance, and the thing you like that we don't - is especially > > > hairy > > > because of the api_servers list. The list, as you know, is just a list > > > of servers, not even of URLs. This means it's not possible to > > > configure > > > nova to talk to glance over SSL (which I know you said works for you, > > > but we'd like for people to be able to choose to SSL all their things) > > > We could add that, but it would be an additional pile of special > > > config. > > > Because of all of that, we also have to attempt to make working URLs > > > from what is usually a list of IP addresses. This is also clunky and > > > prone to failure. > > > > > > The implementation on the underside of the api_servers code is the > > > world's dumbest load balancer. It picks a server from the list at > > > random and uses it. There is no facility for dealing with a server in > > > the list that stops working or for allowing rolling upgrades like > > > there > > > would with a real load-balancer across the set. If one of the API > > > servers goes away, we have no context to know that, so just some of > > > your > > > internal calls to glance fail. > > > > > > Those are the issues - basically: > > > - current config is special and fragile > > > - impossible to SSL > > > - unflexible/unpowerful de-facto software loadbalancer > > > > > > Now - as is often the case - it turns out the combo of those things is > > > working very well for you -so we need to adjust our thinking on the > > > topic a bit. Let me toss out some alternatives and see what you think: > > > > > > Alternative One - Do Both things > > > > > > We add the new "consume from catalog" and make it default. (and make > > > it > > > default to consuming the internal interface by default) We have to do > > > that in parallel with the current glance api_servers setting anyway, > > > because of deprecation periods, so the code to support both approaches > > > will exist. Instead of then deprecating the api_servers list, we keep > > > it- but add a big doc warning listing the gotchas and limitations - > > > but > > > for those folks for whom they are not an issue, you've got an out. > > > > > > Alternative Two - Hybrid Approach - optional list of URLs > > > > > > We go ahead and move to service config being the standard way one > > > lists > > > how to consume a service from the catalog. One of the standard options > > > for consuming services is "endpoint_override" - which is a way an API > > > user can say "hi, please to ignore the catalog and use this endpoint > > > I've given you instead". The endpoint in question is a full URL, so > > > https/http and ports and whatnot are all handled properly. > > > > > > We add, in addition, an additional option "endpoint_override_list" > > > which > > > allows you to provide a list of URLs (not API servers) and if you > > > provide that option, we'll keep the logic of choosing one at random at > > > API call time. It's still a poor load balancer, and we'll still put > > > warnings in the docs about it not being a featureful load balancing > > > solution, but again would be available if needed. > > > > > > Alternative Three - We ignore you and give you docs > > > > > > I'm only including this because in the name of completeness. But we > > > could write a bunch of docs about a recommended way of putting your > > > internal endpoints in a load balancer and registering that with the > > > internal endpoint in keystone. (I would prefer to make the operators > > > happy, so let's say whatever vote I have is not for this option) > > > > > > Alternative Four - We update client libs to understand multiple values > > > from keystone for endpoints > > > > > > I _really_ don't like this one - as I think us doing dumb software > > > loadbalancing client side is prone to a ton of failures. BUT - right > > > now > > > the assumption when consuming endpoints from the catalog is that one > > > and > > > only one endpoint will be returned for a given > > > service_type/service_name/interface. Rather than special-casing the > > > url roundrobin in nova, we could move that round-robin to be in the > > > base > > > client library, update api consumption docs with round-robin > > > recommendations and then have you register the list of endpoints with > > > keystone. > > > > > > I know the keystone team has long been _very_ against using keystone > > > as > > > a list of all the endpoints, and I agree with them. Putting it here > > > for > > > sake of argument. > > > > > > Alternative Five - We update keystone to round-robin lists of > > > endpoints > > > > > > Potentially even worse than four and even more unlikely given the > > > keystone team's feelings, but we could have keystone continue to only > > > return one endpoint, but have it do the round-robin selection at > > > catalog > > > generation time. > > > > > > > > > Sorry - you caught me in early morning brainstorm mode. > > > > > > I am neither nova core nor keystone core. BUT: > > > > > > I think honestly if adding a load balancer in front of your internal > > > endpoints is an undue burden and/or the usefulness of the lists > > > outweighs the limitations they have, we should go with One or Two. (I > > > think three through five are all terrible) > > > > > > My personal preference would be for Two - the round-robin code winds > > > up > > > being the same logic in both cases, but at least in Two folks who want > > > to SSL all the way _can_, and it shouldn't be an undue extra burden on > > > those of you using the api_servers now. We also don't have to do the > > > funky things we currently have to do to turn the api_severs list into > > > workable URLs. > > > > > > > > > On 04/27/2017 11:50 PM, Blair Bethwaite wrote: > > > > We at Nectar are in the same boat as Mike. Our use-case is a little > > > > bit more about geo-distributed operations though - our Cells are in > > > > different States around the country, so the local glance-apis are > > > > particularly important for caching popular images close to the > > > > nova-computes. We consider these glance-apis as part of the > > > underlying > > > > cloud infra rather than user-facing, so I think we'd prefer not to > > > see > > > > them in the service-catalog returned to users either... is there > > > going > > > > to be a (standard) way to hide them? > > > > > > > > On 28 April 2017 at 09:15, Mike Dorman <mdor...@godaddy.com> wrote: > > > >> We make extensive use of the [glance]/api_servers list. We > > > configure that on hypervisors to direct them to Glance servers which are > > > more “local” network-wise (in order to reduce network traffic across > > > security zones/firewalls/etc.) This way nova-compute can fail over in > > > case one of the Glance servers in the list is down, without putting them > > > behind a load balancer. We also don’t run https for these “internal” > > > Glance calls, to save the overhead when transferring images. > > > >> > > > >> End-user calls to Glance DO go through a real load balancer and > > > then are distributed out to the Glance servers on the backend. From the > > > end-user’s perspective, I totally agree there should be one, and only one > > > URL. > > > >> > > > >> However, we would be disappointed to see the change you’re > > > suggesting implemented. We would lose the redundancy we get now by > > > providing a list. Or we would have to shunt all the calls through the > > > user-facing endpoint, which would generate a lot of extra traffic (in > > > places where we don’t want it) for image transfers. > > > >> > > > >> Thanks, > > > >> Mike > > > >> > > > >> > > > >> > > > >> On 4/27/17, 4:02 PM, "Matt Riedemann" <mriede...@gmail.com> wrote: > > > >> > > > >> On 4/27/2017 4:52 PM, Eric Fried wrote: > > > >> > Y'all- > > > >> > > > > >> > TL;DR: Does glance ever really need/use multiple endpoint > > > URLs? > > > >> > > > > >> > I'm working on bp use-service-catalog-for-endpoints[1], > > > which intends > > > >> > to deprecate disparate conf options in various groups, and > > > centralize > > > >> > acquisition of service endpoint URLs. The idea is to > > > introduce > > > >> > nova.utils.get_service_url(group) -- note singular 'url'. > > > >> > > > > >> > One affected conf option is [glance]api_servers[2], which > > > currently > > > >> > accepts a *list* of endpoint URLs. The new API will only > > > ever return *one*. > > > >> > > > > >> > Thus, as planned, this blueprint will have the side effect > > > of > > > >> > deprecating support for multiple glance endpoint URLs in > > > Pike, and > > > >> > removing said support in Queens. > > > >> > > > > >> > Some have asserted that there should only ever be one > > > endpoint URL for > > > >> > a given service_type/interface combo[3]. I'm fine with that > > > - it > > > >> > simplifies things quite a bit for the bp impl - but wanted > > > to make sure > > > >> > there were no loudly-dissenting opinions before we get too > > > far down this > > > >> > path. > > > >> > > > > >> > [1] > > > >> > > > > https://blueprints.launchpad.net/nova/+spec/use-service-catalog-for-endpoints > > > >> > [2] > > > >> > > > > https://github.com/openstack/nova/blob/7e7bdb198ed6412273e22dea72e37a6371fce8bd/nova/conf/glance.py#L27-L37 > > > >> > [3] > > > >> > > > > http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-04-27.log.html#t2017-04-27T20:38:29 > > > >> > > > > >> > Thanks, > > > >> > Eric Fried (efried) > > > >> > . > > > >> > > > > >> > > > > __________________________________________________________________________ > > > >> > OpenStack Development Mailing List (not for usage questions) > > > >> > Unsubscribe: > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > > >> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > >> > > > > >> > > > >> +openstack-operators > > > >> > > > >> -- > > > >> > > > >> Thanks, > > > >> > > > >> Matt > > > >> > > > >> > > > __________________________________________________________________________ > > > >> OpenStack Development Mailing List (not for usage questions) > > > >> Unsubscribe: > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > > >> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > >> > > > >> > > > >> _______________________________________________ > > > >> OpenStack-operators mailing list > > > >> OpenStack-operators@lists.openstack.org > > > >> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > > > > > > > > > > > > > > > > > > > > > > __________________________________________________________________________ > > > OpenStack Development Mailing List (not for usage questions) > > > Unsubscribe: > > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > __________________________________________________________________________ > > > OpenStack Development Mailing List (not for usage questions) > > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > _______________________________________________ > > OpenStack-operators mailing list > > OpenStack-operators@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > -- > @flaper87 > Flavio Percoco > _______________________________________________ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
signature.asc
Description: PGP signature
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
