On Wed, Mar 22, 2017 at 08:24:52AM -0600, Alex Schultz wrote:
> On Wed, Mar 22, 2017 at 7:58 AM, Paul Belanger <pabelan...@redhat.com> wrote:
> > On Tue, Mar 21, 2017 at 05:53:35PM -0600, Alex Schultz wrote:
> >> On Tue, Mar 21, 2017 at 5:35 PM, John Dickinson <m...@not.mn> wrote:
> >> >
> >> >
> >> > On 21 Mar 2017, at 15:34, Alex Schultz wrote:
> >> >
> >> >> On Tue, Mar 21, 2017 at 3:45 PM, John Dickinson <m...@not.mn> wrote:
> >> >>> I've been following this thread, but I must admit I seem to have
> >> >>> missed something.
> >> >>>
> >> >>> What problem is being solved by storing per-server service
> >> >>> configuration options in an external distributed CP system that is
> >> >>> currently not possible with the existing pattern of using local text
> >> >>> files?
> >> >>>
> >> >>
> >> >> This effort is partially to help the path to containerization where we
> >> >> are delivering the service code via container but don't want to
> >> >> necessarily deliver the configuration in the same fashion. It's about
> >> >> ease of configuration where moving service -> config files (on many
> >> >> hosts/containers) to service -> config via etcd (single source
> >> >> cluster). It's also about an alternative to configuration management
> >> >> where today we have many tools handling the files in various ways
> >> >> (templates, from repo, via code providers) and trying to come to a
> >> >> more unified way of representing the configuration such that the end
> >> >> result is the same for every deployment tool. All tools load configs
> >> >> into $place and services can be configured to talk to $place. It
> >> >> should be noted that configuration files won't go away because many of
> >> >> the companion services still rely on them (rabbit/mysql/apache/etc) so
> >> >> we're really talking about services that currently use oslo.
> >> >
> >> > Thanks for the explanation!
> >> >
> >> > So in the future, you expect a node in a clustered OpenStack service to
> >> > be deployed and run as a container, and then that node queries a
> >> > centralized etcd (or other) k/v store to load config options. And other
> >> > services running in the (container? cluster?) will load config from
> >> > local text files managed in some other way.
> >>
> >> No the goal is in the etcd mode, that it may not be necessary to load
> >> the config files locally at all. That being said there would still be
> >> support for having some configuration from a file and optionally
> >> provide a kv store as another config point. 'service --config-file
> >> /etc/service/service.conf --config-etcd proto://ip:port/slug'
> >>
> > Hmm, not sure I like this. Having a service magically read from 2 different
> > configuration source at run time, merge them, and reload, seems overly
> > complicated. And even harder to debug.
> >
>
> That's something inherently supported by oslo.config today. We even do
> it for dist provided packaging (I also don't like it, but it's an
> established pattern).
>
> >> >
> >> > No wait. It's not the *services* that will load the config from a kv
> >> > store--it's the config management system? So in the process of deploying
> >> > a new container instance of a particular service, the deployment tool
> >> > will pull the right values out of the kv system and inject those into
> >> > the container, I'm guessing as a local text file that the service loads
> >> > as normal?
> >> >
> >>
> >> No the thought is to have the services pull their configs from the kv
> >> store via oslo.config. The point is hopefully to not require
> >> configuration files at all for containers. The container would get
> >> where to pull it's configs from (ie. http://11.1.1.1:2730/magic/ or
> >> /etc/myconfigs/). At that point it just becomes another place to load
> >> configurations from via oslo.config. Configuration management comes
> >> in as a way to load the configs either as a file or into etcd. Many
> >> operators (and deployment tools) are already using some form of
> >> configuration management so if we can integrate in a kv store output
> >> option, adoption becomes much easier than making everyone start from
> >> scratch.
> >>
> >> > This means you could have some (OpenStack?) service for inventory
> >> > management (like Karbor) that is seeding the kv store, the cloud
> >> > infrastructure software itself is "cloud aware" and queries the central
> >> > distributed kv system for the correct-right-now config options, and the
> >> > cloud service itself gets all the benefits of dynamic scaling of
> >> > available hardware resources. That's pretty cool. Add hardware to the
> >> > inventory, the cloud infra itself expands to make it available. Hardware
> >> > fails, and the cloud infra resizes to adjust. Apps running on the infra
> >> > keep doing their thing consuming the resources. It's clouds all the way
> >> > down :-)
> >> >
> >> > Despite sounding pretty interesting, it also sounds like a lot of extra
> >> > complexity. Maybe it's worth it. I don't know.
> >> >
> >>
> >> Yea there's extra complexity at least in the
> >> deployment/management/monitoring of the new service or maybe not.
> >> Keeping configuration files synced across 1000s of nodes (or
> >> containers) can be just as hard however.
> >>
> > Please correct me if I am wrong, because I still have my container training
> > wheels on. I understand the need for etcd, and operators to write their
> > configuration into it. Why I am struggling with still, is why you need
> > oslo.config to support it. There is nothing stopping an operator today from
> > using etcd / confd in a container, right? I can only imagine countless
> > other
> > services that run in containers using them.
> >
>
> We want oslo.config to support it as a source for configuration.
> Dealing with files in containers is complicated. If we can remove the
> requirement to munge configurations for containers,
> deployment/updating containers becomes easier. The service container
> becomes a single artifact to be deployed with less moving parts which
> helps reduce complexity and errors. The process for moving a single
> container artifact is a lot easier than moving container and updating
> configurations based on where it's landing.
>
I can understand that from a deployment position. However, it seems the
established pattern for services like apache2, nginx2 are to use config files
inside the container. As far as I can tell, none of these services have native
integration with etcd.
> > Why do we, openstack, need to write our own custom thing and be different in
> > this regard? Why do we need our services to talk directly to etcd? When
> > things
> > like apache2, nginx, other non-openstack service just use etcd / confd?
> >
>
> Openstack service configuration is inherently complex. Since we
> cannot control how supporting services do their configuration
> (apache/mysql/nginx/etc) we aren't talking about how to try and solve
> for that today. What we're looking to focus on is how can we simplify
> the containerization of openstack services and the related
> configurations that we need to apply. Today we have several different
> deployment solutions that attempt to solve for configuring openstack
> services (via templates, static files, providers, etc). What we're
> trying to discuss is how to continue to support the existing as well
> as brainstorming about moving to etcd as the destination from
> configuration management and the consumption by openstack services.
>
Agree, I am happy this discussion is happening.
My only objection would be making openstack a special snowflake when it comes to
configurations of services. As an operator, I'd like to homogeneous
configuration across all services. This where I think confd is already the
(established?) winner. Existing services seem to be recommending operators
configure their servers as such.
> > From reading the thread, it seems the issue is more about keeping files
> > inside
> > the container out of the container, which makes them easier to maintain.
> >
> > And this is the part I need help with. If I was to do a POC this afternoon,
> > inside a container. I would do the following:
> >
> > Create container with keystone
> > Create etcd service
> > Add confd into container
> > - mkdir -p /etc/confd/{conf.d,templates}
> > - vi /etc/confd/conf.d/myconfig.toml
> > - vi /etc/confd/templates/myconfig.conf.tmpl
> > - start confd
> >
>
> This is one possible deployment strategy but it does not support the
> existing deployment tools. It's a valid solution, but why introduce
> confd in the container if you don't have to because the service can
> read directly from etcd? Additionally how does debugging that look
> like when you need to go look at the configurations that get written
> out. Do you have to go inspect each container and what happens when
> the confd sync process breaks down? The thought by having etcd be the
> source of truth is that operators look in etcd for the configuration
> for live/staged configurations. We'd probably want to help with
> tooling to support this, but I think it would be easier to have a tool
> to operate on a single etcd cluster than try and go poll X number of
> containers on Z hosts.
>
So, these are great questions, I don't have all the answers but maybe we apply
them to an operator that is using etcd/confd with apache2?
As for source of truth, I personally don't see etcd as that. Let me explain:
As an operator, my source of truth is git, because it allows me to go back in
time and see all of my changes. As I understand etcd, it just as the realtime
configuration. So, I would not look to etcd for this configuration (I am
assuming you can) but based on your example above, I better understand.
> > I would use ansible or puppet or Dockerfile to properly setup myconfg.toml
> > for
> > keystone. Same with myconf.conf.tmpl, it would be a for d in dict: for key,
> > value in d thing, to glob down any thing from etcd into ini format.
> >
> > This does mean you still need to manage confd templates however, but this is
> > okay, because if I deploy any other application with etcd, this is how I
> > would
> > it.
>
> We're not talking about not supporting such a configuration. In fact
> the integration of etcd with oslo.config means this would still be a
> valid deployment method. You'd still need something that would know
> how to load oslo.config items into etcd which is part of this
> discussion.
>
> >
> > Which gets me to my final question. How are you proposing we configure
> > apache2
> > or nginx with etcd? I can only assume it is using something like the process
> > above?
>
> We're not proposing that. Since there's already legacy ways of
> configuring those (and who knows if the operator actually wants to
> containerize those), we're only scoping the discussion around
> openstack services at the moment to see what that looks like.
>
Understood. Even for discussions, we could assume everything is containerized,
this makes the discussions around etc/confd/oslo.config more realistic I think.
Because while we might only be talking about deploying openstack into
containers, I am sure an operator already has things in containers and could
even have etcd.
> >
> >> > Thanks again for the explanation.
> >> >
> >> >
> >> > --John
> >> >
> >> >
> >> >
> >> >
> >> >>
> >> >> Thanks,
> >> >> -Alex
> >> >>
> >> >>>
> >> >>> --John
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On 21 Mar 2017, at 14:26, Davanum Srinivas wrote:
> >> >>>
> >> >>>> Jay,
> >> >>>>
> >> >>>> the /v3alpha HTTP API (grpc-gateway) supports watch
> >> >>>> https://coreos.com/etcd/docs/latest/dev-guide/apispec/swagger/rpc.swagger.json
> >> >>>>
> >> >>>> -- Dims
> >> >>>>
> >> >>>> On Tue, Mar 21, 2017 at 5:22 PM, Jay Pipes <jaypi...@gmail.com> wrote:
> >> >>>>> On 03/21/2017 04:29 PM, Clint Byrum wrote:
> >> >>>>>>
> >> >>>>>> Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:
> >> >>>>>>>
> >> >>>>>>> Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:
> >> >>>>>>>>
> >> >>>>>>>> On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow
> >> >>>>>>>> <harlo...@fastmail.com>
> >> >>>>>>>> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> * How does reloading work (does it)?
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> No. There is nothing that we can do in oslo that will make
> >> >>>>>>>> services
> >> >>>>>>>> magically reload configuration. It's also unclear to me if that's
> >> >>>>>>>> something to do. In a containerized environment, wouldn't it be
> >> >>>>>>>> simpler to deploy new services? Otherwise, supporting signal based
> >> >>>>>>>> reload as we do today should be trivial.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> Reloading works today with files, that's why the question is
> >> >>>>>>> important
> >> >>>>>>> to think through. There is a special flag to set on options that
> >> >>>>>>> are
> >> >>>>>>> "mutable" and then there are functions within oslo.config to
> >> >>>>>>> reload.
> >> >>>>>>> Those are usually triggered when a service gets a SIGHUP or
> >> >>>>>>> something
> >> >>>>>>> similar.
> >> >>>>>>>
> >> >>>>>>> We need to decide what happens to a service's config when that API
> >> >>>>>>> is used and the backend is etcd. Maybe nothing, because every time
> >> >>>>>>> any config option is accessed the read goes all the way through to
> >> >>>>>>> etcd? Maybe a warning is logged because we don't support reloads?
> >> >>>>>>> Maybe an error is logged? Or maybe we flush the local cache and
> >> >>>>>>> start
> >> >>>>>>> reading from etcd on future accesses?
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>> etcd provides the ability to "watch" keys. So one would start a
> >> >>>>>> thread
> >> >>>>>> that just watches the keys you want to reload on, and when they
> >> >>>>>> change
> >> >>>>>> that thread will see a response and can reload appropriately.
> >> >>>>>>
> >> >>>>>> https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html
> >> >>>>>
> >> >>>>>
> >> >>>>> Yep. Unfortunately, you won't be able to start an eventlet
> >> >>>>> greenthread to
> >> >>>>> watch an etcd3/gRPC key. The python grpc library is incompatible with
> >> >>>>> eventlet/gevent's monkeypatching technique and causes a complete
> >> >>>>> program
> >> >>>>> hang if you try to communicate with the etcd3 server from a
> >> >>>>> greenlet. Fun!
> >> >>>>>
> >> >>>>> So, either use etcd2 (the no-longer-being-worked-on HTTP API) or
> >> >>>>> don't use
> >> >>>>> eventlet in your client service.
> >> >>>>>
> >> >>>>> Best,
> >> >>>>> -jay
> >> >>>>>
> >> >>>>>
> >> >>>>> __________________________________________________________________________
> >> >>>>> OpenStack Development Mailing List (not for usage questions)
> >> >>>>> Unsubscribe:
> >> >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Davanum Srinivas :: https://twitter.com/dims
> >> >>>>
> >> >>>> __________________________________________________________________________
> >> >>>> OpenStack Development Mailing List (not for usage questions)
> >> >>>> Unsubscribe:
> >> >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>>
> >> >>> __________________________________________________________________________
> >> >>> OpenStack Development Mailing List (not for usage questions)
> >> >>> Unsubscribe:
> >> >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>>
> >> >>
> >> >> __________________________________________________________________________
> >> >> OpenStack Development Mailing List (not for usage questions)
> >> >> Unsubscribe:
> >> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >> > __________________________________________________________________________
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
