Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Emilien Macchi]

On Mon, Jun 12, 2017 at 8:02 AM, Jiří Stránský  wrote:
> On 9.6.2017 18:51, Flavio Percoco wrote:
>>
>> A-ha, ok! I figured this was another option. In this case I guess we would
>> have 2 options:
>>
>> 1. Run confd + openstack service in side the container. My concern in this
>> case
>> would be that we'd have to run 2 services inside the container and
>> structure
>> things in a way we can monitor both services and make sure they are both
>> running. Nothing impossible but one more thing to do.
>
>
> I see several cons with this option:
>
> * Even if we do this in a sidecar container like Bogdan mentioned (which is
> better than running 2 "top-level" processes in a single container IMO), we
> still have to figure out when to restart the main service, IIUC. I see confd
> in daemon mode listens on the backend change and updates the conf files, but
> i can't find a mention that it would be able to restart services. Even if we
> implemented this auto-restarting in OpenStack services, we need to deal with
> services like MariaDB, Redis, ..., so additional wrappers might be needed to
> make this a generic solution.
>
> * Assuming we've solved the above, if we push a config change to etcd, all
> services get restarted at roughly the same time, possibly creating downtime
> or capacity issues.

I'm not sure galera1 container would share the same namespace for the
key/values of galera2 container (example); I think we would separate
namespaces by container names or something unique.

> * It complicates the reasoning about container lifecycle, as we have to
> start distinguishing between changes that don't require a new container
> (config change only) vs. changes which do require it (image content change).
> Mutable container config also hides this lifecycle from the operator -- the
> container changes on the inside without COE knowing about it, so any
> operator's queries to COE would look like no changes happened.
>
> I think ideally container config would be immutable, and every time we want
> to change anything, we'd do that via a roll out of a new set of containers.
> This way we have a single way of making changes to reason about, and when
> we're doing rolling updates, it shouldn't result in a downtime or tangible
> performance drop. (Not talking about migrating to a new major OpenStack
> release, which will still remain a special case in foreseeable future.)
>
>>
>> 2. Run confd `-onetime` and then run the openstack service.
>
>
> This sounds simpler both in terms of reasoning and technical complexity, so
> if we go with confd, i'd lean towards this option. We'd have to
> rolling-replace the containers from outside, but that's what k8s can take
> care of, and at least the operator can see what's happening on high level.
>
> The issues that Michał mentioned earlier still remain to be solved -- config
> versioning ("accidentally" picking up latest config), and how to supply
> config elements that differ per host.
>
> Also, it's probably worth diving a bit deeper into comparing `confd
> -onetime` and ConfigMaps...
>
>
> Jirka
>
>>
>>
>> Either would work but #2 means we won't have config files monitored and
>> the
>> container would have to be restarted to update the config files.
>>
>> Thanks, Doug.
>> Flavio
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Fox, Kevin M]

"Otherwise, -onetime will need to launch new containers each config change." 
You say that like its a bad thing

That sounds like a good feature to me. atomic containers. You always know the 
state of the system. As an Operator, I want to know which containers have the 
new config, which have the old, and which are stuck transitioning so I can fix 
brokenness. If its all hidden inside the containers, its much harder to Operate.

Thanks,
Kevin

From: Paul Belanger [pabelan...@redhat.com]
Sent: Friday, June 09, 2017 10:39 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
[helm] Configuration management with etcd / confd

On Fri, Jun 09, 2017 at 04:52:25PM +, Flavio Percoco wrote:
> On Fri, Jun 9, 2017 at 11:30 AM Britt Houser (bhouser) 
> wrote:
>
> > How does confd run inside the container?  Does this mean we’d need some
> > kind of systemd in every container which would spawn both confd and the
> > real service?  That seems like a very large architectural change.  But
> > maybe I’m misunderstanding it.
> >
> >
> Copying part of my reply to Doug's email:
>
> 1. Run confd + openstack service in side the container. My concern in this
> case
> would be that we'd have to run 2 services inside the container and structure
> things in a way we can monitor both services and make sure they are both
> running. Nothing impossible but one more thing to do.
>
> 2. Run confd `-onetime` and then run the openstack service.
>
>
> I either case, we could run confd as part of the entrypoint and have it run
> in
> background for the case #1 or just run it sequentially for case #2.
>
Both approached are valid, it all depends on your use case.  I suspect in the
case of openstack, you'll be running 2 daemons in your containers. Otherwise,
-onetime will need to launch new containers each config change.

>
> > Thx,
> > britt
> >
> > On 6/9/17, 9:04 AM, "Doug Hellmann"  wrote:
> >
> > Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:
> >
> > > Unless I'm missing something, to use confd with an OpenStack
> > deployment on
> > > k8s, we'll have to do something like this:
> > >
> > > * Deploy confd in every node where we may want to run a pod
> > (basically
> > > wvery node)
> >
> > Oh, no, no. That's not how it works at all.
> >
> > confd runs *inside* the containers. It's input files and command line
> > arguments tell it how to watch for the settings to be used just for
> > that
> > one container instance. It does all of its work (reading templates,
> > watching settings, HUPing services, etc.) from inside the container.
> >
> > The only inputs confd needs from outside of the container are the
> > connection information to get to etcd. Everything else can be put
> > in the system package for the application.
> >
> > Doug
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Fox, Kevin M]

+1 for putting confd in a side car with shared namespaces. much more k8s native.

Still generally -1 on the approach of using confd instead of configmaps. You 
loose all the atomicity that k8s provides with deployments. It breaks 
upgrade/downgrade behavior.

Would it be possible to have confd run in k8s, generate the configmaps, and 
push them to k8s? That might be even more k8s native.

Thanks,
Kevin

From: Bogdan Dobrelya [bdobr...@redhat.com]
Sent: Monday, June 12, 2017 1:07 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
[helm] Configuration management with etcd / confd

On 09.06.2017 18:51, Flavio Percoco wrote:
>
>
> On Fri, Jun 9, 2017 at 8:07 AM Doug Hellmann  <mailto:d...@doughellmann.com>> wrote:
>
> Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:
>
> > Unless I'm missing something, to use confd with an OpenStack
> deployment on
> > k8s, we'll have to do something like this:
> >
> > * Deploy confd in every node where we may want to run a pod (basically
> > wvery node)
>
> Oh, no, no. That's not how it works at all.
>
> confd runs *inside* the containers. It's input files and command line
> arguments tell it how to watch for the settings to be used just for that
> one container instance. It does all of its work (reading templates,
> watching settings, HUPing services, etc.) from inside the container.
>
> The only inputs confd needs from outside of the container are the
> connection information to get to etcd. Everything else can be put
> in the system package for the application.
>
>
> A-ha, ok! I figured this was another option. In this case I guess we
> would have 2 options:
>
> 1. Run confd + openstack service in side the container. My concern in
> this case
> would be that we'd have to run 2 services inside the container and structure
> things in a way we can monitor both services and make sure they are both
> running. Nothing impossible but one more thing to do.
>
> 2. Run confd `-onetime` and then run the openstack service.
>

A sidecar confd container running in a shared pod, which is having a
shared PID namespace with the managed service, would look much more
containerish. So confd could still HUP the service or signal it to be
restarted w/o baking itself into the container image. We have to deal
with the Pod abstraction as we want to be prepared for future
integration with k8s.

>
> Either would work but #2 means we won't have config files monitored and the
> container would have to be restarted to update the config files.
>
> Thanks, Doug.
> Flavio
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Bogdan Dobrelya]

On 12.06.2017 14:02, Jiří Stránský wrote:
> On 9.6.2017 18:51, Flavio Percoco wrote:
>> A-ha, ok! I figured this was another option. In this case I guess we
>> would
>> have 2 options:
>>
>> 1. Run confd + openstack service in side the container. My concern in
>> this
>> case
>> would be that we'd have to run 2 services inside the container and
>> structure
>> things in a way we can monitor both services and make sure they are both
>> running. Nothing impossible but one more thing to do.
> 
> I see several cons with this option:
> 
> * Even if we do this in a sidecar container like Bogdan mentioned (which
> is better than running 2 "top-level" processes in a single container
> IMO), we still have to figure out when to restart the main service,
> IIUC. I see confd in daemon mode listens on the backend change and
> updates the conf files, but i can't find a mention that it would be able
> to restart services. Even if we implemented this auto-restarting in
> OpenStack services, we need to deal with services like MariaDB, Redis,
> ..., so additional wrappers might be needed to make this a generic
> solution.

AFAIK, confd can send a signal to the process, so actions to be taken
are up to the service, either to refresh from its configs [0] or just
exit to be restarted by the container manager (which is docker-daemon,
currently, in tripleo).

Speaking of (tripleo specific) HA services you've mentioned, let
pacemaker to handle it on its own, but the same way, based on signals
sent to services by confd. For example, a galera service instance may
exit on the signal from the confd sidecar, then picked up by the next
monitor action causing it to be restarted by pcmk resources managemend
logic.

[0] https://bugs.launchpad.net/oslo-incubator/+bug/1276694

> 
> * Assuming we've solved the above, if we push a config change to etcd,
> all services get restarted at roughly the same time, possibly creating
> downtime or capacity issues.
> 
> * It complicates the reasoning about container lifecycle, as we have to
> start distinguishing between changes that don't require a new container
> (config change only) vs. changes which do require it (image content
> change). Mutable container config also hides this lifecycle from the
> operator -- the container changes on the inside without COE knowing
> about it, so any operator's queries to COE would look like no changes
> happened.
> 
> I think ideally container config would be immutable, and every time we
> want to change anything, we'd do that via a roll out of a new set of
> containers. This way we have a single way of making changes to reason
> about, and when we're doing rolling updates, it shouldn't result in a
> downtime or tangible performance drop. (Not talking about migrating to a
> new major OpenStack release, which will still remain a special case in
> foreseeable future.)
> 
>>
>> 2. Run confd `-onetime` and then run the openstack service.
> 
> This sounds simpler both in terms of reasoning and technical complexity,
> so if we go with confd, i'd lean towards this option. We'd have to
> rolling-replace the containers from outside, but that's what k8s can
> take care of, and at least the operator can see what's happening on high
> level.
> 
> The issues that Michał mentioned earlier still remain to be solved --
> config versioning ("accidentally" picking up latest config), and how to
> supply config elements that differ per host.
> 
> Also, it's probably worth diving a bit deeper into comparing `confd
> -onetime` and ConfigMaps...
> 
> 
> Jirka
> 
>>
>>
>> Either would work but #2 means we won't have config files monitored
>> and the
>> container would have to be restarted to update the config files.
>>
>> Thanks, Doug.
>> Flavio
>>
>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Flavio Percoco]

On 12/06/17 10:07 +0200, Bogdan Dobrelya wrote:

On 09.06.2017 18:51, Flavio Percoco wrote:



On Fri, Jun 9, 2017 at 8:07 AM Doug Hellmann mailto:d...@doughellmann.com>> wrote:

Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:

> Unless I'm missing something, to use confd with an OpenStack
deployment on
> k8s, we'll have to do something like this:
>
> * Deploy confd in every node where we may want to run a pod (basically
> wvery node)

Oh, no, no. That's not how it works at all.

confd runs *inside* the containers. It's input files and command line
arguments tell it how to watch for the settings to be used just for that
one container instance. It does all of its work (reading templates,
watching settings, HUPing services, etc.) from inside the container.

The only inputs confd needs from outside of the container are the
connection information to get to etcd. Everything else can be put
in the system package for the application.


A-ha, ok! I figured this was another option. In this case I guess we
would have 2 options:

1. Run confd + openstack service in side the container. My concern in
this case
would be that we'd have to run 2 services inside the container and structure
things in a way we can monitor both services and make sure they are both
running. Nothing impossible but one more thing to do.

2. Run confd `-onetime` and then run the openstack service.



A sidecar confd container running in a shared pod, which is having a
shared PID namespace with the managed service, would look much more
containerish. So confd could still HUP the service or signal it to be
restarted w/o baking itself into the container image. We have to deal
with the Pod abstraction as we want to be prepared for future
integration with k8s.


Yeah, this might work too. I was just trying to think of options that were
generic enough. In an k8s scenario, this should do the job.

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Jiří Stránský]

On 9.6.2017 18:51, Flavio Percoco wrote:

A-ha, ok! I figured this was another option. In this case I guess we would
have 2 options:

1. Run confd + openstack service in side the container. My concern in this
case
would be that we'd have to run 2 services inside the container and structure
things in a way we can monitor both services and make sure they are both
running. Nothing impossible but one more thing to do.


I see several cons with this option:

* Even if we do this in a sidecar container like Bogdan mentioned (which 
is better than running 2 "top-level" processes in a single container 
IMO), we still have to figure out when to restart the main service, 
IIUC. I see confd in daemon mode listens on the backend change and 
updates the conf files, but i can't find a mention that it would be able 
to restart services. Even if we implemented this auto-restarting in 
OpenStack services, we need to deal with services like MariaDB, Redis, 
..., so additional wrappers might be needed to make this a generic solution.


* Assuming we've solved the above, if we push a config change to etcd, 
all services get restarted at roughly the same time, possibly creating 
downtime or capacity issues.


* It complicates the reasoning about container lifecycle, as we have to 
start distinguishing between changes that don't require a new container 
(config change only) vs. changes which do require it (image content 
change). Mutable container config also hides this lifecycle from the 
operator -- the container changes on the inside without COE knowing 
about it, so any operator's queries to COE would look like no changes 
happened.


I think ideally container config would be immutable, and every time we 
want to change anything, we'd do that via a roll out of a new set of 
containers. This way we have a single way of making changes to reason 
about, and when we're doing rolling updates, it shouldn't result in a 
downtime or tangible performance drop. (Not talking about migrating to a 
new major OpenStack release, which will still remain a special case in 
foreseeable future.)




2. Run confd `-onetime` and then run the openstack service.


This sounds simpler both in terms of reasoning and technical complexity, 
so if we go with confd, i'd lean towards this option. We'd have to 
rolling-replace the containers from outside, but that's what k8s can 
take care of, and at least the operator can see what's happening on high 
level.


The issues that Michał mentioned earlier still remain to be solved -- 
config versioning ("accidentally" picking up latest config), and how to 
supply config elements that differ per host.


Also, it's probably worth diving a bit deeper into comparing `confd 
-onetime` and ConfigMaps...



Jirka




Either would work but #2 means we won't have config files monitored and the
container would have to be restarted to update the config files.

Thanks, Doug.
Flavio



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Bogdan Dobrelya]

On 09.06.2017 18:51, Flavio Percoco wrote:
> 
> 
> On Fri, Jun 9, 2017 at 8:07 AM Doug Hellmann  > wrote:
> 
> Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:
> 
> > Unless I'm missing something, to use confd with an OpenStack
> deployment on
> > k8s, we'll have to do something like this:
> >
> > * Deploy confd in every node where we may want to run a pod (basically
> > wvery node)
> 
> Oh, no, no. That's not how it works at all.
> 
> confd runs *inside* the containers. It's input files and command line
> arguments tell it how to watch for the settings to be used just for that
> one container instance. It does all of its work (reading templates,
> watching settings, HUPing services, etc.) from inside the container.
> 
> The only inputs confd needs from outside of the container are the
> connection information to get to etcd. Everything else can be put
> in the system package for the application.
> 
> 
> A-ha, ok! I figured this was another option. In this case I guess we
> would have 2 options:
> 
> 1. Run confd + openstack service in side the container. My concern in
> this case
> would be that we'd have to run 2 services inside the container and structure
> things in a way we can monitor both services and make sure they are both
> running. Nothing impossible but one more thing to do.
> 
> 2. Run confd `-onetime` and then run the openstack service.
> 

A sidecar confd container running in a shared pod, which is having a
shared PID namespace with the managed service, would look much more
containerish. So confd could still HUP the service or signal it to be
restarted w/o baking itself into the container image. We have to deal
with the Pod abstraction as we want to be prepared for future
integration with k8s.

> 
> Either would work but #2 means we won't have config files monitored and the
> container would have to be restarted to update the config files.
> 
> Thanks, Doug.
> Flavio
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-06-09 16:52:25 +:
> On Fri, Jun 9, 2017 at 11:30 AM Britt Houser (bhouser) 
> wrote:
> 
> > How does confd run inside the container?  Does this mean we’d need some
> > kind of systemd in every container which would spawn both confd and the
> > real service?  That seems like a very large architectural change.  But
> > maybe I’m misunderstanding it.
> >
> >
> Copying part of my reply to Doug's email:
> 
> 1. Run confd + openstack service in side the container. My concern in this
> case
> would be that we'd have to run 2 services inside the container and structure
> things in a way we can monitor both services and make sure they are both
> running. Nothing impossible but one more thing to do.
> 
> 2. Run confd `-onetime` and then run the openstack service.
> 
> 
> I either case, we could run confd as part of the entrypoint and have it run
> in
> background for the case #1 or just run it sequentially for case #2.

I think all of this is moot unless we can solve the case where we don't
know in advance of the deployment what settings to tell confd to look at
(what I've been calling the "cinder case", since that's where I saw it
come up first).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Paul Belanger]

On Fri, Jun 09, 2017 at 04:52:25PM +, Flavio Percoco wrote:
> On Fri, Jun 9, 2017 at 11:30 AM Britt Houser (bhouser) 
> wrote:
> 
> > How does confd run inside the container?  Does this mean we’d need some
> > kind of systemd in every container which would spawn both confd and the
> > real service?  That seems like a very large architectural change.  But
> > maybe I’m misunderstanding it.
> >
> >
> Copying part of my reply to Doug's email:
> 
> 1. Run confd + openstack service in side the container. My concern in this
> case
> would be that we'd have to run 2 services inside the container and structure
> things in a way we can monitor both services and make sure they are both
> running. Nothing impossible but one more thing to do.
> 
> 2. Run confd `-onetime` and then run the openstack service.
> 
> 
> I either case, we could run confd as part of the entrypoint and have it run
> in
> background for the case #1 or just run it sequentially for case #2.
> 
Both approached are valid, it all depends on your use case.  I suspect in the
case of openstack, you'll be running 2 daemons in your containers. Otherwise,
-onetime will need to launch new containers each config change.

> 
> > Thx,
> > britt
> >
> > On 6/9/17, 9:04 AM, "Doug Hellmann"  wrote:
> >
> > Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:
> >
> > > Unless I'm missing something, to use confd with an OpenStack
> > deployment on
> > > k8s, we'll have to do something like this:
> > >
> > > * Deploy confd in every node where we may want to run a pod
> > (basically
> > > wvery node)
> >
> > Oh, no, no. That's not how it works at all.
> >
> > confd runs *inside* the containers. It's input files and command line
> > arguments tell it how to watch for the settings to be used just for
> > that
> > one container instance. It does all of its work (reading templates,
> > watching settings, HUPing services, etc.) from inside the container.
> >
> > The only inputs confd needs from outside of the container are the
> > connection information to get to etcd. Everything else can be put
> > in the system package for the application.
> >
> > Doug
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Flavio Percoco]

On Fri, Jun 9, 2017 at 11:30 AM Britt Houser (bhouser) 
wrote:

> How does confd run inside the container?  Does this mean we’d need some
> kind of systemd in every container which would spawn both confd and the
> real service?  That seems like a very large architectural change.  But
> maybe I’m misunderstanding it.
>
>
Copying part of my reply to Doug's email:

1. Run confd + openstack service in side the container. My concern in this
case
would be that we'd have to run 2 services inside the container and structure
things in a way we can monitor both services and make sure they are both
running. Nothing impossible but one more thing to do.

2. Run confd `-onetime` and then run the openstack service.


I either case, we could run confd as part of the entrypoint and have it run
in
background for the case #1 or just run it sequentially for case #2.


> Thx,
> britt
>
> On 6/9/17, 9:04 AM, "Doug Hellmann"  wrote:
>
> Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:
>
> > Unless I'm missing something, to use confd with an OpenStack
> deployment on
> > k8s, we'll have to do something like this:
> >
> > * Deploy confd in every node where we may want to run a pod
> (basically
> > wvery node)
>
> Oh, no, no. That's not how it works at all.
>
> confd runs *inside* the containers. It's input files and command line
> arguments tell it how to watch for the settings to be used just for
> that
> one container instance. It does all of its work (reading templates,
> watching settings, HUPing services, etc.) from inside the container.
>
> The only inputs confd needs from outside of the container are the
> connection information to get to etcd. Everything else can be put
> in the system package for the application.
>
> Doug
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Flavio Percoco]

On Fri, Jun 9, 2017 at 8:07 AM Doug Hellmann  wrote:

> Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:
>
> > Unless I'm missing something, to use confd with an OpenStack deployment
> on
> > k8s, we'll have to do something like this:
> >
> > * Deploy confd in every node where we may want to run a pod (basically
> > wvery node)
>
> Oh, no, no. That's not how it works at all.
>
> confd runs *inside* the containers. It's input files and command line
> arguments tell it how to watch for the settings to be used just for that
> one container instance. It does all of its work (reading templates,
> watching settings, HUPing services, etc.) from inside the container.
>
> The only inputs confd needs from outside of the container are the
> connection information to get to etcd. Everything else can be put
> in the system package for the application.
>

A-ha, ok! I figured this was another option. In this case I guess we would
have 2 options:

1. Run confd + openstack service in side the container. My concern in this
case
would be that we'd have to run 2 services inside the container and structure
things in a way we can monitor both services and make sure they are both
running. Nothing impossible but one more thing to do.

2. Run confd `-onetime` and then run the openstack service.


Either would work but #2 means we won't have config files monitored and the
container would have to be restarted to update the config files.

Thanks, Doug.
Flavio
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Britt Houser (bhouser)]

How does confd run inside the container?  Does this mean we’d need some kind of 
systemd in every container which would spawn both confd and the real service?  
That seems like a very large architectural change.  But maybe I’m 
misunderstanding it.

Thx,
britt

On 6/9/17, 9:04 AM, "Doug Hellmann"  wrote:

Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:

> Unless I'm missing something, to use confd with an OpenStack deployment on
> k8s, we'll have to do something like this:
> 
> * Deploy confd in every node where we may want to run a pod (basically
> wvery node)

Oh, no, no. That's not how it works at all.

confd runs *inside* the containers. It's input files and command line
arguments tell it how to watch for the settings to be used just for that
one container instance. It does all of its work (reading templates,
watching settings, HUPing services, etc.) from inside the container.

The only inputs confd needs from outside of the container are the
connection information to get to etcd. Everything else can be put
in the system package for the application.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-06-08 22:28:05 +:

> Unless I'm missing something, to use confd with an OpenStack deployment on
> k8s, we'll have to do something like this:
> 
> * Deploy confd in every node where we may want to run a pod (basically
> wvery node)

Oh, no, no. That's not how it works at all.

confd runs *inside* the containers. It's input files and command line
arguments tell it how to watch for the settings to be used just for that
one container instance. It does all of its work (reading templates,
watching settings, HUPing services, etc.) from inside the container.

The only inputs confd needs from outside of the container are the
connection information to get to etcd. Everything else can be put
in the system package for the application.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Fox, Kevin M]

Flavio: I think your right. k8s configmaps and confd are doing very similar 
things. The one thing confd seems to add is dynamic templates on the host side. 
This is still accomplished in k8s with a sidecar watching for config changes 
with the templating engine in it and an emptyDir. or statically with an init 
container and an emptyDir (kolla-kubernetes does the latter)

But, for k8s, I actually prefer a fully atomic container config model, where 
you do a rolling upgrade any time you want to do a configmap change. k8s gives 
you the plumbing to do that and you can more easily roll forward/backward, 
allowing you versioning too.

So, I think your right. etcd/confd is more suited to the non k8s deployments.

Thanks,
Kevin

From: Flavio Percoco [fla...@redhat.com]
Sent: Thursday, June 08, 2017 3:28 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
[helm] Configuration management with etcd / confd



On Thu, Jun 8, 2017, 19:14 Doug Hellmann 
mailto:d...@doughellmann.com>> wrote:
Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> >>On 06.06.2017 18:08, Emilien Macchi wrote:
> >>>Another benefit is that confd will generate a configuration file when
> >>>the application will start. So if etcd is down *after* the app
> >>>startup, it shouldn't break the service restart if we don't ask confd
> >>>to re-generate the config. It's good for operators who were concerned
> >>>about the fact the infrastructure would rely on etcd. In that case, we
> >>>would only need etcd at the initial deployment (and during lifecycle
> >>>actions like upgrades, etc).
> >>>
> >>>The downside is that in the case of containers, they would still have
> >>>a configuration file within the container, and the whole goal of this
> >>>feature was to externalize configuration data and stop having
> >>>configuration files.
> >>
> >>It doesn't look a strict requirement. Those configs may (and should) be
> >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> >>something what *does* make embedded configs a strict requirement?..
> >
> >mmh, one thing I liked about this effort was possibility of stop 
> >bind-mounting
> >config files into the containers. I'd rather find a way to not need any
> >bindmount and have the services get their configs themselves.
>
> Probably sent too early!
>
> If we're not talking about OpenStack containers running in a COE, I guess this
> is fine. For k8s based deployments, I think I'd prefer having installers
> creating configmaps directly and use that. The reason is that depending on 
> files
> that are in the host is not ideal for these scenarios. I hate this idea 
> because
> it makes deployments inconsistent and I don't want that.
>
> Flavio
>

I'm not sure I understand how a configmap is any different from what is
proposed with confd in terms of deployment-specific data being added to
a container before it launches. Can you elaborate on that?


Unless I'm missing something, to use confd with an OpenStack deployment on k8s, 
we'll have to do something like this:

* Deploy confd in every node where we may want to run a pod (basically wvery 
node)
* Configure it to download all configs from etcd locally (we won't be able to 
download just some of them because we don't know what services may run in 
specific nodes. Except, perhaps, in the case of compute nodes and some other 
similar nodes)
* Enable hostpath volumes (iirc it's disabled by default) so that we can mount 
these files in the pod
* Run the pods and mount the files assuming the files are there.

All of the above is needed because  confd syncs files locally from etcd. Having 
a centralized place to manage these configs allows for controlling the 
deployment better. For example, if a configmap doesn't exist, then stop 
everything.

Not trying to be negative but rather explain why I think confd may not work 
well for the k8s based deployments. I think it's a good fit for the rest of the 
deployments.

Am I missing something? Am I overcomplicating things?

Flavio
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Flavio Percoco]

On Thu, Jun 8, 2017, 19:14 Doug Hellmann  wrote:

> Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> > On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> > >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> > >>On 06.06.2017 18:08, Emilien Macchi wrote:
> > >>>Another benefit is that confd will generate a configuration file when
> > >>>the application will start. So if etcd is down *after* the app
> > >>>startup, it shouldn't break the service restart if we don't ask confd
> > >>>to re-generate the config. It's good for operators who were concerned
> > >>>about the fact the infrastructure would rely on etcd. In that case, we
> > >>>would only need etcd at the initial deployment (and during lifecycle
> > >>>actions like upgrades, etc).
> > >>>
> > >>>The downside is that in the case of containers, they would still have
> > >>>a configuration file within the container, and the whole goal of this
> > >>>feature was to externalize configuration data and stop having
> > >>>configuration files.
> > >>
> > >>It doesn't look a strict requirement. Those configs may (and should) be
> > >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> > >>something what *does* make embedded configs a strict requirement?..
> > >
> > >mmh, one thing I liked about this effort was possibility of stop
> bind-mounting
> > >config files into the containers. I'd rather find a way to not need any
> > >bindmount and have the services get their configs themselves.
> >
> > Probably sent too early!
> >
> > If we're not talking about OpenStack containers running in a COE, I
> guess this
> > is fine. For k8s based deployments, I think I'd prefer having installers
> > creating configmaps directly and use that. The reason is that depending
> on files
> > that are in the host is not ideal for these scenarios. I hate this idea
> because
> > it makes deployments inconsistent and I don't want that.
> >
> > Flavio
> >
>
> I'm not sure I understand how a configmap is any different from what is
> proposed with confd in terms of deployment-specific data being added to
> a container before it launches. Can you elaborate on that?
>
>
Unless I'm missing something, to use confd with an OpenStack deployment on
k8s, we'll have to do something like this:

* Deploy confd in every node where we may want to run a pod (basically
wvery node)
* Configure it to download all configs from etcd locally (we won't be able
to download just some of them because we don't know what services may run
in specific nodes. Except, perhaps, in the case of compute nodes and some
other similar nodes)
* Enable hostpath volumes (iirc it's disabled by default) so that we can
mount these files in the pod
* Run the pods and mount the files assuming the files are there.

All of the above is needed because  confd syncs files locally from etcd.
Having a centralized place to manage these configs allows for controlling
the deployment better. For example, if a configmap doesn't exist, then stop
everything.

Not trying to be negative but rather explain why I think confd may not work
well for the k8s based deployments. I think it's a good fit for the rest of
the deployments.

Am I missing something? Am I overcomplicating things?

Flavio
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Flavio Percoco]

On Thu, Jun 8, 2017, 19:51 Steven Dake (stdake)  wrote:

> Flavio,
>
> Atleast for the kubernetes variant of kolla, bindmounting will always be
> used as this is fundamentally how configmaps operate.  In order to maintain
> maximum flexilbility and compatibility with kubernetes, I am not keen to
> try a non-configmap way of doing things.
>

I was referring​ to bindmounts of files that were created in the host and
reside in the host. While configmaps are bindmounts, they don't really live
in the host until the pod/container is created.

Flavio


> Regards
> -steve
>
> -Original Message-
> From: Flavio Percoco 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Thursday, June 8, 2017 at 9:23 AM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo]
> [kolla] [helm] Configuration management with etcd / confd
>
> On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> >On 06.06.2017 18:08, Emilien Macchi wrote:
> >> Another benefit is that confd will generate a configuration file
> when
> >> the application will start. So if etcd is down *after* the app
> >> startup, it shouldn't break the service restart if we don't ask
> confd
> >> to re-generate the config. It's good for operators who were
> concerned
> >> about the fact the infrastructure would rely on etcd. In that case,
> we
> >> would only need etcd at the initial deployment (and during lifecycle
> >> actions like upgrades, etc).
> >>
> >> The downside is that in the case of containers, they would still
> have
> >> a configuration file within the container, and the whole goal of
> this
> >> feature was to externalize configuration data and stop having
> >> configuration files.
> >
> >It doesn't look a strict requirement. Those configs may (and should)
> be
> >bind-mounted into containers, as hostpath volumes. Or, am I missing
> >something what *does* make embedded configs a strict requirement?..
>
> mmh, one thing I liked about this effort was possibility of stop
> bind-mounting
> config files into the containers. I'd rather find a way to not need any
> bindmount and have the services get their configs themselves.
>
> Flavio
>
>
> --
> @flaper87
> Flavio Percoco
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Fox, Kevin M]

Because tools to manipulate json and or yaml are very common.

Tools to manipulate a psudo ini file format that isn't standards compliant are 
not. :/

Thanks,
Kevin

From: Doug Hellmann [d...@doughellmann.com]
Sent: Thursday, June 08, 2017 1:39 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
[helm] Configuration management with etcd / confd

> On Jun 8, 2017, at 4:29 PM, Fox, Kevin M  wrote:
>
> That is possible. But, a yaml/json driver might still be good, regardless of 
> the mechanism used to transfer the file.
>
> So the driver abstraction still might be useful.

Why would it be useful to have oslo.config read files in more than one format?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

> On Jun 8, 2017, at 4:29 PM, Fox, Kevin M  wrote:
> 
> That is possible. But, a yaml/json driver might still be good, regardless of 
> the mechanism used to transfer the file.
> 
> So the driver abstraction still might be useful.

Why would it be useful to have oslo.config read files in more than one format?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Fox, Kevin M]

There are two issues conflated here maybe?

The first is a mechanism to use oslo.config to dump out example settings that 
could be loaded into a reference ConfigMap or etcd or something. I think there 
is a PS up for that.

The other is a way to get the data back into oslo.config.

etcd is one way.
using a ConfigMap to ship a file into a container to be read by oslo.config 
with a json/yaml/ini file driver is another.

Thanks,
Kevin

From: Emilien Macchi [emil...@redhat.com]
Sent: Thursday, June 08, 2017 1:20 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
[helm] Configuration management with etcd / confd

On Thu, Jun 8, 2017 at 8:49 PM, Doug Hellmann  wrote:
> Excerpts from Steven Dake (stdake)'s message of 2017-06-08 17:51:48 +:
>> Doug,
>>
>> In short, a configmap takes a bunch of config files, bundles them in a 
>> kubernetes object called a configmap, and then ships them to etcd.  When a 
>> pod is launched, the pod mounts the configmaps using tmpfs and the raw 
>> config files are available for use by the openstack services.
>
> That sounds like what confd does. Something puts data into one of
> several possible databases. confd takes it out and writes it to
> file(s) when the container starts. The app in the container reads
> the file(s).
>
> It sounds like configmaps would work well, too, it just doesn't
> sound like a fundamentally different solution.

Sorry for my lack of knowledge in ConfigMap but I'm trying to see how
we could bring pieces together.
Doug and I are currently investigating how oslo.config can be useful
to generate the parameters loaded by the application at startup,
without having to manage config with Puppet or Ansible.

If I understand correctly (and if not, please correct me, and maybe
propose something), we could use oslo.config to generate a portion of
ConfigMap (that can be imported in another ConfigMap iiuc) where we
would have parameters for one app.

Example with Keystone:

  apiVersion: v1
  kind: ConfigMap
  metadata:
name: keystone-config
namespace: DEFAULT
  data:
debug: true
rpc_backend: rabbit
... (parameters generated by oslo.config, and data fed by installers)

So iiuc we would give this file to k8s when deploying pods. Parameters
values would be automatically pushed into etcd, and used when
generating the configuration. Am I correct? (I need to understand if
we need to manually manage etcd key/values).

In that case, what deployments tools (like Kolla, TripleO, etc) would
expect from OpenStack to provide (tooling in oslo.config to generate
ConfigMap? etc.

Thanks for your help,

> Doug
>
>>
>> Operating on configmaps is much simpler and safer than using a different 
>> backing database for the configuration data.
>>
>> Hope the information helps.
>>
>> Ping me in #openstack-kolla if you have more questions.
>>
>> Regards
>> -steve
>>
>> -Original Message-
>> From: Doug Hellmann 
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> Date: Thursday, June 8, 2017 at 10:12 AM
>> To: openstack-dev 
>> Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
>>[helm] Configuration management with etcd / confd
>>
>> Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
>> > On 08/06/17 18:23 +0200, Flavio Percoco wrote:
>> > >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
>> > >>On 06.06.2017 18:08, Emilien Macchi wrote:
>> > >>>Another benefit is that confd will generate a configuration file 
>> when
>> > >>>the application will start. So if etcd is down *after* the app
>> > >>>startup, it shouldn't break the service restart if we don't ask 
>> confd
>> > >>>to re-generate the config. It's good for operators who were 
>> concerned
>> > >>>about the fact the infrastructure would rely on etcd. In that case, 
>> we
>> > >>>would only need etcd at the initial deployment (and during lifecycle
>> > >>>actions like upgrades, etc).
>> > >>>
>> > >>>The downside is that in the case of containers, they would still 
>> have
>> > >>>a configuration file within the container, and the whole goal of 
>> this
>> > >>>feature was to externalize configuration data and stop having
>> > >>>configuration files.
>> > >>
>> > >>I

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Emilien Macchi's message of 2017-06-08 22:20:34 +0200:
> On Thu, Jun 8, 2017 at 8:49 PM, Doug Hellmann  wrote:
> > Excerpts from Steven Dake (stdake)'s message of 2017-06-08 17:51:48 +:
> >> Doug,
> >>
> >> In short, a configmap takes a bunch of config files, bundles them in a 
> >> kubernetes object called a configmap, and then ships them to etcd.  When a 
> >> pod is launched, the pod mounts the configmaps using tmpfs and the raw 
> >> config files are available for use by the openstack services.
> >
> > That sounds like what confd does. Something puts data into one of
> > several possible databases. confd takes it out and writes it to
> > file(s) when the container starts. The app in the container reads
> > the file(s).
> >
> > It sounds like configmaps would work well, too, it just doesn't
> > sound like a fundamentally different solution.
> 
> Sorry for my lack of knowledge in ConfigMap but I'm trying to see how
> we could bring pieces together.
> Doug and I are currently investigating how oslo.config can be useful
> to generate the parameters loaded by the application at startup,
> without having to manage config with Puppet or Ansible.
> 
> If I understand correctly (and if not, please correct me, and maybe
> propose something), we could use oslo.config to generate a portion of
> ConfigMap (that can be imported in another ConfigMap iiuc) where we
> would have parameters for one app.
> 
> Example with Keystone:
> 
>   apiVersion: v1
>   kind: ConfigMap
>   metadata:
> name: keystone-config
> namespace: DEFAULT
>   data:
> debug: true
> rpc_backend: rabbit
> ... (parameters generated by oslo.config, and data fed by installers)
> 
> So iiuc we would give this file to k8s when deploying pods. Parameters
> values would be automatically pushed into etcd, and used when
> generating the configuration. Am I correct? (I need to understand if
> we need to manually manage etcd key/values).
> 
> In that case, what deployments tools (like Kolla, TripleO, etc) would
> expect from OpenStack to provide (tooling in oslo.config to generate
> ConfigMap? etc.
> 
> Thanks for your help,

Based on [1] I think the idea is to write the entire config file
for the service outside of the container, upload it to the configmap,
then configure the pod to create a volume and write the configmap
contents to the volume before launching the container. It's sort of like
nova's file-injection feature.

The approach seems appealing, although I don't fully understand the
issues others have raised with adding volumes to containers.

Doug

[1] 
https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#populate-a-volume-with-data-stored-in-a-configmap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Fox, Kevin M]

That is possible. But, a yaml/json driver might still be good, regardless of 
the mechanism used to transfer the file.

So the driver abstraction still might be useful.

Thanks,
Kevin

From: Doug Hellmann [d...@doughellmann.com]
Sent: Thursday, June 08, 2017 1:19 PM
To: openstack-dev
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla]
[helm] Configuration management with etcd / confd

Excerpts from Fox, Kevin M's message of 2017-06-08 20:08:25 +:
> Yeah, I think k8s configmaps might be a good config mechanism for k8s based 
> openstack deployment.
>
> One feature that might help which is related to the etcd plugin would be a 
> yaml/json plugin. It would allow more native looking configmaps.

We have at least 2 mechanisms for getting config files into containers
without such significant changes to oslo.config.  At this point I'm
not sure it's necessary to do the driver work at all.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Emilien Macchi]

On Thu, Jun 8, 2017 at 8:49 PM, Doug Hellmann  wrote:
> Excerpts from Steven Dake (stdake)'s message of 2017-06-08 17:51:48 +:
>> Doug,
>>
>> In short, a configmap takes a bunch of config files, bundles them in a 
>> kubernetes object called a configmap, and then ships them to etcd.  When a 
>> pod is launched, the pod mounts the configmaps using tmpfs and the raw 
>> config files are available for use by the openstack services.
>
> That sounds like what confd does. Something puts data into one of
> several possible databases. confd takes it out and writes it to
> file(s) when the container starts. The app in the container reads
> the file(s).
>
> It sounds like configmaps would work well, too, it just doesn't
> sound like a fundamentally different solution.

Sorry for my lack of knowledge in ConfigMap but I'm trying to see how
we could bring pieces together.
Doug and I are currently investigating how oslo.config can be useful
to generate the parameters loaded by the application at startup,
without having to manage config with Puppet or Ansible.

If I understand correctly (and if not, please correct me, and maybe
propose something), we could use oslo.config to generate a portion of
ConfigMap (that can be imported in another ConfigMap iiuc) where we
would have parameters for one app.

Example with Keystone:

  apiVersion: v1
  kind: ConfigMap
  metadata:
name: keystone-config
namespace: DEFAULT
  data:
debug: true
rpc_backend: rabbit
... (parameters generated by oslo.config, and data fed by installers)

So iiuc we would give this file to k8s when deploying pods. Parameters
values would be automatically pushed into etcd, and used when
generating the configuration. Am I correct? (I need to understand if
we need to manually manage etcd key/values).

In that case, what deployments tools (like Kolla, TripleO, etc) would
expect from OpenStack to provide (tooling in oslo.config to generate
ConfigMap? etc.

Thanks for your help,

> Doug
>
>>
>> Operating on configmaps is much simpler and safer than using a different 
>> backing database for the configuration data.
>>
>> Hope the information helps.
>>
>> Ping me in #openstack-kolla if you have more questions.
>>
>> Regards
>> -steve
>>
>> -Original Message-
>> From: Doug Hellmann 
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> Date: Thursday, June 8, 2017 at 10:12 AM
>> To: openstack-dev 
>> Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
>>[helm] Configuration management with etcd / confd
>>
>> Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
>> > On 08/06/17 18:23 +0200, Flavio Percoco wrote:
>> > >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
>> > >>On 06.06.2017 18:08, Emilien Macchi wrote:
>> > >>>Another benefit is that confd will generate a configuration file 
>> when
>> > >>>the application will start. So if etcd is down *after* the app
>> > >>>startup, it shouldn't break the service restart if we don't ask 
>> confd
>> > >>>to re-generate the config. It's good for operators who were 
>> concerned
>> > >>>about the fact the infrastructure would rely on etcd. In that case, 
>> we
>> > >>>would only need etcd at the initial deployment (and during lifecycle
>> > >>>actions like upgrades, etc).
>> > >>>
>> > >>>The downside is that in the case of containers, they would still 
>> have
>> > >>>a configuration file within the container, and the whole goal of 
>> this
>> > >>>feature was to externalize configuration data and stop having
>> > >>>configuration files.
>> > >>
>> > >>It doesn't look a strict requirement. Those configs may (and should) 
>> be
>> > >>bind-mounted into containers, as hostpath volumes. Or, am I missing
>> > >>something what *does* make embedded configs a strict requirement?..
>> > >
>> > >mmh, one thing I liked about this effort was possibility of stop 
>> bind-mounting
>> > >config files into the containers. I'd rather find a way to not need 
>> any
>> > >bindmount and have the services get their configs themselves.
>> >
>> > Probably sent too early!
>> >
>> &g

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Fox, Kevin M's message of 2017-06-08 20:08:25 +:
> Yeah, I think k8s configmaps might be a good config mechanism for k8s based 
> openstack deployment.
> 
> One feature that might help which is related to the etcd plugin would be a 
> yaml/json plugin. It would allow more native looking configmaps.

We have at least 2 mechanisms for getting config files into containers
without such significant changes to oslo.config.  At this point I'm
not sure it's necessary to do the driver work at all.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Fox, Kevin M]

Yeah, I think k8s configmaps might be a good config mechanism for k8s based 
openstack deployment.

One feature that might help which is related to the etcd plugin would be a 
yaml/json plugin. It would allow more native looking configmaps.

Thanks,
Kevin

From: Doug Hellmann [d...@doughellmann.com]
Sent: Thursday, June 08, 2017 11:49 AM
To: openstack-dev
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla]
[helm] Configuration management with etcd / confd

Excerpts from Steven Dake (stdake)'s message of 2017-06-08 17:51:48 +:
> Doug,
>
> In short, a configmap takes a bunch of config files, bundles them in a 
> kubernetes object called a configmap, and then ships them to etcd.  When a 
> pod is launched, the pod mounts the configmaps using tmpfs and the raw config 
> files are available for use by the openstack services.

That sounds like what confd does. Something puts data into one of
several possible databases. confd takes it out and writes it to
file(s) when the container starts. The app in the container reads
the file(s).

It sounds like configmaps would work well, too, it just doesn't
sound like a fundamentally different solution.

Doug

>
> Operating on configmaps is much simpler and safer than using a different 
> backing database for the configuration data.
>
> Hope the information helps.
>
> Ping me in #openstack-kolla if you have more questions.
>
> Regards
> -steve
>
> -Original Message-
> From: Doug Hellmann 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: Thursday, June 8, 2017 at 10:12 AM
> To: openstack-dev 
> Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla]  
>   [helm] Configuration management with etcd / confd
>
> Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> > On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> > >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> > >>On 06.06.2017 18:08, Emilien Macchi wrote:
> > >>>Another benefit is that confd will generate a configuration file when
> > >>>the application will start. So if etcd is down *after* the app
> > >>>startup, it shouldn't break the service restart if we don't ask confd
> > >>>to re-generate the config. It's good for operators who were concerned
> > >>>about the fact the infrastructure would rely on etcd. In that case, 
> we
> > >>>would only need etcd at the initial deployment (and during lifecycle
> > >>>actions like upgrades, etc).
> > >>>
> > >>>The downside is that in the case of containers, they would still have
> > >>>a configuration file within the container, and the whole goal of this
> > >>>feature was to externalize configuration data and stop having
> > >>>configuration files.
> > >>
> > >>It doesn't look a strict requirement. Those configs may (and should) 
> be
> > >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> > >>something what *does* make embedded configs a strict requirement?..
> > >
> > >mmh, one thing I liked about this effort was possibility of stop 
> bind-mounting
> > >config files into the containers. I'd rather find a way to not need any
> > >bindmount and have the services get their configs themselves.
> >
> > Probably sent too early!
> >
> > If we're not talking about OpenStack containers running in a COE, I 
> guess this
> > is fine. For k8s based deployments, I think I'd prefer having installers
> > creating configmaps directly and use that. The reason is that depending 
> on files
> > that are in the host is not ideal for these scenarios. I hate this idea 
> because
> > it makes deployments inconsistent and I don't want that.
> >
> > Flavio
> >
>
> I'm not sure I understand how a configmap is any different from what is
> proposed with confd in terms of deployment-specific data being added to
> a container before it launches. Can you elaborate on that?
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
&g

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Steven Dake (stdake)'s message of 2017-06-08 17:51:48 +:
> Doug,
> 
> In short, a configmap takes a bunch of config files, bundles them in a 
> kubernetes object called a configmap, and then ships them to etcd.  When a 
> pod is launched, the pod mounts the configmaps using tmpfs and the raw config 
> files are available for use by the openstack services.

That sounds like what confd does. Something puts data into one of
several possible databases. confd takes it out and writes it to
file(s) when the container starts. The app in the container reads
the file(s).

It sounds like configmaps would work well, too, it just doesn't
sound like a fundamentally different solution.

Doug

> 
> Operating on configmaps is much simpler and safer than using a different 
> backing database for the configuration data.
> 
> Hope the information helps.
> 
> Ping me in #openstack-kolla if you have more questions.
> 
> Regards
> -steve
> 
> -Original Message-
> From: Doug Hellmann 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: Thursday, June 8, 2017 at 10:12 AM
> To: openstack-dev 
> Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla]  
>   [helm] Configuration management with etcd / confd
> 
> Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> > On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> > >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> > >>On 06.06.2017 18:08, Emilien Macchi wrote:
> > >>>Another benefit is that confd will generate a configuration file when
> > >>>the application will start. So if etcd is down *after* the app
> > >>>startup, it shouldn't break the service restart if we don't ask confd
> > >>>to re-generate the config. It's good for operators who were concerned
> > >>>about the fact the infrastructure would rely on etcd. In that case, 
> we
> > >>>would only need etcd at the initial deployment (and during lifecycle
> > >>>actions like upgrades, etc).
> > >>>
> > >>>The downside is that in the case of containers, they would still have
> > >>>a configuration file within the container, and the whole goal of this
> > >>>feature was to externalize configuration data and stop having
> > >>>configuration files.
> > >>
> > >>It doesn't look a strict requirement. Those configs may (and should) 
> be
> > >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> > >>something what *does* make embedded configs a strict requirement?..
> > >
> > >mmh, one thing I liked about this effort was possibility of stop 
> bind-mounting
> > >config files into the containers. I'd rather find a way to not need any
> > >bindmount and have the services get their configs themselves.
> > 
> > Probably sent too early!
> > 
> > If we're not talking about OpenStack containers running in a COE, I 
> guess this
> > is fine. For k8s based deployments, I think I'd prefer having installers
> > creating configmaps directly and use that. The reason is that depending 
> on files
> > that are in the host is not ideal for these scenarios. I hate this idea 
> because
> > it makes deployments inconsistent and I don't want that.
> > 
> > Flavio
> > 
> 
> I'm not sure I understand how a configmap is any different from what is
> proposed with confd in terms of deployment-specific data being added to
> a container before it launches. Can you elaborate on that?
> 
> Doug
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Michał Jastrzębski]

On 8 June 2017 at 09:50, Michał Jastrzębski  wrote:
> On 8 June 2017 at 09:27, Flavio Percoco  wrote:
>> On 08/06/17 18:23 +0200, Flavio Percoco wrote:
>>>
>>> On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:

 On 06.06.2017 18:08, Emilien Macchi wrote:
>
> Another benefit is that confd will generate a configuration file when
> the application will start. So if etcd is down *after* the app
> startup, it shouldn't break the service restart if we don't ask confd
> to re-generate the config. It's good for operators who were concerned
> about the fact the infrastructure would rely on etcd. In that case, we
> would only need etcd at the initial deployment (and during lifecycle
> actions like upgrades, etc).
>
> The downside is that in the case of containers, they would still have
> a configuration file within the container, and the whole goal of this
> feature was to externalize configuration data and stop having
> configuration files.


 It doesn't look a strict requirement. Those configs may (and should) be
 bind-mounted into containers, as hostpath volumes. Or, am I missing
 something what *does* make embedded configs a strict requirement?..
>>>
>>>
>>> mmh, one thing I liked about this effort was possibility of stop
>>> bind-mounting
>>> config files into the containers. I'd rather find a way to not need any
>>> bindmount and have the services get their configs themselves.
>>
>>
>> Probably sent too early!
>>
>> If we're not talking about OpenStack containers running in a COE, I guess
>> this
>> is fine. For k8s based deployments, I think I'd prefer having installers
>> creating configmaps directly and use that. The reason is that depending on
>> files
>> that are in the host is not ideal for these scenarios. I hate this idea
>> because
>> it makes deployments inconsistent and I don't want that.
>
> Well, I disagree. If we're doing this we're essentially getting rid of
> "files" at all. It might actually be easier to handle from COE than
> configmap, as configmap has to be generated and when you get to host
> specific things it's quite a pain to handle. I, for one, would happily
> use cantral DB for config options if we define schema correctly.
>
> That being said defining schema correctly is quite a challenge. Few
> hard cases I see right now can be found in single use case - PCI
> Passthrough
>
> 1. I have multiple PCI devices in host, I need to specify list of them
> 2. PCI buses differes host to host, I need to specify groups of hosts
> that will share same bus configuration and reflect that in service
> config
>
> Maybe we should gather few of hard use cases like that and make sure
> we can address them in our config schema?

Speaking of hard use cases: here's another - config rolling upgrade +
config rollback. If we have single option in etcd, when service
restarts it automatically gets new config which creates funny edge
cases when you want to do rolling upgrade of config and some other
node fails->service restarts->config gets updated "accidentally".

>>
>> Flavio
>>
>> --
>> @flaper87
>> Flavio Percoco
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Steven Dake (stdake)]

Doug,

In short, a configmap takes a bunch of config files, bundles them in a 
kubernetes object called a configmap, and then ships them to etcd.  When a pod 
is launched, the pod mounts the configmaps using tmpfs and the raw config files 
are available for use by the openstack services.

Operating on configmaps is much simpler and safer than using a different 
backing database for the configuration data.

Hope the information helps.

Ping me in #openstack-kolla if you have more questions.

Regards
-steve


-Original Message-
From: Doug Hellmann 
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 

Date: Thursday, June 8, 2017 at 10:12 AM
To: openstack-dev 
Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla]    
[helm] Configuration management with etcd / confd

Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> >>On 06.06.2017 18:08, Emilien Macchi wrote:
> >>>Another benefit is that confd will generate a configuration file when
> >>>the application will start. So if etcd is down *after* the app
> >>>startup, it shouldn't break the service restart if we don't ask confd
> >>>to re-generate the config. It's good for operators who were concerned
> >>>about the fact the infrastructure would rely on etcd. In that case, we
> >>>would only need etcd at the initial deployment (and during lifecycle
> >>>actions like upgrades, etc).
> >>>
> >>>The downside is that in the case of containers, they would still have
> >>>a configuration file within the container, and the whole goal of this
> >>>feature was to externalize configuration data and stop having
> >>>configuration files.
> >>
> >>It doesn't look a strict requirement. Those configs may (and should) be
> >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> >>something what *does* make embedded configs a strict requirement?..
> >
> >mmh, one thing I liked about this effort was possibility of stop 
bind-mounting
> >config files into the containers. I'd rather find a way to not need any
> >bindmount and have the services get their configs themselves.
> 
> Probably sent too early!
> 
> If we're not talking about OpenStack containers running in a COE, I guess 
this
> is fine. For k8s based deployments, I think I'd prefer having installers
> creating configmaps directly and use that. The reason is that depending 
on files
> that are in the host is not ideal for these scenarios. I hate this idea 
because
> it makes deployments inconsistent and I don't want that.
> 
> Flavio
> 

I'm not sure I understand how a configmap is any different from what is
proposed with confd in terms of deployment-specific data being added to
a container before it launches. Can you elaborate on that?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Steven Dake (stdake)]

Flavio,

Atleast for the kubernetes variant of kolla, bindmounting will always be used 
as this is fundamentally how configmaps operate.  In order to maintain maximum 
flexilbility and compatibility with kubernetes, I am not keen to try a 
non-configmap way of doing things.

Regards
-steve

-Original Message-
From: Flavio Percoco 
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 

Date: Thursday, June 8, 2017 at 9:23 AM
To: "OpenStack Development Mailing List (not for usage questions)" 

Subject: Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] 
[helm] Configuration management with etcd / confd

On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
>On 06.06.2017 18:08, Emilien Macchi wrote:
>> Another benefit is that confd will generate a configuration file when
>> the application will start. So if etcd is down *after* the app
>> startup, it shouldn't break the service restart if we don't ask confd
>> to re-generate the config. It's good for operators who were concerned
>> about the fact the infrastructure would rely on etcd. In that case, we
>> would only need etcd at the initial deployment (and during lifecycle
>> actions like upgrades, etc).
>>
>> The downside is that in the case of containers, they would still have
>> a configuration file within the container, and the whole goal of this
>> feature was to externalize configuration data and stop having
>> configuration files.
>
>It doesn't look a strict requirement. Those configs may (and should) be
>bind-mounted into containers, as hostpath volumes. Or, am I missing
>something what *does* make embedded configs a strict requirement?..

mmh, one thing I liked about this effort was possibility of stop 
bind-mounting
config files into the containers. I'd rather find a way to not need any
bindmount and have the services get their configs themselves.

Flavio


-- 
@flaper87
Flavio Percoco


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

Excerpts from Flavio Percoco's message of 2017-06-08 18:27:51 +0200:
> On 08/06/17 18:23 +0200, Flavio Percoco wrote:
> >On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
> >>On 06.06.2017 18:08, Emilien Macchi wrote:
> >>>Another benefit is that confd will generate a configuration file when
> >>>the application will start. So if etcd is down *after* the app
> >>>startup, it shouldn't break the service restart if we don't ask confd
> >>>to re-generate the config. It's good for operators who were concerned
> >>>about the fact the infrastructure would rely on etcd. In that case, we
> >>>would only need etcd at the initial deployment (and during lifecycle
> >>>actions like upgrades, etc).
> >>>
> >>>The downside is that in the case of containers, they would still have
> >>>a configuration file within the container, and the whole goal of this
> >>>feature was to externalize configuration data and stop having
> >>>configuration files.
> >>
> >>It doesn't look a strict requirement. Those configs may (and should) be
> >>bind-mounted into containers, as hostpath volumes. Or, am I missing
> >>something what *does* make embedded configs a strict requirement?..
> >
> >mmh, one thing I liked about this effort was possibility of stop 
> >bind-mounting
> >config files into the containers. I'd rather find a way to not need any
> >bindmount and have the services get their configs themselves.
> 
> Probably sent too early!
> 
> If we're not talking about OpenStack containers running in a COE, I guess this
> is fine. For k8s based deployments, I think I'd prefer having installers
> creating configmaps directly and use that. The reason is that depending on 
> files
> that are in the host is not ideal for these scenarios. I hate this idea 
> because
> it makes deployments inconsistent and I don't want that.
> 
> Flavio
> 

I'm not sure I understand how a configmap is any different from what is
proposed with confd in terms of deployment-specific data being added to
a container before it launches. Can you elaborate on that?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Michał Jastrzębski]

On 8 June 2017 at 09:27, Flavio Percoco  wrote:
> On 08/06/17 18:23 +0200, Flavio Percoco wrote:
>>
>> On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:
>>>
>>> On 06.06.2017 18:08, Emilien Macchi wrote:

 Another benefit is that confd will generate a configuration file when
 the application will start. So if etcd is down *after* the app
 startup, it shouldn't break the service restart if we don't ask confd
 to re-generate the config. It's good for operators who were concerned
 about the fact the infrastructure would rely on etcd. In that case, we
 would only need etcd at the initial deployment (and during lifecycle
 actions like upgrades, etc).

 The downside is that in the case of containers, they would still have
 a configuration file within the container, and the whole goal of this
 feature was to externalize configuration data and stop having
 configuration files.
>>>
>>>
>>> It doesn't look a strict requirement. Those configs may (and should) be
>>> bind-mounted into containers, as hostpath volumes. Or, am I missing
>>> something what *does* make embedded configs a strict requirement?..
>>
>>
>> mmh, one thing I liked about this effort was possibility of stop
>> bind-mounting
>> config files into the containers. I'd rather find a way to not need any
>> bindmount and have the services get their configs themselves.
>
>
> Probably sent too early!
>
> If we're not talking about OpenStack containers running in a COE, I guess
> this
> is fine. For k8s based deployments, I think I'd prefer having installers
> creating configmaps directly and use that. The reason is that depending on
> files
> that are in the host is not ideal for these scenarios. I hate this idea
> because
> it makes deployments inconsistent and I don't want that.

Well, I disagree. If we're doing this we're essentially getting rid of
"files" at all. It might actually be easier to handle from COE than
configmap, as configmap has to be generated and when you get to host
specific things it's quite a pain to handle. I, for one, would happily
use cantral DB for config options if we define schema correctly.

That being said defining schema correctly is quite a challenge. Few
hard cases I see right now can be found in single use case - PCI
Passthrough

1. I have multiple PCI devices in host, I need to specify list of them
2. PCI buses differes host to host, I need to specify groups of hosts
that will share same bus configuration and reflect that in service
config

Maybe we should gather few of hard use cases like that and make sure
we can address them in our config schema?

>
> Flavio
>
> --
> @flaper87
> Flavio Percoco
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Flavio Percoco]

On 08/06/17 18:23 +0200, Flavio Percoco wrote:

On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:

On 06.06.2017 18:08, Emilien Macchi wrote:

Another benefit is that confd will generate a configuration file when
the application will start. So if etcd is down *after* the app
startup, it shouldn't break the service restart if we don't ask confd
to re-generate the config. It's good for operators who were concerned
about the fact the infrastructure would rely on etcd. In that case, we
would only need etcd at the initial deployment (and during lifecycle
actions like upgrades, etc).

The downside is that in the case of containers, they would still have
a configuration file within the container, and the whole goal of this
feature was to externalize configuration data and stop having
configuration files.


It doesn't look a strict requirement. Those configs may (and should) be
bind-mounted into containers, as hostpath volumes. Or, am I missing
something what *does* make embedded configs a strict requirement?..


mmh, one thing I liked about this effort was possibility of stop bind-mounting
config files into the containers. I'd rather find a way to not need any
bindmount and have the services get their configs themselves.


Probably sent too early!

If we're not talking about OpenStack containers running in a COE, I guess this
is fine. For k8s based deployments, I think I'd prefer having installers
creating configmaps directly and use that. The reason is that depending on files
that are in the host is not ideal for these scenarios. I hate this idea because
it makes deployments inconsistent and I don't want that.

Flavio

--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Flavio Percoco]

On 07/06/17 12:04 +0200, Bogdan Dobrelya wrote:

On 06.06.2017 18:08, Emilien Macchi wrote:

Another benefit is that confd will generate a configuration file when
the application will start. So if etcd is down *after* the app
startup, it shouldn't break the service restart if we don't ask confd
to re-generate the config. It's good for operators who were concerned
about the fact the infrastructure would rely on etcd. In that case, we
would only need etcd at the initial deployment (and during lifecycle
actions like upgrades, etc).

The downside is that in the case of containers, they would still have
a configuration file within the container, and the whole goal of this
feature was to externalize configuration data and stop having
configuration files.


It doesn't look a strict requirement. Those configs may (and should) be
bind-mounted into containers, as hostpath volumes. Or, am I missing
something what *does* make embedded configs a strict requirement?..


mmh, one thing I liked about this effort was possibility of stop bind-mounting
config files into the containers. I'd rather find a way to not need any
bindmount and have the services get their configs themselves.

Flavio


--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Doug Hellmann]

> On Jun 7, 2017, at 7:20 AM, Emilien Macchi  wrote:
> 
> On Tue, Jun 6, 2017 at 6:08 PM, Emilien Macchi  > wrote:
>> Following-up the session that we had in Boston:
>> https://etherpad.openstack.org/p/BOS-forum-future-of-configuration-management
>> 
>> Here's an update on where we are and what is being done.
>> 
>> 
>> == Machine Readable Sample Config
>> 
>> Ben's spec has been merged: https://review.openstack.org/#/c/440835/
>> And also the code which implements it: 
>> https://review.openstack.org/#/c/451081/
>> He's now working on the documentation on how to use the feature.
>> 
>> $ oslo-config-generator --namespace keystone --formal yaml > keystone.yaml
>> 
>> Here's an example of the output for Keystone config: https://clbin.com/EAfFM
>> This feature was asked at the PTG, and it's already done!
>> 
>> 
>> == Pluggable drivers for oslo.config
>> 
>> Doug's spec has been well written and the feedback from Summit and the
>> review was taken in account: https://review.openstack.org/#/c/454897/
>> It's now paused because we think we could use confd (with etcd driver)
>> to generate configuration files.
>> 
>> Imagine the work done by Ben in Machine Readable Sample Config, that
>> would allow us to generate Confd templates for all services (Keystone,
>> Nova, etc) via a tool provided in oslo.config with all the options
>> available for a namespace.
> 
> I'm also wondering if we could use oslo-config-generate directly to
> generate confd templates, with a new format. So we would have ini,
> yaml, json and confd.
> "confd" format would be useful when building rpms that we ship in containers.
> "yaml" format would be useful for installers to expose the options
> directly to the User Interface, so we know which params OpenStack
> provide and we could re-use the data to push it into etcd.
> 
> Would it make sense?

I did think about making oslo-config-generator also take the YAML file as input 
instead of scanning plugins, and then including all the output formats in the 
single command. I haven’t looked to see how much extra complexity that would 
add.

> 
>> We could have packaging builds (e.g. RDO distgit) using the tooling
>> when building packages so we could ship confd templates in addition of
>> ini configuration files.
>> When services would start (e.g. in containers), confd would generate
>> configuration files from the templates that is part of the container,
>> and read the values from etcd.
>> 
>> The benefit of doing this, is that a very little work is required in
>> oslo.config to make this happen (only a tool to generate confd
>> templates). It could be a first iteration.
>> Another benefit is that confd will generate a configuration file when
>> the application will start. So if etcd is down *after* the app
>> startup, it shouldn't break the service restart if we don't ask confd
>> to re-generate the config. It's good for operators who were concerned
>> about the fact the infrastructure would rely on etcd. In that case, we
>> would only need etcd at the initial deployment (and during lifecycle
>> actions like upgrades, etc).
>> 
>> The downside is that in the case of containers, they would still have
>> a configuration file within the container, and the whole goal of this
>> feature was to externalize configuration data and stop having
>> configuration files.
>> 
>> 
>> == What's next
>> 
>> I see 2 short-term actions that we can work on:
>> 
>> 1) Decide if whether or not confd solution would be acceptable for a
>> start. I'm asking Kolla, TripleO, Helm, Ansible projects if they would
>> be willing to use this feature. I'm also asking operators to give
>> feedback on it.
>> 
>> 2) Investigate how to expose parameters generated by Ben's work on
>> Machine Readable Sample Config to the users (without having to
>> manually maintain all options) - I think this has to be solved on the
>> installers side, but I might be wrong; and also investigate how to
>> populate parameters data into etcd. This tool could be provided by
>> oslo.config probably.
>> 
>> 
>> 
>> Any feedback from folks working on installers or from operators would
>> be more than welcome!
>> 
>> Thanks,
>> --
>> Emilien Macchi
> 
> 
> 
> -- 
> Emilien Macchi
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org 
> ?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
> 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Emilien Macchi]

On Tue, Jun 6, 2017 at 6:08 PM, Emilien Macchi  wrote:
> Following-up the session that we had in Boston:
> https://etherpad.openstack.org/p/BOS-forum-future-of-configuration-management
>
> Here's an update on where we are and what is being done.
>
>
> == Machine Readable Sample Config
>
> Ben's spec has been merged: https://review.openstack.org/#/c/440835/
> And also the code which implements it: 
> https://review.openstack.org/#/c/451081/
> He's now working on the documentation on how to use the feature.
>
> $ oslo-config-generator --namespace keystone --formal yaml > keystone.yaml
>
> Here's an example of the output for Keystone config: https://clbin.com/EAfFM
> This feature was asked at the PTG, and it's already done!
>
>
> == Pluggable drivers for oslo.config
>
> Doug's spec has been well written and the feedback from Summit and the
> review was taken in account: https://review.openstack.org/#/c/454897/
> It's now paused because we think we could use confd (with etcd driver)
> to generate configuration files.
>
> Imagine the work done by Ben in Machine Readable Sample Config, that
> would allow us to generate Confd templates for all services (Keystone,
> Nova, etc) via a tool provided in oslo.config with all the options
> available for a namespace.

I'm also wondering if we could use oslo-config-generate directly to
generate confd templates, with a new format. So we would have ini,
yaml, json and confd.
"confd" format would be useful when building rpms that we ship in containers.
"yaml" format would be useful for installers to expose the options
directly to the User Interface, so we know which params OpenStack
provide and we could re-use the data to push it into etcd.

Would it make sense?

> We could have packaging builds (e.g. RDO distgit) using the tooling
> when building packages so we could ship confd templates in addition of
> ini configuration files.
> When services would start (e.g. in containers), confd would generate
> configuration files from the templates that is part of the container,
> and read the values from etcd.
>
> The benefit of doing this, is that a very little work is required in
> oslo.config to make this happen (only a tool to generate confd
> templates). It could be a first iteration.
> Another benefit is that confd will generate a configuration file when
> the application will start. So if etcd is down *after* the app
> startup, it shouldn't break the service restart if we don't ask confd
> to re-generate the config. It's good for operators who were concerned
> about the fact the infrastructure would rely on etcd. In that case, we
> would only need etcd at the initial deployment (and during lifecycle
> actions like upgrades, etc).
>
> The downside is that in the case of containers, they would still have
> a configuration file within the container, and the whole goal of this
> feature was to externalize configuration data and stop having
> configuration files.
>
>
> == What's next
>
> I see 2 short-term actions that we can work on:
>
> 1) Decide if whether or not confd solution would be acceptable for a
> start. I'm asking Kolla, TripleO, Helm, Ansible projects if they would
> be willing to use this feature. I'm also asking operators to give
> feedback on it.
>
> 2) Investigate how to expose parameters generated by Ben's work on
> Machine Readable Sample Config to the users (without having to
> manually maintain all options) - I think this has to be solved on the
> installers side, but I might be wrong; and also investigate how to
> populate parameters data into etcd. This tool could be provided by
> oslo.config probably.
>
>
>
> Any feedback from folks working on installers or from operators would
> be more than welcome!
>
> Thanks,
> --
> Emilien Macchi



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Bogdan Dobrelya]

On 06.06.2017 18:08, Emilien Macchi wrote:
> Following-up the session that we had in Boston:
> https://etherpad.openstack.org/p/BOS-forum-future-of-configuration-management
> 
> Here's an update on where we are and what is being done.
> 
> 
> == Machine Readable Sample Config
> 
> Ben's spec has been merged: https://review.openstack.org/#/c/440835/
> And also the code which implements it: 
> https://review.openstack.org/#/c/451081/
> He's now working on the documentation on how to use the feature.
> 
> $ oslo-config-generator --namespace keystone --formal yaml > keystone.yaml
> 
> Here's an example of the output for Keystone config: https://clbin.com/EAfFM
> This feature was asked at the PTG, and it's already done!
> 

Great progress, well done!

> 
> == Pluggable drivers for oslo.config
> 
> Doug's spec has been well written and the feedback from Summit and the
> review was taken in account: https://review.openstack.org/#/c/454897/
> It's now paused because we think we could use confd (with etcd driver)
> to generate configuration files.
> 
> Imagine the work done by Ben in Machine Readable Sample Config, that
> would allow us to generate Confd templates for all services (Keystone,
> Nova, etc) via a tool provided in oslo.config with all the options
> available for a namespace.
> We could have packaging builds (e.g. RDO distgit) using the tooling
> when building packages so we could ship confd templates in addition of
> ini configuration files.
> When services would start (e.g. in containers), confd would generate
> configuration files from the templates that is part of the container,
> and read the values from etcd.

This sounds like a plan to start following just immediately :-)

> 
> The benefit of doing this, is that a very little work is required in
> oslo.config to make this happen (only a tool to generate confd
> templates). It could be a first iteration.

And that's really great as we need no to reimplement confd for oslo.config.

> Another benefit is that confd will generate a configuration file when
> the application will start. So if etcd is down *after* the app
> startup, it shouldn't break the service restart if we don't ask confd
> to re-generate the config. It's good for operators who were concerned
> about the fact the infrastructure would rely on etcd. In that case, we
> would only need etcd at the initial deployment (and during lifecycle
> actions like upgrades, etc).
> 
> The downside is that in the case of containers, they would still have
> a configuration file within the container, and the whole goal of this
> feature was to externalize configuration data and stop having
> configuration files.

It doesn't look a strict requirement. Those configs may (and should) be
bind-mounted into containers, as hostpath volumes. Or, am I missing
something what *does* make embedded configs a strict requirement?..

> 
> 
> == What's next
> 
> I see 2 short-term actions that we can work on:
> 
> 1) Decide if whether or not confd solution would be acceptable for a
> start. I'm asking Kolla, TripleO, Helm, Ansible projects if they would
> be willing to use this feature. I'm also asking operators to give
> feedback on it.
> 
> 2) Investigate how to expose parameters generated by Ben's work on
> Machine Readable Sample Config to the users (without having to
> manually maintain all options) - I think this has to be solved on the
> installers side, but I might be wrong; and also investigate how to
> populate parameters data into etcd. This tool could be provided by
> oslo.config probably.
> 
> 
> 
> Any feedback from folks working on installers or from operators would
> be more than welcome!
> 
> Thanks,
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [deployment] [oslo] [ansible] [tripleo] [kolla] [helm] Configuration management with etcd / confd

[Emilien Macchi]

Following-up the session that we had in Boston:
https://etherpad.openstack.org/p/BOS-forum-future-of-configuration-management

Here's an update on where we are and what is being done.


== Machine Readable Sample Config

Ben's spec has been merged: https://review.openstack.org/#/c/440835/
And also the code which implements it: https://review.openstack.org/#/c/451081/
He's now working on the documentation on how to use the feature.

$ oslo-config-generator --namespace keystone --formal yaml > keystone.yaml

Here's an example of the output for Keystone config: https://clbin.com/EAfFM
This feature was asked at the PTG, and it's already done!


== Pluggable drivers for oslo.config

Doug's spec has been well written and the feedback from Summit and the
review was taken in account: https://review.openstack.org/#/c/454897/
It's now paused because we think we could use confd (with etcd driver)
to generate configuration files.

Imagine the work done by Ben in Machine Readable Sample Config, that
would allow us to generate Confd templates for all services (Keystone,
Nova, etc) via a tool provided in oslo.config with all the options
available for a namespace.
We could have packaging builds (e.g. RDO distgit) using the tooling
when building packages so we could ship confd templates in addition of
ini configuration files.
When services would start (e.g. in containers), confd would generate
configuration files from the templates that is part of the container,
and read the values from etcd.

The benefit of doing this, is that a very little work is required in
oslo.config to make this happen (only a tool to generate confd
templates). It could be a first iteration.
Another benefit is that confd will generate a configuration file when
the application will start. So if etcd is down *after* the app
startup, it shouldn't break the service restart if we don't ask confd
to re-generate the config. It's good for operators who were concerned
about the fact the infrastructure would rely on etcd. In that case, we
would only need etcd at the initial deployment (and during lifecycle
actions like upgrades, etc).

The downside is that in the case of containers, they would still have
a configuration file within the container, and the whole goal of this
feature was to externalize configuration data and stop having
configuration files.


== What's next

I see 2 short-term actions that we can work on:

1) Decide if whether or not confd solution would be acceptable for a
start. I'm asking Kolla, TripleO, Helm, Ansible projects if they would
be willing to use this feature. I'm also asking operators to give
feedback on it.

2) Investigate how to expose parameters generated by Ben's work on
Machine Readable Sample Config to the users (without having to
manually maintain all options) - I think this has to be solved on the
installers side, but I might be wrong; and also investigate how to
populate parameters data into etcd. This tool could be provided by
oslo.config probably.



Any feedback from folks working on installers or from operators would
be more than welcome!

Thanks,
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev