Re: [openstack-dev] [TripleO] [Puppet] Deploying OpenStack with Puppet modules on Docker with Heat
On 06/08/15 06:29, Dan Prince wrote: Hi, There is a lot of interest in getting support for container based deployment within TripleO and many different ideas and opinions on how to go about doing that. One idea on the table is to use Heat to help orchestrate the deployment of docker containers. This would work similar to our tripleo-heat -templates implementation except that when using docker you would swap in a nested stack template that would configure containers on baremetal. We've even got a nice example that shows what a containerized TripleO overcloud might look like here [1]. The approach outlines how you might use kolla docker containers alongside of the tripleo-heat-templates to do this sort of deployment. This is all cool stuff but one area of concern is how we do the actual configuration of the containers. The above implementation relies on passing environment variables into kolla built docker containers which then self configure all the required config files and start the service. This sounds like a start... but creating (and maintaining) another from scratch OpenStack configuration tool isn't high on my list of things to spend time on. Sure there is already a kolla community helping to build and maintain this configuration tooling (mostly thinking config files here) but this sounds a bit like what tripleo -image-elements initially tried to do and it turns out there are much more capable configuration tools out there. Since we are already using a good bit of Puppet in tripleo-heat -templates the idea came up that we would try to configure Docker containers using Puppet. Again, here there are several ideas in the Puppet community with regards to how docker might best be configured with Puppet. Keeping those in mind we've been throwing some ideas out on an etherpad here [2] that describes using Heat for orchestration, Puppet for configuration, and Kolla docker images for containers. A quick outline of the approach is: -Extend the heat-container-agent [3] that runs os-collect-config and all the required hooks we require for deployment. This includes docker -compute, bash scripts, and Puppet. NOTE: As described in the etherpad I've taken to using DIB to build this container. I found this to be faster from a TripleO development baseline. -To create config files the heat-container-agent would run a puppet manifest for a given role and generate a directory tree of config files (/var/lib/etc-data for example). -We then run a docker-compose software deployment that mounts those configuration file(s) into a read only volume and uses them to start the containerized service. The approach could look something like this [4]. This nice thing about this is that it requires no modification to OpenStack Puppet modules. We can use those today, as-is. Additionally, although Puppet runs in the agent container we've created a mechanism to set all the resources to noop mode except for those that generate config files. And lastly, we can use exactly the same role manifest for docker that we do for baremetal. Lots of re-use here... and although we are disabling a lot of Puppet functionality in setting all the non-config resources to noop the Kolla containers already do some of that stuff for us (starting services, etc.). This sounds like a viable approach, my only suggestion would be for there to be an option to build a puppet-container-agent which contains only puppet (not the heat hook too). This could allow the openstack-puppet and kolla communities to collaborate quickly without pulling in the whole tripleo stack. Then some simple docker-compose (or whatever) templates could be written to bring up puppet-container-agent with a given manifest hieradata, then bring up a single node kolla container based cloud. This would be useful for CI and local development of the puppet modules supporting containers. Then heat-container-agent can be puppet-container-agent plus the heat hook tooling. All that said (and trying to keep this short) we've still got a bit of work to do around wiring up externally created config files to kolla build docker containers. A couple of issues are: -The external config file mechanism for Kolla containers only seems to support a single config file. Some services (Neutron) can have multiple files. Could we extend the external config support to use multiple files? -If a service has multiple files kolla may need to adjust its service startup script to use multiple files. Perhaps a conf.d approach would work here? -We are missing published version of some key kolla containers. Namely openvswitch and the neutron-openvswitch-agent for starters but I'd also like to have a Ceilometer agent and SNMP agent container as well so we have feature parity with the non-docker compute role. Once we have solutions for the above I think we'll be very close to a fully dockerized compute role with TripleO heat templates. From there we can expand the idea to cover other roles
[openstack-dev] [TripleO] [Puppet] Deploying OpenStack with Puppet modules on Docker with Heat
Hi, There is a lot of interest in getting support for container based deployment within TripleO and many different ideas and opinions on how to go about doing that. One idea on the table is to use Heat to help orchestrate the deployment of docker containers. This would work similar to our tripleo-heat -templates implementation except that when using docker you would swap in a nested stack template that would configure containers on baremetal. We've even got a nice example that shows what a containerized TripleO overcloud might look like here [1]. The approach outlines how you might use kolla docker containers alongside of the tripleo-heat-templates to do this sort of deployment. This is all cool stuff but one area of concern is how we do the actual configuration of the containers. The above implementation relies on passing environment variables into kolla built docker containers which then self configure all the required config files and start the service. This sounds like a start... but creating (and maintaining) another from scratch OpenStack configuration tool isn't high on my list of things to spend time on. Sure there is already a kolla community helping to build and maintain this configuration tooling (mostly thinking config files here) but this sounds a bit like what tripleo -image-elements initially tried to do and it turns out there are much more capable configuration tools out there. Since we are already using a good bit of Puppet in tripleo-heat -templates the idea came up that we would try to configure Docker containers using Puppet. Again, here there are several ideas in the Puppet community with regards to how docker might best be configured with Puppet. Keeping those in mind we've been throwing some ideas out on an etherpad here [2] that describes using Heat for orchestration, Puppet for configuration, and Kolla docker images for containers. A quick outline of the approach is: -Extend the heat-container-agent [3] that runs os-collect-config and all the required hooks we require for deployment. This includes docker -compute, bash scripts, and Puppet. NOTE: As described in the etherpad I've taken to using DIB to build this container. I found this to be faster from a TripleO development baseline. -To create config files the heat-container-agent would run a puppet manifest for a given role and generate a directory tree of config files (/var/lib/etc-data for example). -We then run a docker-compose software deployment that mounts those configuration file(s) into a read only volume and uses them to start the containerized service. The approach could look something like this [4]. This nice thing about this is that it requires no modification to OpenStack Puppet modules. We can use those today, as-is. Additionally, although Puppet runs in the agent container we've created a mechanism to set all the resources to noop mode except for those that generate config files. And lastly, we can use exactly the same role manifest for docker that we do for baremetal. Lots of re-use here... and although we are disabling a lot of Puppet functionality in setting all the non-config resources to noop the Kolla containers already do some of that stuff for us (starting services, etc.). All that said (and trying to keep this short) we've still got a bit of work to do around wiring up externally created config files to kolla build docker containers. A couple of issues are: -The external config file mechanism for Kolla containers only seems to support a single config file. Some services (Neutron) can have multiple files. Could we extend the external config support to use multiple files? -If a service has multiple files kolla may need to adjust its service startup script to use multiple files. Perhaps a conf.d approach would work here? -We are missing published version of some key kolla containers. Namely openvswitch and the neutron-openvswitch-agent for starters but I'd also like to have a Ceilometer agent and SNMP agent container as well so we have feature parity with the non-docker compute role. Once we have solutions for the above I think we'll be very close to a fully dockerized compute role with TripleO heat templates. From there we can expand the idea to cover other roles within the tripleo-heat -templates too. I'll stop there for now. Any ideas and thoughts appreciated. Dan - [1] https://review.openstack.org/#/c/178840/ (Containerized TripleO Overcloud.) [2] https://etherpad.openstack.org/p/tripleo-docker-puppet [3] http://git.openstack.org/cgit/openstack/heat -templates/log/hot/software-config/heat-container-agent [4] https://review.openstack.org/#/c/209505/ (Docker compute role configured via Puppet) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] [Puppet] Deploying OpenStack with Puppet modules on Docker with Heat
On Wed, Aug 5, 2015 at 1:29 PM, Dan Prince dpri...@redhat.com wrote: ...snip... -The external config file mechanism for Kolla containers only seems to support a single config file. Some services (Neutron) can have multiple files. Could we extend the external config support to use multiple files? Yes! I would actually prefer a rework. We implemented that in a hurry but if you look at the initial commit messages we knew it was a stop-gap until a better idea came along. We need to have some way to do this dynamically or at least in a more readable way. I had a thought of laying down a json file with the contents being which files to copy/move/change permissions on and reading that in. In that way the config-external script never actually changes and the deploy tool can determine which files would get pulled in by the way it lays down that json file. I rejected using a '*' match for security reasons but also because some configs need to go to different places. In the case of neutron, the neutron.conf will be in /etc/neutron and the ml2_conf.ini will be in /etc/neutron/plugins/ml2. So I don't think '*' matching will work. We are open to ideas! __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev