[openstack-dev] [Heat] image requirements for Heat software config
Hi all, I have been experimenting a lot with Heat software config to check out what works today, and to think about potential next steps. I've also worked on an internal project where we are leveraging software config as of the Icehouse release. I think what we can do now from a user's perspective in a HOT template is really nice and resonates well also with customers I've talked to. One of the points where we are constantly having issues, and also got some push back from customers, are the requirements on the in-instance tools and the process of building base images. One observation is that building a base image with all the right stuff inside sometimes is a brittle process; the other point is that a lot of customers do not like a lot of requirements on their base images. They want to maintain one set of corporate base images, with as little modification on top as possible. Regarding the process of building base images, the currently documented way [1] of using diskimage-builder turns out to be a bit unstable sometimes. Not because diskimage-builder is unstable, but probably because it pulls in components from a couple of sources: #1 we have a dependency on implementation of the Heat engine of course (So this is not pulled in to the image building process, but the dependency is there) #2 we depend on features in python-heatclient (and other python-* clients) #3 we pull in implementation from the heat-templates repo #4 we depend on tripleo-image-elements #5 we depend on os-collect-config, os-refresh-config and os-apply-config #6 we depend on diskimage-builder itself Heat itself and python-heatclient are reasonably well in synch because there is a release process for both, so we can tell users with some certainty that a feature will work with release X of OpenStack and Heat and version x.z.y of python-heatclient. For the other 4 sources, success sometimes depends on the time of day when you try to build an image (depending on what changes are currently included in each repo). So basically there does not seem to be a consolidated release process across all that is currently needed for software config. The ideal solution would be to have one self-contained package that is easy to install on various distributions (an rpm, deb, MSI ...). Secondly, it would be ideal to not have to bake additional things into the image but doing bootstrapping during instance creation based on an existing cloud-init enabled image. For that we would have to strip requirements down to a bare minimum required for software config. One thing that comes to my mind is the cirros software config example [2] that Steven Hardy created. It is admittedly no up to what one could do with an image built according to [1] but on the other hand is really slick, whereas [1] installs a whole set of things into the image (some of which do not really seem to be needed for software config). Another issue that comes to mind: what about operating systems not supported by diskimage-builder (Windows), or other hypervisor platforms? Any, not really suggestions from my side but more observations and thoughts. I wanted to share those and raise some discussion on possible options. Regards, Thomas [1] https://github.com/openstack/heat-templates/blob/master/hot/software-config/elements/README.rst [2] https://github.com/openstack/heat-templates/tree/master/hot/software-config/example-templates/cirros-example ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] image requirements for Heat software config
On 10/14/2014 01:13 PM, Thomas Spatzier wrote: > > Hi all, > > I have been experimenting a lot with Heat software config to check out > what works today, and to think about potential next steps. > I've also worked on an internal project where we are leveraging software > config as of the Icehouse release. > > I think what we can do now from a user's perspective in a HOT template is > really nice and resonates well also with customers I've talked to. > One of the points where we are constantly having issues, and also got some > push back from customers, are the requirements on the in-instance tools and > the process of building base images. > One observation is that building a base image with all the right stuff > inside sometimes is a brittle process; the other point is that a lot of > customers do not like a lot of requirements on their base images. They want > to maintain one set of corporate base images, with as little modification > on top as possible. > > Regarding the process of building base images, the currently documented way > [1] of using diskimage-builder turns out to be a bit unstable sometimes. > Not because diskimage-builder is unstable, but probably because it pulls in > components from a couple of sources: > #1 we have a dependency on implementation of the Heat engine of course (So > this is not pulled in to the image building process, but the dependency is > there) > #2 we depend on features in python-heatclient (and other python-* clients) > #3 we pull in implementation from the heat-templates repo > #4 we depend on tripleo-image-elements > #5 we depend on os-collect-config, os-refresh-config and os-apply-config > #6 we depend on diskimage-builder itself > > Heat itself and python-heatclient are reasonably well in synch because > there is a release process for both, so we can tell users with some > certainty that a feature will work with release X of OpenStack and Heat and > version x.z.y of python-heatclient. For the other 4 sources, success > sometimes depends on the time of day when you try to build an image > (depending on what changes are currently included in each repo). So > basically there does not seem to be a consolidated release process across > all that is currently needed for software config. > > The ideal solution would be to have one self-contained package that is easy > to install on various distributions (an rpm, deb, MSI ...). It would be simple enough to make an RPM metapackage that just installs the deps. The definition of self-contained I'm using here is "one install command" and not "has its own vendored python and every module". > Secondly, it would be ideal to not have to bake additional things into the > image but doing bootstrapping during instance creation based on an existing > cloud-init enabled image. For that we would have to strip requirements down > to a bare minimum required for software config. One thing that comes to my > mind is the cirros software config example [2] that Steven Hardy created. > It is admittedly no up to what one could do with an image built according > to [1] but on the other hand is really slick, whereas [1] installs a whole > set of things into the image (some of which do not really seem to be needed > for software config). I like this option much better, actually. Idoubt many deployers would have complaints since cloud-init is pretty much standard. The downside here is that it wouldn't be all that feasible to include bootstrap scripts for every platform. Maybe it would be enough to have the ability to bootstrap one or two popular distros (Ubuntu, Fedora, Cent, etc) and accept patches for other platforms. > > Another issue that comes to mind: what about operating systems not > supported by diskimage-builder (Windows), or other hypervisor platforms? > > Any, not really suggestions from my side but more observations and > thoughts. I wanted to share those and raise some discussion on possible > options. > > Regards, > Thomas > > [1] > https://github.com/openstack/heat-templates/blob/master/hot/software-config/elements/README.rst > [2] > https://github.com/openstack/heat-templates/tree/master/hot/software-config/example-templates/cirros-example > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] image requirements for Heat software config
Excerpts from Thomas Spatzier's message of 2014-10-14 10:13:27 -0700: > > Hi all, > > I have been experimenting a lot with Heat software config to check out > what works today, and to think about potential next steps. > I've also worked on an internal project where we are leveraging software > config as of the Icehouse release. > > I think what we can do now from a user's perspective in a HOT template is > really nice and resonates well also with customers I've talked to. > One of the points where we are constantly having issues, and also got some > push back from customers, are the requirements on the in-instance tools and > the process of building base images. > One observation is that building a base image with all the right stuff > inside sometimes is a brittle process; the other point is that a lot of > customers do not like a lot of requirements on their base images. They want > to maintain one set of corporate base images, with as little modification > on top as possible. > > Regarding the process of building base images, the currently documented way > [1] of using diskimage-builder turns out to be a bit unstable sometimes. > Not because diskimage-builder is unstable, but probably because it pulls in > components from a couple of sources: > #1 we have a dependency on implementation of the Heat engine of course (So > this is not pulled in to the image building process, but the dependency is > there) > #2 we depend on features in python-heatclient (and other python-* clients) > #3 we pull in implementation from the heat-templates repo > #4 we depend on tripleo-image-elements > #5 we depend on os-collect-config, os-refresh-config and os-apply-config > #6 we depend on diskimage-builder itself > > Heat itself and python-heatclient are reasonably well in synch because > there is a release process for both, so we can tell users with some > certainty that a feature will work with release X of OpenStack and Heat and > version x.z.y of python-heatclient. For the other 4 sources, success > sometimes depends on the time of day when you try to build an image > (depending on what changes are currently included in each repo). So > basically there does not seem to be a consolidated release process across > all that is currently needed for software config. > I don't really understand why a "consolidated release process across all" would be desired or needed. #3 is pretty odd. You're pulling in templates from the examples repo? For #4-#6, those are all on pypi and released on a regular basis. Build yourself a bandersnatch mirror and you'll have locally controlled access to them which should eliminate any reliability issues. > The ideal solution would be to have one self-contained package that is easy > to install on various distributions (an rpm, deb, MSI ...). > Secondly, it would be ideal to not have to bake additional things into the > image but doing bootstrapping during instance creation based on an existing > cloud-init enabled image. For that we would have to strip requirements down > to a bare minimum required for software config. One thing that comes to my > mind is the cirros software config example [2] that Steven Hardy created. > It is admittedly no up to what one could do with an image built according > to [1] but on the other hand is really slick, whereas [1] installs a whole > set of things into the image (some of which do not really seem to be needed > for software config). The agent problem is one reason I've been drifting away from Heat for software configuration, and toward Ansible. Mind you, I wrote os-collect-config to have as few dependencies as possible as one attempt around this problem. Still it isn't capable enough to do the job on its own, so you end up needing os-apply-config and then os-refresh-config to tie the two together. Ansible requires sshd, and python, with a strong recommendation for sudo. These are all things that pretty much every Linux distribution is going to have available. > > Another issue that comes to mind: what about operating systems not > supported by diskimage-builder (Windows), or other hypervisor platforms? > There is a windows-diskimage-builder: https://git.openstack.org/cgit/stackforge/windows-diskimage-builder diskimage-builder can produce raw images, so that should be convertible to pretty much any other hypervisor's preferred disk format. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] image requirements for Heat software config
On 15/10/14 06:13, Thomas Spatzier wrote: Hi all, I have been experimenting a lot with Heat software config to check out what works today, and to think about potential next steps. I've also worked on an internal project where we are leveraging software config as of the Icehouse release. I think what we can do now from a user's perspective in a HOT template is really nice and resonates well also with customers I've talked to. One of the points where we are constantly having issues, and also got some push back from customers, are the requirements on the in-instance tools and the process of building base images. One observation is that building a base image with all the right stuff inside sometimes is a brittle process; the other point is that a lot of customers do not like a lot of requirements on their base images. They want to maintain one set of corporate base images, with as little modification on top as possible. Regarding the process of building base images, the currently documented way [1] of using diskimage-builder turns out to be a bit unstable sometimes. Not because diskimage-builder is unstable, but probably because it pulls in components from a couple of sources: #1 we have a dependency on implementation of the Heat engine of course (So this is not pulled in to the image building process, but the dependency is there) #2 we depend on features in python-heatclient (and other python-* clients) #3 we pull in implementation from the heat-templates repo #4 we depend on tripleo-image-elements #5 we depend on os-collect-config, os-refresh-config and os-apply-config #6 we depend on diskimage-builder itself Heat itself and python-heatclient are reasonably well in synch because there is a release process for both, so we can tell users with some certainty that a feature will work with release X of OpenStack and Heat and version x.z.y of python-heatclient. For the other 4 sources, success sometimes depends on the time of day when you try to build an image (depending on what changes are currently included in each repo). So basically there does not seem to be a consolidated release process across all that is currently needed for software config. The ideal solution would be to have one self-contained package that is easy to install on various distributions (an rpm, deb, MSI ...). Secondly, it would be ideal to not have to bake additional things into the image but doing bootstrapping during instance creation based on an existing cloud-init enabled image. For that we would have to strip requirements down to a bare minimum required for software config. One thing that comes to my mind is the cirros software config example [2] that Steven Hardy created. It is admittedly no up to what one could do with an image built according to [1] but on the other hand is really slick, whereas [1] installs a whole set of things into the image (some of which do not really seem to be needed for software config). Building an image from git repos was the best chance of having a single set of instructions which works for most cases, since the tools were not packaged for debian derived distros. This seems to be improving though; the whole build stack is now packaged for Debian Unstable, Testing and also Ubuntu Utopic (which isn't released yet). Another option is switching the default instructions to installing from pip rather than git, but that still gets into distro-specific quirks which complicate the instructions. Until these packages are on the recent releases of common distros then we'll be stuck in this slightly awkward situation. I wrote a cloud-init boot script to install the agents from packages from a pristine Fedora 20 [3] and it seems like a reasonable approach for when building a custom image isn't practical. Somebody submitting the equivalent for Debian and Ubuntu would be most welcome. We need to decide whether *everything* should be packaged or if some things can be delivered by cloud-init on boot (os-collect-config.conf template, 55-heat-config, the actual desired config hook...) I'm all for there being documentation for the different ways of getting the agent and hooks onto a running server for a given distro. I think the hot-guide would be the best place to do that, and I've been making a start on that recently [4][5] (help welcome!). The README in [1] should eventually refer to the hot-guide once it is published so we're not maintaining multiple build instructions. Another issue that comes to mind: what about operating systems not supported by diskimage-builder (Windows), or other hypervisor platforms? The Cloudbase folk have contributed some useful cloudbase-init templates this cycle [6], so that is a start. I think there is interest in porting os-*-config to Windows as the way of enabling deployment resources (help welcome!). Any, not really suggestions from my side but more observations and thoughts. I wanted to share those and raise some discussion on possible options. Tha
Re: [openstack-dev] [Heat] image requirements for Heat software config
On 15/10/14 10:38, Clint Byrum wrote: Excerpts from Thomas Spatzier's message of 2014-10-14 10:13:27 -0700: Hi all, I have been experimenting a lot with Heat software config to check out what works today, and to think about potential next steps. I've also worked on an internal project where we are leveraging software config as of the Icehouse release. I think what we can do now from a user's perspective in a HOT template is really nice and resonates well also with customers I've talked to. One of the points where we are constantly having issues, and also got some push back from customers, are the requirements on the in-instance tools and the process of building base images. One observation is that building a base image with all the right stuff inside sometimes is a brittle process; the other point is that a lot of customers do not like a lot of requirements on their base images. They want to maintain one set of corporate base images, with as little modification on top as possible. Regarding the process of building base images, the currently documented way [1] of using diskimage-builder turns out to be a bit unstable sometimes. Not because diskimage-builder is unstable, but probably because it pulls in components from a couple of sources: #1 we have a dependency on implementation of the Heat engine of course (So this is not pulled in to the image building process, but the dependency is there) #2 we depend on features in python-heatclient (and other python-* clients) #3 we pull in implementation from the heat-templates repo #4 we depend on tripleo-image-elements #5 we depend on os-collect-config, os-refresh-config and os-apply-config #6 we depend on diskimage-builder itself Heat itself and python-heatclient are reasonably well in synch because there is a release process for both, so we can tell users with some certainty that a feature will work with release X of OpenStack and Heat and version x.z.y of python-heatclient. For the other 4 sources, success sometimes depends on the time of day when you try to build an image (depending on what changes are currently included in each repo). So basically there does not seem to be a consolidated release process across all that is currently needed for software config. I don't really understand why a "consolidated release process across all" would be desired or needed. #3 is pretty odd. You're pulling in templates from the examples repo? heat-templates is where our hook elements live: [1] https://github.com/openstack/heat-templates/blob/master/hot/software-config/elements/README.rst For #4-#6, those are all on pypi and released on a regular basis. Build yourself a bandersnatch mirror and you'll have locally controlled access to them which should eliminate any reliability issues. The ideal solution would be to have one self-contained package that is easy to install on various distributions (an rpm, deb, MSI ...). Secondly, it would be ideal to not have to bake additional things into the image but doing bootstrapping during instance creation based on an existing cloud-init enabled image. For that we would have to strip requirements down to a bare minimum required for software config. One thing that comes to my mind is the cirros software config example [2] that Steven Hardy created. It is admittedly no up to what one could do with an image built according to [1] but on the other hand is really slick, whereas [1] installs a whole set of things into the image (some of which do not really seem to be needed for software config). The agent problem is one reason I've been drifting away from Heat for software configuration, and toward Ansible. Mind you, I wrote os-collect-config to have as few dependencies as possible as one attempt around this problem. Still it isn't capable enough to do the job on its own, so you end up needing os-apply-config and then os-refresh-config to tie the two together. For heat's requirements, os-refresh-config or os-apply-config is overkill to go from occ to the config hooks, but they're tiny and useful in their own right so I'm OK with requiring them. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] image requirements for Heat software config
Excerpts from Clint Byrum's message on 14/10/2014 23:38:46: > > > > Regarding the process of building base images, the currently documented way > > [1] of using diskimage-builder turns out to be a bit unstable sometimes. > > Not because diskimage-builder is unstable, but probably because it pulls in > > components from a couple of sources: > > #1 we have a dependency on implementation of the Heat engine of course (So > > this is not pulled in to the image building process, but the dependency is > > there) > > #2 we depend on features in python-heatclient (and other python-* clients) > > #3 we pull in implementation from the heat-templates repo > > #4 we depend on tripleo-image-elements > > #5 we depend on os-collect-config, os-refresh-config and os-apply-config > > #6 we depend on diskimage-builder itself > > > > Heat itself and python-heatclient are reasonably well in synch because > > there is a release process for both, so we can tell users with some > > certainty that a feature will work with release X of OpenStack and Heat and > > version x.z.y of python-heatclient. For the other 4 sources, success > > sometimes depends on the time of day when you try to build an image > > (depending on what changes are currently included in each repo). So > > basically there does not seem to be a consolidated release process across > > all that is currently needed for software config. > > > > I don't really understand why a "consolidated release process across > all" would be desired or needed. Well, all pieces have to fit together so everything work. I had many situations where I used the currently up-to-date version of each piece but something just did not work. Then I found that some patch was in review on any of those, so trying a few days later worked. I would be good for users to have one verified package of everything instead of going thru a trial and error process. Maybe this is going to improve in the future, since so far or until recently a lot of software config was still work in progress. But up to now, the image building has been a challenge at some time. > > #3 is pretty odd. You're pulling in templates from the examples repo? We have to pull in the image elements and hooks for software config from there. > > For #4-#6, those are all on pypi and released on a regular basis. Build > yourself a bandersnatch mirror and you'll have locally controlled access > to them which should eliminate any reliability issues. So switching from git repo based installed as described in [1] to pypi based installs, where I can specify a version number would help? Then what we would still need is a set of version for each package that are verified to work together (my previous point). > > > The ideal solution would be to have one self-contained package that is easy > > to install on various distributions (an rpm, deb, MSI ...). > > Secondly, it would be ideal to not have to bake additional things into the > > image but doing bootstrapping during instance creation based on an existing > > cloud-init enabled image. For that we would have to strip requirements down > > to a bare minimum required for software config. One thing that comes to my > > mind is the cirros software config example [2] that Steven Hardy created. > > It is admittedly no up to what one could do with an image built according > > to [1] but on the other hand is really slick, whereas [1] installs a whole > > set of things into the image (some of which do not really seem to be needed > > for software config). > > The agent problem is one reason I've been drifting away from Heat > for software configuration, and toward Ansible. Mind you, I wrote > os-collect-config to have as few dependencies as possible as one attempt > around this problem. Still it isn't capable enough to do the job on its > own, so you end up needing os-apply-config and then os-refresh-config > to tie the two together. > > Ansible requires sshd, and python, with a strong recommendation for > sudo. These are all things that pretty much every Linux distribution is > going to have available. Interesting, I have to investigate this. Thanks for the hint. > > > > > Another issue that comes to mind: what about operating systems not > > supported by diskimage-builder (Windows), or other hypervisor platforms? > > > > There is a windows-diskimage-builder: > > https://git.openstack.org/cgit/stackforge/windows-diskimage-builder Good to know; I wasn't aware of it. Thanks! > > diskimage-builder can produce raw images, so that should be convertible > to pretty much any other hypervisor's preferred disk format. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] image requirements for Heat software config
Excerpts from Steve Baker's message on 14/10/2014 23:52:41: > Regarding the process of building base images, the currently documented way > [1] of using diskimage-builder turns out to be a bit unstable sometimes. > Not because diskimage-builder is unstable, but probably because it pulls in > components from a couple of sources: > #1 we have a dependency on implementation of the Heat engine of course (So > this is not pulled in to the image building process, but the dependency is > there) > #2 we depend on features in python-heatclient (and other python-* clients) > #3 we pull in implementation from the heat-templates repo > #4 we depend on tripleo-image-elements > #5 we depend on os-collect-config, os-refresh-config and os-apply-config > #6 we depend on diskimage-builder itself > > Heat itself and python-heatclient are reasonably well in synch because > there is a release process for both, so we can tell users with some > certainty that a feature will work with release X of OpenStack and Heat and > version x.z.y of python-heatclient. For the other 4 sources, success > sometimes depends on the time of day when you try to build an image > (depending on what changes are currently included in each repo). So > basically there does not seem to be a consolidated release process across > all that is currently needed for software config. > > The ideal solution would be to have one self-contained package that is easy > to install on various distributions (an rpm, deb, MSI ...). > Secondly, it would be ideal to not have to bake additional things into the > image but doing bootstrapping during instance creation based on an existing > cloud-init enabled image. For that we would have to strip requirements down > to a bare minimum required for software config. One thing that comes to my > mind is the cirros software config example [2] that Steven Hardy created. > It is admittedly no up to what one could do with an image built according > to [1] but on the other hand is really slick, whereas [1] installs a whole > set of things into the image (some of which do not really seem to be needed > for software config). > > Building an image from git repos was the best chance of having a > single set of instructions which works for most cases, since the > tools were not packaged for debian derived distros. This seems to be > improving though; the whole build stack is now packaged for Debian > Unstable, Testing and also Ubuntu Utopic (which isn't released yet). > Another option is switching the default instructions to installing > from pip rather than git, but that still gets into distro-specific > quirks which complicate the instructions. Until these packages are > on the recent releases of common distros then we'll be stuck in this > slightly awkward situation. Yeah, I understand that the current situation is probably there because we are so close to the point where the features get developed. So hopefully this will improve and stabilize in the future. > > I wrote a cloud-init boot script to install the agents from packages > from a pristine Fedora 20 [3] and it seems like a reasonable > approach for when building a custom image isn't practical. Somebody > submitting the equivalent for Debian and Ubuntu would be most > welcome. We need to decide whether *everything* should be packaged > or if some things can be delivered by cloud-init on boot (os- > collect-config.conf template, 55-heat-config, the actual desired > config hook...) Thanks for the pointer. I'll have a look. I think if we can put as little requirements on the base image and do as much as possible at boot, that would be good. If help is needed for getting this done for other distros (and for Windows) we can certainly work on this. We just have to agree and be convinced that this is the right path. > > I'm all for there being documentation for the different ways of > getting the agent and hooks onto a running server for a given > distro. I think the hot-guide would be the best place to do that, > and I've been making a start on that recently [4][5] (help > welcome!). The README in [1] should eventually refer to the hot- > guide once it is published so we're not maintaining multiple build > instructions. I'll have a look at all the pointers. Agree that this is extremely useful. BTW: the unit testing work you started on the software config hooks will definitely help as well! > > Another issue that comes to mind: what about operating systems not > supported by diskimage-builder (Windows), or other hypervisor platforms? > The Cloudbase folk have contributed some useful cloudbase-init > templates this cycle [6], so that is a start. I think there is > interest in porting os-*-config to Windows as the way of enabling > deployment resources (help welcome!). Yes, I've seen those templates. As long as there is an image that work with them, this is great. I have to look closer into the Windows things. > > Any, not really suggestions from my side but more observati
