Hi, Today we discussed a challenge around image tags that mostly boils down to limitations in how overcloud_containers.yaml is constructed and used.
TL;DR, we need a smart and easy way to work with the overcloud_containers.yaml file (especially tags). Let's highlight a few use cases that we need to work through: #1. Building containers For building containers, all we really care is the name of the images we need to build. Today, we install a trunk repository and then install tripleo-common-containers from that trunk repository. We then mostly grep/sed/awk our way from overcloud_containers.yaml to a clean list of images to build and then build those. Relatively okay with this but prone to things breaking -- a clean way to get just the list of images out of there would be nice. #2. Testing and promoting containers This comes right after use case #1 where we build containers in the pipeline. For those familiar with the CI pipeline to do promotions [1], this would look a bit like this [2]. In practice, this works basically the same way as we build, test and promote tripleo-quickstart images. We pick a particular trunk repository hash and build containers for that hash. These are then pushed with both the tags ":latest" and ":<dlrn_hash>". We're then supposed to test those containers in the pipeline but to do that, we need to be pulling from :<dlrn_hash>, not :latest... although they are in theory equivalent at that given time, this might not always be true. So the testing job(s) need a way to easily customize/pull from that particular hash instead of the hardcoded latest we have right now. #3. Upstream gate jobs Ideally, gate jobs should never use ":latest". This is in trunk/dlrn terms the equivalent of "/current/" or "/consistent/". They'd use something like ":latest-passed-ci" which would be the proper equivalent of "/current-passed-ci/" or "/current-tripleo/". This brings an interesting challenge around how we currently add new images to overcloud_containers.yaml (example [3]). It is expected that, when you add the image, the image is already present on the registry because otherwise the container jobs will fail since this new image cannot be pulled (example [4]). My understanding currently is that humans may build and push images to the registry ahead of time so that this works. We can keep a similar approach if that's what we really want with the new private registry, the job that builds container is made generic exactly to be able to build just a specific set of image(s) if we want. Here's the catch, though: this new container image will have the ":latest" tag, it will not have ":latest-passed-ci" because it hasn't passed CI yet, it's being added just now. So how do we address this ? Note: We've already discussed that some containers need to pick up the latest and the greatest from the "/current/" repository, either because they are "direct" tripleo packages or if "Depends-On" is used. So far, the solution we seem to be going towards is to pick up the containers from ":latest-passed-ci" and then more or less add a 'yum update' layer to the images needing an update. This is the option that is in the best interest of time, we'd otherwise be spending too much time building containers in jobs that are already taking way too long to run. #4. Test days When doing test days, we know to point testers to /current-passed-ci/ as well as tested quickstart images. How can we make it easy for containers ? If the container list from tripleo-common is hardcoded to latest, that won't work. If it's hardcoded to :latest-passed-ci, it won't work for other use cases. Ideally this would be super easy for end users as well as developers so that they can get started easily. #5 Stable releases, users, operators & co The packaging workflow is not the same for trunk (dlrn out of git source on trunk.rdoproject.oirg) and for stable releases (CentOS build system out of released tarballs on mirror.centos.org). It's also going to be different for containers. For trunk, we'll be building containers with trunk repositories and publishing them to a private registry analogous to trunk.rdoproject.org repositories. For stable releases, while still hand-wavy and foggy, we seem to be headed in the direction of the CentOS official registry which takes Dockerfiles in pseudo dist-git repositories and builds/publishes the containers through Jenkins jobs. This is sort of similar to how downstream would work if you replace Jenkins by Brew/OSBS. So here, we want to use the overcloud_containers.yaml file to "compile" Dockerfiles which will be shipped to git repositories and then built by another process. These containers will be published somewhere that tripleo-common is sort of expected to know ahead of time because users, customers and developers need to be pulling from that "stable" source and from a "stable" tag. So... what do we do ? [1]: https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ [2]: http://i.imgur.com/I26njTz.png [3]: https://review.openstack.org/#/c/466305/ [4]: http://logs.openstack.org/51/467651/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/dc61632/logs/oooq/undercloud/home/jenkins/overcloud_prep_containers.log.txt.gz#_2017-05-24_19_24_13 David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev