Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy wrote: > Hi all, > > I wanted to raise visibility of a problem we're experiencing in TripleO CI, > which I think will potentially affect other projects that consume trunk > builds from the RDO repositories (and potentially other distros too I > guess) > > The problem is that we tag our final ocata releases after branching > stable/ocata, but there is then a period prior to cutting pike-1 milestone > releases (or some other intermediate release for projects not following the > milestone model) where trunk package builds n-v-r is older than the stable > branches. > > This presents a big problem if you want to test package-derived updates > from stable/$foo to master, as the pacakges don't get updated where the > installed stable one is newer than the one built from master. > > I raised this bug initially thinking it was specific to the puppet-* > projects, but it seems from subsequent discussion that it's a broader issue > that may impact many OpenStack projects. > > https://bugs.launchpad.net/tripleo/+bug/1669462 > > I'm not clear on the best path forward at this point, but the simplest one > suggested so far is to simply tag a new pre-milestone/alpha release for all > master branches, which will enable testing upgrades to master. > As reference, puppet team proposed a solution for this issue in openstack puppet modules in http://lists.openstack.org/pipermail/openstack-dev/2017-March/113494.html However, as mentioned before this is affecting all projects, not only puppet ones. > I know we don't expect to fully support upgrades to pre-milestone releases, > but the requirement here is to simply enable testing them. > > A side-benefit of this regular testing e.g via CI is we'll find upgrade issues > much faster than waiting for one or more milestone releases to happen then > doing an upgrade-debug firedrill later in the cycle (which has been bad for > project and deployment teams IMO, so it'd be good to figure out this first > step to enable earlier testing of upgrades to the development release). > > Any thoughts on how we can resolve this would be much appreciated, thanks! > > Steve > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso : > On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy wrote: >> https://bugs.launchpad.net/tripleo/+bug/1669462 >> >> I'm not clear on the best path forward at this point, but the simplest one >> suggested so far is to simply tag a new pre-milestone/alpha release for all >> master branches, which will enable testing upgrades to master. That would be indeed simplest and cleanest solution but it has been rejected in the past, IIUC with reasoning that projects are not sure what will be their next version. IMHO that should not be the case if the project has decided to have a stable branch and follows semver and stable branch policy: in that case project will keep X.Y frozen on stable so master should be bumped to at least X.Y+1 immediately after branching stable. For a project using PBR postversioning (almost all by now) this can be achieved by either pushing the appropriate alpha git tag or by using PBR Sem-Ver: feature https://docs.openstack.org/developer/pbr/#version Now, purists would complain that artificial commit right after branching is not really adding a feature but you could look at it as a credit because you surely will implement new features in the new version! If project is without stable:follows-policy tag and it will keep adding features on the stable branch, increasing Y part of semver or if they already decided to bump X major version, Sem-Ver: api-break can be used. Example for TripleO projects for opening Ocata: https://review.openstack.org/#/q/topic:open-ocata+Sem-Ver: > As reference, puppet team proposed a solution for this issue in > openstack puppet modules in > http://lists.openstack.org/pipermail/openstack-dev/2017-March/113494.html > > However, as mentioned before this is affecting all projects, not only > puppet ones. Solution for OpenStack Puppet modules is to explicitly set pre-version in metadata.json. We had used pre-versioning before in regular OpenStack projects by setting version in setup.cfg but this has been abandoned and instead we rely on PBR to compute the next version. >> I know we don't expect to fully support upgrades to pre-milestone releases, >> but the requirement here is to simply enable testing them. >> >> A side-benefit of this regular testing e.g via CI is we'll find upgrade >> issues >> much faster than waiting for one or more milestone releases to happen then >> doing an upgrade-debug firedrill later in the cycle (which has been bad for >> project and deployment teams IMO, so it'd be good to figure out this first >> step to enable earlier testing of upgrades to the development release). >> >> Any thoughts on how we can resolve this would be much appreciated, thanks! I was suggested that upgrade jobs could force install i.e. basically doing downgrades before milestone1 but that's not testing upgrades! Versioning is very fundamental for packaging and reflects upstream version, we cannot fake versions in packaging. In the spirit of Continuous Everything, versioning should also be continuous and ensured that master version is always > previous releases. Projects doing stable branches could take action individually using Sem-Ver keyword but better would be to define this as a rule for having stable branches and enforce it cross projects by the release procedure. Cheers, Alan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
On 2017-03-09 12:12:01 +0100 (+0100), Alan Pevec wrote: > 2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso : > > On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy wrote: > >> https://bugs.launchpad.net/tripleo/+bug/1669462 > >> > >> I'm not clear on the best path forward at this point, but the simplest one > >> suggested so far is to simply tag a new pre-milestone/alpha release for all > >> master branches, which will enable testing upgrades to master. > > That would be indeed simplest and cleanest solution but it has been > rejected in the past, IIUC with reasoning that projects are not sure > what will be their next version. IMHO that should not be the case if > the project has decided to have a stable branch and follows semver and > stable branch policy: in that case project will keep X.Y frozen on > stable so master should be bumped to at least X.Y+1 immediately after > branching stable. [...] In the past we addressed this by automatically merging the release tag back into master, but we stopped doing that a cycle ago because it complicated release note generation. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
Jeremy Stanley wrote: > On 2017-03-09 12:12:01 +0100 (+0100), Alan Pevec wrote: >> 2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso : >>> On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy wrote: https://bugs.launchpad.net/tripleo/+bug/1669462 I'm not clear on the best path forward at this point, but the simplest one suggested so far is to simply tag a new pre-milestone/alpha release for all master branches, which will enable testing upgrades to master. >> >> That would be indeed simplest and cleanest solution but it has been >> rejected in the past, IIUC with reasoning that projects are not sure >> what will be their next version. IMHO that should not be the case if >> the project has decided to have a stable branch and follows semver and >> stable branch policy: in that case project will keep X.Y frozen on >> stable so master should be bumped to at least X.Y+1 immediately after >> branching stable. > [...] > > In the past we addressed this by automatically merging the release > tag back into master, but we stopped doing that a cycle ago because > it complicated release note generation. Right, there is a bit of history here. One option is to tag the first commit on master after the stable branch point, to reduce (but not eliminate) the window where stable > master, but that relied on a lot of discipline and checks (since it's hard to automate). We preferred the option to merge tags back on master, which was not perfect (creates overlap in versions automatically generated) but at least the dev version was always > last released version. However, the merged tags history prevented reno from determining correctly where it was and generate appropriate release notes. So we dropped it... Robert Collins argued that users should consume from a single channel anyway: either follow master or follow a stable branch, and not combine both. Here the problem is that you test upgrades from stable/$foo to master, which is basically something the model does not support. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
Excerpts from Thierry Carrez's message of 2017-03-13 15:13:18 +0100: > Jeremy Stanley wrote: > > On 2017-03-09 12:12:01 +0100 (+0100), Alan Pevec wrote: > >> 2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso : > >>> On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy wrote: > https://bugs.launchpad.net/tripleo/+bug/1669462 > > I'm not clear on the best path forward at this point, but the simplest > one > suggested so far is to simply tag a new pre-milestone/alpha release for > all > master branches, which will enable testing upgrades to master. > >> > >> That would be indeed simplest and cleanest solution but it has been > >> rejected in the past, IIUC with reasoning that projects are not sure > >> what will be their next version. IMHO that should not be the case if There is the problem that some deliverables don't know their next version in advance. There is also the problem that the vast majority of our deliverables do not use pre-release versions (like alphas) at all. > >> the project has decided to have a stable branch and follows semver and > >> stable branch policy: in that case project will keep X.Y frozen on > >> stable so master should be bumped to at least X.Y+1 immediately after > >> branching stable. > > [...] > > > > In the past we addressed this by automatically merging the release > > tag back into master, but we stopped doing that a cycle ago because > > it complicated release note generation. > > Right, there is a bit of history here. > > One option is to tag the first commit on master after the stable branch > point, to reduce (but not eliminate) the window where stable > master, > but that relied on a lot of discipline and checks (since it's hard to > automate). Yes, I think the level of coordination involved in making this work for all projects would be challenging to accomplish with the small release team. We have the release task list down to a fairly small minimum right now, and I don't relish the idea of adding more complexity if we can avoid it. > We preferred the option to merge tags back on master, which was not > perfect (creates overlap in versions automatically generated) but at > least the dev version was always > last released version. > > However, the merged tags history prevented reno from determining > correctly where it was and generate appropriate release notes. So we The problem was with a tag appearing on multiple branches, and basically being inserted into the master branch at a random spot, it wasn't possible to actually tell which series a version belonged to and which notes should go with which versions. > dropped it... Robert Collins argued that users should consume from a > single channel anyway: either follow master or follow a stable branch, > and not combine both. Here the problem is that you test upgrades from > stable/$foo to master, which is basically something the model does not > support. We test this upgrade scenario in the upstream CI, too. The difference is that grenade can tell pip "install exactly the version I am pointing to in this directory on disk," rather than relying on version numbers to notice that an upgrade is needed (or should be avoided, as the case may be). Is it possible to do that with system packages in some way, too, without pinning package versions in puppet or tripleo? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
2017-03-09 14:58 GMT+01:00 Jeremy Stanley : > In the past we addressed this by automatically merging the release > tag back into master, but we stopped doing that a cycle ago because > it complicated release note generation. Also this was including RC >= 2 and final tags so as soon as the first stable maintenance version was released, master was again lower version. Cheers, Alan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
2017-03-13 15:49 GMT+01:00 Doug Hellmann : ... > We test this upgrade scenario in the upstream CI, too. The difference > is that grenade can tell pip "install exactly the version I am > pointing to in this directory on disk," rather than relying on > version numbers to notice that an upgrade is needed (or should be > avoided, as the case may be). > > Is it possible to do that with system packages in some way, too, without > pinning package versions in puppet or tripleo? That would not test the regular update flow, we'd have to use workaround for the period before milestone 1 and run explicit yum downgrade with the explicit package list to force lower version packages to install. That means real upgrades would not be tested until this kludge is removed after first milestone is released. Instead, I'll look at adding automatic bump in RDO Trunk builder before milestone1 i.e. when it detects master version is lower than stable branches. Cheers, Alan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering
On Mon, Mar 13, 2017 at 12:29 PM, Alan Pevec wrote: > 2017-03-09 14:58 GMT+01:00 Jeremy Stanley : >> In the past we addressed this by automatically merging the release >> tag back into master, but we stopped doing that a cycle ago because >> it complicated release note generation. > > Also this was including RC >= 2 and final tags so as soon as the first > stable maintenance version was released, master was again lower > version. topic sounds staled. Alan, do we have an ETA on the RDO workaround? Thanks, > Cheers, > Alan > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev