Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On Thu, May 28, 2015 at 12:55:50PM -0700, Jim Rollenhagen wrote: [snip] I also put an informational spec about this change up in the ironic-specs repo: https://review.openstack.org/#/c/185171/. My goal was to discuss this in the spec, but the mailing list is fine too. There are some unanswered questions in that review that we should make sure we cover. I've incorporated feedback from this thread and the spec review into a new version of this spec. Thanks to everyone for the great feedback and support, keep it coming. :) // jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On 28 May 2015 at 15:55, Jim Rollenhagen j...@jimrollenhagen.com wrote: ... I also put an informational spec about this change up in the ironic-specs repo: https://review.openstack.org/#/c/185171/. My goal was to discuss this in the spec, but the mailing list is fine too. There are some unanswered questions in that review that we should make sure we cover. I'm really excited about this, and hope it can lead to good outcomes for the rest of OpenStack in the future. :) // jim Hi Jim, thanks for the spec! It'll be good to incorporate the feedback from this thread, into the spec. To be honest, I'm apprehensive and worried about this. As Devananda mentioned, the devil is in the details. I hope we get enough of the details right, so that we aren't spending time wondering about the process instead of doing coding and reviewing. My main concern is that with my reviewer's hat on, I don't feel like it will help me much. We are proposing more (smaller) releases, and as long as people know that there is an upcoming release planned, they will try to get their patches merged. So I think we're just spreading out the 'please review my patch' requests, but at least I won't feel so bad now when I say no, because they won't have to wait 6+ months to land their feature :) Anyway, I'm open to this (or am I, maybe I'm just resigned to it cause the train seems to be leaving), and hope to be pleasantly surprised ;) --ruby __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On 05/28/2015 06:41 PM, Devananda van der Veen wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Longer version... We captured some notes from our discussion on Thursday afternoon's etherpad: https://etherpad.openstack.org/p/liberty-ironic-scaling-the-dev-team Which I've summarized below, and mixed in several themes which didn't get captured on the 'pad but were discussed somewhere, possibly in a hallway or on Friday. Current challenges / observations: - six weeks' feature freeze is not actually having the desired stabilizing effect - the post-release/pre-summit slump only makes that worse - many folks stop reviewing during this time because they know their own features won't land, and instead focus their time downstream - this creates pressure to land features which aren't ready, and leaves few people to find bugs, write docs, and generally prepare the release during this window The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available Note, that will need some scheduling anyway, so that we can slow down a week before. So probably still some milestone process required, wdyt? - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't So no stable branch for other independent releases? What if serious security issue is found in one of these? Will we advice users to downgrade to the latest coordinated release? We think this will accomplish a few things: - make the developer experience better by being more consistent, thus keeping developers engaged year-round and increase the likelyhood they'll find and fix bugs - reduce stress on core reviewers since there's no crunch time at the end of a cycle - allow big changes to bake in a feature branch, rather than in a series of gerrit patches that need to be continually re-reviewed and cherry-picked to test them. - allow operators who wish to use Ironic outside of OpenStack to consume feature releases more rapidly, while still consuming approved releases instead of being forced to deploy from trunk For reference, Michael has posted a tracking change to the governance repo here: https://review.openstack.org/#/c/185203/ Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track features (maybe it means we stop doing that?). I don't see problem with using launchpad tbh. Re versions my biggest concern is pbr: I don't know how it's version detection black magic is going to react if we go from 2015.1 to say 1.0. Input appreciated. Thanks, Devananda __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
Hi Note, that will need some scheduling anyway, so that we can slow down a week before. So probably still some milestone process required, wdyt? We can cut a release and establish it one or two weeks before the official OpenStack release. But prior to that we can just cut a release whenever we think it's good without following any time scheduling. - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't So no stable branch for other independent releases? What if serious security issue is found in one of these? Will we advice users to downgrade to the latest coordinated release? Good point, but I think that would extremely hard and costly to maintain a bunch of stable branches at once. So having a stable branch every 6 months to follow the OpenStack model seems enough. If we find a serious security issue, we could advice the user to downgrade to the last coordinate release which the fix was backported or if the user is following the feature based release of Ironic we can fix the problem and cut a new release and advice the user to use that instead. Input appreciated. I think it's a good plan and we have to keep in mind that maybe it doesn't work but I think we should definitely try it out and see how it goes. That said, I'm also confused about the versioning (with launchpad and pbr) but Swift already does something similar right? We should take a look at how they do it. Cheers, Lucas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On 2015-05-29 11:47:36 +0200 (+0200), Thierry Carrez wrote: [...] As far as vulnerability management goes, we already publish the master fix as part of the advisory, so people can easily find that. The only thing the VMT might want to reconsider is: when an issue is /only/ present in the master branch and was never part of a release, it currently gets fixed silently there, without an advisory being published. I guess that could be evolved to publish an advisory if the issue was in any released version. That would still not give users of intermediary versions a pure backport for their version, but give them notice and a patch to apply. I also suspect that for critical issues Ironic would issue a new intermediary release sooner rather than later. This is what we've historically done for master-branch-only projects anyway, so I don't see it as a new process. Works just fine, but as you say we should make sure we know at the time of writing the advisory what the next release version number will be (and hopefully it comes along shortly after the fix merges so people can just upgrade to it). -- Jeremy Stanley signature.asc Description: Digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
Devananda van der Veen wrote: [...] The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't With the already-mentioned caveats on feature branch usage, I think this makes sense, for simpler or more stable projects that can actually release something that works without a large stabilization period. That said, it's worth noting that the way Swift currently aligns with the coordinated release at the end of a cycle is a little different: - for intermediary releases, Swift would just soft-freeze at a proposed SHA and tag that if everyone is fine with it after some time (which is what you propose here) - for the final release Swift tags a RC1 (which triggers a stable/$SERIES release branch) and has the option of doing other RCs if a critical issue is found before the coordinated release date, then the final is re-tagged from the last RC In all cases, for stable maintenance/CI reasons, we need to cut a stable/$SERIES branch for every project in the weeks preceding the coordinated release date -- but I guess we have two options there. (1) we could adopt the Swift RC model and special-case the release process for the final release. (2) we could just create the stable branch from your presumed last release, and in case a critical issue is found, backport the fix and tag a point release there (and consider that point release the $SERIES release) Since I would only recommend simpler / more stable projects to switch to that model for the time being (projects that are less likely to need release candidates), I think (2) makes sense (and I could see Swift adopting it as well). [...] Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track features (maybe it means we stop doing that?). As far as semver goes, since you switch to independent releases you can't stick to a common (2015.1.0) version anyway, so I think it's less confusing to use a semver versioning than using conflicting date-based ones. As far as Launchpad goes, I don't think switching to that model changes anything. Oslo libraries (which also follow a semver-driven, multiple-release + final one model) track features and bugfixes using Launchpad alright, and the series graph in LP actually helps in figuring out where you are. Together with the proposed changes in release tracking to report after the fact more than try to predict (see thread I posted yesterday), I think it would work just fine. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
Lucas Alvares Gomes wrote: - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't So no stable branch for other independent releases? What if serious security issue is found in one of these? Will we advice users to downgrade to the latest coordinated release? Good point, but I think that would extremely hard and costly to maintain a bunch of stable branches at once. So having a stable branch every 6 months to follow the OpenStack model seems enough. If we find a serious security issue, we could advice the user to downgrade to the last coordinate release which the fix was backported or if the user is following the feature based release of Ironic we can fix the problem and cut a new release and advice the user to use that instead. Right, there is no way under our current setup to support more than a (common) stable branch every 6 months. That means if people use intermediary releases and a vulnerability is found, they can either backport the fix to their code, workaround the issue until the next version is released, or downgrade to tracking the last stable branch. As far as vulnerability management goes, we already publish the master fix as part of the advisory, so people can easily find that. The only thing the VMT might want to reconsider is: when an issue is /only/ present in the master branch and was never part of a release, it currently gets fixed silently there, without an advisory being published. I guess that could be evolved to publish an advisory if the issue was in any released version. That would still not give users of intermediary versions a pure backport for their version, but give them notice and a patch to apply. I also suspect that for critical issues Ironic would issue a new intermediary release sooner rather than later. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On 05/28/2015 06:41 PM, Devananda van der Veen wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Just a quick follow-up to voice the Distro's opinion (I'm sure other package maintainers will agree with what I'm going to write). It's fine if you use whatever release schedule that you want. Though what's important for us, downstream distro, is that you make sure to release a longer term version at the same time as everything else, and that you make sure security follow-up is done properly. It's really important that you clearly identify the versions for which you will do security patch backports, so that we don't (by mistake) release a stable version of a given distribution with a something you wont ever give long term support for. I'm sure you guys know that already, but better be safe than sorry, and other less experience projects may find the above useful. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
Excerpts from Thierry Carrez's message of 2015-05-29 12:12:51 +0200: Devananda van der Veen wrote: [...] The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't That last part is important -- you don't want to get bogged down with managing umpteen different backports, so you want to maintain one stable branch per cycle. Deployers running from master will get new stuff and fixes quickly, and deployers running from stable releases will have stability and fixes. We'll be serving both types of downstream consumers well. With the already-mentioned caveats on feature branch usage, I think this makes sense, for simpler or more stable projects that can actually release something that works without a large stabilization period. That said, it's worth noting that the way Swift currently aligns with the coordinated release at the end of a cycle is a little different: - for intermediary releases, Swift would just soft-freeze at a proposed SHA and tag that if everyone is fine with it after some time (which is what you propose here) - for the final release Swift tags a RC1 (which triggers a stable/$SERIES release branch) and has the option of doing other RCs if a critical issue is found before the coordinated release date, then the final is re-tagged from the last RC In all cases, for stable maintenance/CI reasons, we need to cut a stable/$SERIES branch for every project in the weeks preceding the coordinated release date -- but I guess we have two options there. (1) we could adopt the Swift RC model and special-case the release process for the final release. (2) we could just create the stable branch from your presumed last release, and in case a critical issue is found, backport the fix and tag a point release there (and consider that point release the $SERIES release) Since I would only recommend simpler / more stable projects to switch to that model for the time being (projects that are less likely to need release candidates), I think (2) makes sense (and I could see Swift adopting it as well). If I understand you correctly, option 2 is what we do for Oslo libraries that release this way. [...] Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track features (maybe it means we stop doing that?). As far as semver goes, since you switch to independent releases you can't stick to a common (2015.1.0) version anyway, so I think it's less confusing to use a semver versioning than using conflicting date-based ones. I have a todo on my list to write up a spec summarizing the summit discussion on the new version numbering scheme for all projects. tl;dr is moving to semver, starting with version 12.0.0 (since Kilo was our 11th release). Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On Fri, May 29, 2015 at 8:11 AM, Thomas Goirand z...@debian.org wrote: On 05/28/2015 06:41 PM, Devananda van der Veen wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Just a quick follow-up to voice the Distro's opinion (I'm sure other package maintainers will agree with what I'm going to write). It's fine if you use whatever release schedule that you want. Though what's important for us, downstream distro, is that you make sure to release a longer term version at the same time as everything else, and that you make sure security follow-up is done properly. It's really important that you clearly identify the versions for which you will do security patch backports, so that we don't (by mistake) release a stable version of a given distribution with a something you wont ever give long term support for. The only way I know of to get this information now is to look in the git repos, so for example for keystoneclient I can find the security-supported releases are 0.11 (icehouse) -- http://git.openstack.org/cgit/openstack/python-keystoneclient/log/?h=stable/icehouse 1.1 (juno) 1.3 (kilo) The rest (for example, 1.2) are not supported. The place I would expect to find this information is here: https://wiki.openstack.org/wiki/Releases tschüß -- Brant I'm sure you guys know that already, but better be safe than sorry, and other less experience projects may find the above useful. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
Excerpts from Thomas Goirand's message of 2015-05-29 15:11:25 +0200: On 05/28/2015 06:41 PM, Devananda van der Veen wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Just a quick follow-up to voice the Distro's opinion (I'm sure other package maintainers will agree with what I'm going to write). It's fine if you use whatever release schedule that you want. Though what's important for us, downstream distro, is that you make sure to release a longer term version at the same time as everything else, and that you make sure security follow-up is done properly. It's really important that you clearly identify the versions for which you will do security patch backports, so that we don't (by mistake) release a stable version of a given distribution with a something you wont ever give long term support for. I'm sure you guys know that already, but better be safe than sorry, and other less experience projects may find the above useful. We do still plan to use stable branches for those supported releases, so as long as you're pulling from the right branch when packaging it should work fine, right? Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On Thu, May 28, 2015 at 12:54:58PM -0400, Monty Taylor wrote: On 05/28/2015 12:41 PM, Devananda van der Veen wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Longer version... We captured some notes from our discussion on Thursday afternoon's etherpad: https://etherpad.openstack.org/p/liberty-ironic-scaling-the-dev-team Which I've summarized below, and mixed in several themes which didn't get captured on the 'pad but were discussed somewhere, possibly in a hallway or on Friday. Current challenges / observations: - six weeks' feature freeze is not actually having the desired stabilizing effect - the post-release/pre-summit slump only makes that worse - many folks stop reviewing during this time because they know their own features won't land, and instead focus their time downstream - this creates pressure to land features which aren't ready, and leaves few people to find bugs, write docs, and generally prepare the release during this window The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't This sounds very much like what swift has been doing, which seems to work well. One thing I'll point out about feature branches - swift does the same thing - but by low what we mean _really_ is one that tends to have at least a one-cycle lifespan that represents SIGNIFICANT work Erasure Coding was the most recent one. I mention that because feature branches carry a cost here- but there are plenty of things on the internets that talk about the awesome things you can do with them - most of which are ways to work around not actually having real CI. In fact, for most risky things I'd suggest a feature flag and landing the patches into master. However, there are things that involve big refactors where hiding behind a flag would quickly get bong. So - sounds great and there is successful prior art - but just be wary of actually making use of the feature branch - and I counsel always asking yourself first is it possible to do this without one because the pain will be less. I completely agree with this. Someone mentioned feature branches at the summit discussion for this, and everyone started talking about using them for all the things. Not sure where that came from. I'd love to keep them to a minimum. I also put an informational spec about this change up in the ironic-specs repo: https://review.openstack.org/#/c/185171/. My goal was to discuss this in the spec, but the mailing list is fine too. There are some unanswered questions in that review that we should make sure we cover. I'm really excited about this, and hope it can lead to good outcomes for the rest of OpenStack in the future. :) // jim We think this will accomplish a few things: - make the developer experience better by being more consistent, thus keeping developers engaged year-round and increase the likelyhood they'll find and fix bugs - reduce stress on core reviewers since there's no crunch time at the end of a cycle - allow big changes to bake in a feature branch, rather than in a series of gerrit patches that need to be continually re-reviewed and cherry-picked to test them. - allow operators who wish to use Ironic outside of OpenStack to consume feature releases more rapidly, while still consuming approved releases instead of being forced to deploy from trunk For reference, Michael has posted a tracking change to the governance repo here: https://review.openstack.org/#/c/185203/ Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track features
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On Thu, May 28, 2015 at 9:41 AM, Devananda van der Veen devananda@gmail.com wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Longer version... We captured some notes from our discussion on Thursday afternoon's etherpad: https://etherpad.openstack.org/p/liberty-ironic-scaling-the-dev-team Which I've summarized below, and mixed in several themes which didn't get captured on the 'pad but were discussed somewhere, possibly in a hallway or on Friday. Current challenges / observations: - six weeks' feature freeze is not actually having the desired stabilizing effect - the post-release/pre-summit slump only makes that worse - many folks stop reviewing during this time because they know their own features won't land, and instead focus their time downstream - this creates pressure to land features which aren't ready, and leaves few people to find bugs, write docs, and generally prepare the release during this window The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't We think this will accomplish a few things: - make the developer experience better by being more consistent, thus keeping developers engaged year-round and increase the likelyhood they'll find and fix bugs - reduce stress on core reviewers since there's no crunch time at the end of a cycle - allow big changes to bake in a feature branch, rather than in a series of gerrit patches that need to be continually re-reviewed and cherry-picked to test them. - allow operators who wish to use Ironic outside of OpenStack to consume feature releases more rapidly, while still consuming approved releases instead of being forced to deploy from trunk For reference, Michael has posted a tracking change to the governance repo here: https://review.openstack.org/#/c/185203/ Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track features (maybe it means we stop doing that?). Sounds like a great plan. Input appreciated. Thanks, Devananda __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On Thursday, May 28, 2015, Devananda van der Veen devananda@gmail.com wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Longer version... We captured some notes from our discussion on Thursday afternoon's etherpad: https://etherpad.openstack.org/p/liberty-ironic-scaling-the-dev-team Which I've summarized below, and mixed in several themes which didn't get captured on the 'pad but were discussed somewhere, possibly in a hallway or on Friday. Current challenges / observations: - six weeks' feature freeze is not actually having the desired stabilizing effect - the post-release/pre-summit slump only makes that worse - many folks stop reviewing during this time because they know their own features won't land, and instead focus their time downstream - this creates pressure to land features which aren't ready, and leaves few people to find bugs, write docs, and generally prepare the release during this window The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't We think this will accomplish a few things: - make the developer experience better by being more consistent, thus keeping developers engaged year-round and increase the likelyhood they'll find and fix bugs - reduce stress on core reviewers since there's no crunch time at the end of a cycle - allow big changes to bake in a feature branch, rather than in a series of gerrit patches that need to be continually re-reviewed and cherry-picked to test them. - allow operators who wish to use Ironic outside of OpenStack to consume feature releases more rapidly, while still consuming approved releases instead of being forced to deploy from trunk For reference, Michael has posted a tracking change to the governance repo here: https://review.openstack.org/#/c/185203/ Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track features (maybe it means we stop doing that?). Input appreciated. Thanks, Devananda I am looking forward to see how this transition works for Ironic and how it could apply to other projects. This sounds like a great approach! --Morgan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On May 28, 2015, at 9:54 AM, Monty Taylor mord...@inaugust.com wrote: On 05/28/2015 12:41 PM, Devananda van der Veen wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Longer version... We captured some notes from our discussion on Thursday afternoon's etherpad: https://etherpad.openstack.org/p/liberty-ironic-scaling-the-dev-team Which I've summarized below, and mixed in several themes which didn't get captured on the 'pad but were discussed somewhere, possibly in a hallway or on Friday. Current challenges / observations: - six weeks' feature freeze is not actually having the desired stabilizing effect - the post-release/pre-summit slump only makes that worse - many folks stop reviewing during this time because they know their own features won't land, and instead focus their time downstream - this creates pressure to land features which aren't ready, and leaves few people to find bugs, write docs, and generally prepare the release during this window The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't This sounds very much like what swift has been doing, which seems to work well. One thing I'll point out about feature branches - swift does the same thing - but by low what we mean _really_ is one that tends to have at least a one-cycle lifespan that represents SIGNIFICANT work Erasure Coding was the most recent one. I mention that because feature branches carry a cost here- but there are plenty of things on the internets that talk about the awesome things you can do with them - most of which are ways to work around not actually having real CI. In fact, for most risky things I'd suggest a feature flag and landing the patches into master. However, there are things that involve big refactors where hiding behind a flag would quickly get bong. So - sounds great and there is successful prior art - but just be wary of actually making use of the feature branch - and I counsel always asking yourself first is it possible to do this without one because the pain will be less. I affirm what Monty says here. The feature branches that we've used in Swift have been for significant efforts that cut across large parts of the codebase and span more than one openstack cycle. The ones we've used so far are for storage policies (1 year) and erasure codes (9 months), and we've currently got two: one for encryption (about 4 months so far) and the newest is for the golang object server. While I like the feature branches, I'd recommend only using them as necessary for very large efforts. If you do, then I'm happy to share what has and hasn't worked for us over the last 24-30 months of using them. Overall, I'm thrilled to see Ironic's new plan for releases. --John We think this will accomplish a few things: - make the developer experience better by being more consistent, thus keeping developers engaged year-round and increase the likelyhood they'll find and fix bugs - reduce stress on core reviewers since there's no crunch time at the end of a cycle - allow big changes to bake in a feature branch, rather than in a series of gerrit patches that need to be continually re-reviewed and cherry-picked to test them. - allow operators who wish to use Ironic outside of OpenStack to consume feature releases more rapidly, while still consuming approved releases instead of being forced to deploy from trunk For reference, Michael has posted a tracking change to the governance repo here: https://review.openstack.org/#/c/185203/ Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track
Re: [openstack-dev] [Ironic] [TC] Discussion: changing Ironic's release model
On 05/28/2015 12:41 PM, Devananda van der Veen wrote: Hi all, tl;dr; At the summit, the Ironic team discussed the challenges we've had with the current release model and came up with some ideas to address them. I had a brief follow-up conversation with Doug and Thierry, but I'd like this to be discussed more openly and for us (the Ironic dev community) to agree on a clear plan before we take action. If Ironic moves to a release:independent model, it shouldn't have any direct effect on other projects we integrate with -- we will continue to follow release:at-6mo-cycle-end -- but our processes for how we get there would be different, and that will have an effect on the larger community. Longer version... We captured some notes from our discussion on Thursday afternoon's etherpad: https://etherpad.openstack.org/p/liberty-ironic-scaling-the-dev-team Which I've summarized below, and mixed in several themes which didn't get captured on the 'pad but were discussed somewhere, possibly in a hallway or on Friday. Current challenges / observations: - six weeks' feature freeze is not actually having the desired stabilizing effect - the post-release/pre-summit slump only makes that worse - many folks stop reviewing during this time because they know their own features won't land, and instead focus their time downstream - this creates pressure to land features which aren't ready, and leaves few people to find bugs, write docs, and generally prepare the release during this window The alternative we discussed: - use feature branches for risky / large work, keeping total # of branches small, and rebasing them regularly on master - keep trunk moving quickly for smaller / less risky / refactoring changes - slow down for a week or two before a release, but dont actually freeze master - cut releases when new features are available - OpenStack coordinated releases are taken from latest independent release - that release will then get backports stable maintenance, other independent releases don't This sounds very much like what swift has been doing, which seems to work well. One thing I'll point out about feature branches - swift does the same thing - but by low what we mean _really_ is one that tends to have at least a one-cycle lifespan that represents SIGNIFICANT work Erasure Coding was the most recent one. I mention that because feature branches carry a cost here- but there are plenty of things on the internets that talk about the awesome things you can do with them - most of which are ways to work around not actually having real CI. In fact, for most risky things I'd suggest a feature flag and landing the patches into master. However, there are things that involve big refactors where hiding behind a flag would quickly get bong. So - sounds great and there is successful prior art - but just be wary of actually making use of the feature branch - and I counsel always asking yourself first is it possible to do this without one because the pain will be less. We think this will accomplish a few things: - make the developer experience better by being more consistent, thus keeping developers engaged year-round and increase the likelyhood they'll find and fix bugs - reduce stress on core reviewers since there's no crunch time at the end of a cycle - allow big changes to bake in a feature branch, rather than in a series of gerrit patches that need to be continually re-reviewed and cherry-picked to test them. - allow operators who wish to use Ironic outside of OpenStack to consume feature releases more rapidly, while still consuming approved releases instead of being forced to deploy from trunk For reference, Michael has posted a tracking change to the governance repo here: https://review.openstack.org/#/c/185203/ Before Ironic actually makes the switch, I would like us to discuss and document the approach we're going to take more fully, and get input from other teams on this approach. Often, the devil is in the details - and, for instance, I don't yet understand how we'll fit this approach into SemVer, or how this will affect our use of launchpad to track features (maybe it means we stop doing that?). Input appreciated. Thanks, Devananda __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
