Re: [DISCUSS] Release principles for Apache CloudStack
I’m back online and will continue working on this as soon as I worked through the many mails ;-) Happy to be your RM together with Rajani! Regards, Remi On 17 Jul 2015, at 17:04, sebgoa run...@gmail.commailto:run...@gmail.com wrote: Finally read the thread, It seems to me that a way forward is to have Remi and Rajani RM 4.6 (which is currently master). The two of them can discuss and start RMing 4.6 (PR merge etc) and then we can iterate on the wiki release scenario. @Remi @Rajani, would that work for you and you ready to get started ? -sebastien On Jul 10, 2015, at 8:17 PM, Daan Hoogland daan.hoogl...@gmail.commailto:daan.hoogl...@gmail.com wrote: I hate to be as opiniated as I am but here it comes ;) On Fri, Jul 10, 2015 at 7:39 PM, Rohit Yadav rohit.ya...@shapeblue.commailto:rohit.ya...@shapeblue.com wrote: While I like the ideas generally [1], some concerns and observations that I wish could be considered; - active contributor crunch: we don’t have large number of active people working towards testing, fixing bugs and release, and reviewing/merging PRs on *master*; this affects the agility of any process or workflow we want to put in, or expect resolution in a certain window (3-5 days etc.); This is a very valid concern. We are a large community but not in any way big enough. One approach is to let no backporting of bugfixes happen! it sound contrary to some of your points but I think it is actually a mitigation (see below). - diverse interests: our user-base may not necessarily want to upgrade to newer version of CloudStack even if they can proved to be quite stable; in-fact commercially some of us are paid to maintain stable branches and support users who are still on 4.2/4.3/4.4/4.5 etc; based on my experience, a typical enterprise users usually stick with a version (that works for them) for at least 6 months, while smb user or in-house consumers are quite agile who may upgrade as quickly as when new releases are made; User do go for bug fixes and are not concerned with any backwards compatible changes to functionality. If we guard against those, point releases are replaced by the minors and people can be as 'sticky' as they want. In the end it is a matter of naming and discipline. Of course we need to sell our policy. - diverse branching/merging workflow usage and understanding: the bugfix workflow may not be acceptable (to go on master first), a lot of people have their own way of branching/merging in their organisations that affect how they do it in the the project I do not think it is. If you want something fixed you should fix it on the oldest version it needs fixing on. No backporting at all. this only mystifies our git tree and prohibits good administration. Bug-fixes can be merged forward and whether anyone has one of infinite other possible release management schemes internally should not influence the way they work on this project. - waiting time on new changes: since we don’t have large number of active testers and developers who can fix bugs rapidly, freezing master and doing the release may take a lot of time (unless if we can put a hard deadline or have some schedule to support that?), in which case new features and refactoring work will have to lay hanging (and some rebase/code-rework may be needed later when they are merged, when master is open) lso very valid. No freeze, just release candidates would be a solution to that. One point in time is proposed as candidate and voted on. if it goes it goes, if it doesn't there will be new chances in the near future. We do depend on good quality control on master for this. - release risk: after a release x.y.0 is made and since master can receive new features, refactoring work; the next x.y.1 can potentially add more regressions due to those new features and refactoring/re-architectural work I don't agree here; any x.y.1 will be on a branch other then master. - release maintenance and support demands: historically there has been an assumed or known stable release that is fairly tested in the wild and has built trust due to usage by users (through meetups, users ML, customer interactions etc), in the past those versions were 4.2.1 then 4.3.1/4.3.2, now based on my experience the last stable release is 4.5.1 You know we don't agree on 4.4 vs 4.5 and I don't want to fight that fight here and now but how is this a concern either way? Any minor can have point releases following it if needed. [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack On 02-Jul-2015, at 5:16 pm, Remi Bergsma r...@remi.nl wrote: Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article:
Re: [DISCUSS] Release principles for Apache CloudStack
Finally read the thread, It seems to me that a way forward is to have Remi and Rajani RM 4.6 (which is currently master). The two of them can discuss and start RMing 4.6 (PR merge etc) and then we can iterate on the wiki release scenario. @Remi @Rajani, would that work for you and you ready to get started ? -sebastien On Jul 10, 2015, at 8:17 PM, Daan Hoogland daan.hoogl...@gmail.com wrote: I hate to be as opiniated as I am but here it comes ;) On Fri, Jul 10, 2015 at 7:39 PM, Rohit Yadav rohit.ya...@shapeblue.com wrote: While I like the ideas generally [1], some concerns and observations that I wish could be considered; - active contributor crunch: we don’t have large number of active people working towards testing, fixing bugs and release, and reviewing/merging PRs on *master*; this affects the agility of any process or workflow we want to put in, or expect resolution in a certain window (3-5 days etc.); This is a very valid concern. We are a large community but not in any way big enough. One approach is to let no backporting of bugfixes happen! it sound contrary to some of your points but I think it is actually a mitigation (see below). - diverse interests: our user-base may not necessarily want to upgrade to newer version of CloudStack even if they can proved to be quite stable; in-fact commercially some of us are paid to maintain stable branches and support users who are still on 4.2/4.3/4.4/4.5 etc; based on my experience, a typical enterprise users usually stick with a version (that works for them) for at least 6 months, while smb user or in-house consumers are quite agile who may upgrade as quickly as when new releases are made; User do go for bug fixes and are not concerned with any backwards compatible changes to functionality. If we guard against those, point releases are replaced by the minors and people can be as 'sticky' as they want. In the end it is a matter of naming and discipline. Of course we need to sell our policy. - diverse branching/merging workflow usage and understanding: the bugfix workflow may not be acceptable (to go on master first), a lot of people have their own way of branching/merging in their organisations that affect how they do it in the the project I do not think it is. If you want something fixed you should fix it on the oldest version it needs fixing on. No backporting at all. this only mystifies our git tree and prohibits good administration. Bug-fixes can be merged forward and whether anyone has one of infinite other possible release management schemes internally should not influence the way they work on this project. - waiting time on new changes: since we don’t have large number of active testers and developers who can fix bugs rapidly, freezing master and doing the release may take a lot of time (unless if we can put a hard deadline or have some schedule to support that?), in which case new features and refactoring work will have to lay hanging (and some rebase/code-rework may be needed later when they are merged, when master is open) lso very valid. No freeze, just release candidates would be a solution to that. One point in time is proposed as candidate and voted on. if it goes it goes, if it doesn't there will be new chances in the near future. We do depend on good quality control on master for this. - release risk: after a release x.y.0 is made and since master can receive new features, refactoring work; the next x.y.1 can potentially add more regressions due to those new features and refactoring/re-architectural work I don't agree here; any x.y.1 will be on a branch other then master. - release maintenance and support demands: historically there has been an assumed or known stable release that is fairly tested in the wild and has built trust due to usage by users (through meetups, users ML, customer interactions etc), in the past those versions were 4.2.1 then 4.3.1/4.3.2, now based on my experience the last stable release is 4.5.1 You know we don't agree on 4.4 vs 4.5 and I don't want to fight that fight here and now but how is this a concern either way? Any minor can have point releases following it if needed. [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack On 02-Jul-2015, at 5:16 pm, Remi Bergsma r...@remi.nl wrote: Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudSta ck https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer
Re: [DISCUSS] Release principles for Apache CloudStack
Rene, Remi, I read back the thread and have another answer on this for you; The branch contains a point version number instead of a x.y.z-SNAPSHOT, without the branch a revert commit must follow if we vote them out. a branch can be simply neglected. On Thu, Jul 2, 2015 at 4:46 PM, Remi Bergsma r...@remi.nl wrote: Hi René, The reason is that I tried to stay close to how it is done now so we could reuse the scrips. You do have a valid point, as indeed no commits are expected (nor should be allowed) until the vote passes. Pinging @dahn to ask if he knows of other reasons to use a branch for RC and if he foresees any issues if we'd switch to a release tag instead (and branch off of it once vote passes). Regards, Remi On 02 Jul 2015, at 16:16, Rene Moser m...@renemoser.net wrote: Hi Remi On 02.07.2015 13:46, Remi Bergsma wrote: I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) It is always good to see this release process which looks +/- identical to successful Linux kernel release process. :) The only thing I am curious about, if I get it right: Why do you branch off for RC? Why not just tag a commit in branch master as RC and branch off once it is releases from a release tag v.x.y? Because it is unlikely you make any commits on that branch anyway. Or did I miss anything? Yours René -- Daan
Re: [DISCUSS] Release principles for Apache CloudStack
While I like the ideas generally [1], some concerns and observations that I wish could be considered; - active contributor crunch: we don’t have large number of active people working towards testing, fixing bugs and release, and reviewing/merging PRs on *master*; this affects the agility of any process or workflow we want to put in, or expect resolution in a certain window (3-5 days etc.); - diverse interests: our user-base may not necessarily want to upgrade to newer version of CloudStack even if they can proved to be quite stable; in-fact commercially some of us are paid to maintain stable branches and support users who are still on 4.2/4.3/4.4/4.5 etc; based on my experience, a typical enterprise users usually stick with a version (that works for them) for at least 6 months, while smb user or in-house consumers are quite agile who may upgrade as quickly as when new releases are made; - diverse branching/merging workflow usage and understanding: the bugfix workflow may not be acceptable (to go on master first), a lot of people have their own way of branching/merging in their organisations that affect how they do it in the the project - waiting time on new changes: since we don’t have large number of active testers and developers who can fix bugs rapidly, freezing master and doing the release may take a lot of time (unless if we can put a hard deadline or have some schedule to support that?), in which case new features and refactoring work will have to lay hanging (and some rebase/code-rework may be needed later when they are merged, when master is open) - release risk: after a release x.y.0 is made and since master can receive new features, refactoring work; the next x.y.1 can potentially add more regressions due to those new features and refactoring/re-architectural work - release maintenance and support demands: historically there has been an assumed or known stable release that is fairly tested in the wild and has built trust due to usage by users (through meetups, users ML, customer interactions etc), in the past those versions were 4.2.1 then 4.3.1/4.3.2, now based on my experience the last stable release is 4.5.1 [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack On 02-Jul-2015, at 5:16 pm, Remi Bergsma r...@remi.nlmailto:r...@remi.nl wrote: Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) Like folks suggested earlier, it would be nice to work on this with multiple people. So, feel free to join. Maybe @dahn @bhaisaab and/or others are willing to teach me some of their tricks. Regards, Remi Regards, Rohit Yadav Software Architect, ShapeBlue [cid:9DD97B41-04C5-45F0-92A7-951F3E962F7A] M. +91 88 262 30892 | rohit.ya...@shapeblue.commailto:rohit.ya...@shapeblue.com Blog: bhaisaab.orghttp://bhaisaab.org | Twitter: @_bhaisaab Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design Buildhttp://shapeblue.com/iaas-cloud-design-and-build// CSForge – rapid IaaS deployment frameworkhttp://shapeblue.com/csforge/ CloudStack Consultinghttp://shapeblue.com/cloudstack-consultancy/ CloudStack Software Engineeringhttp://shapeblue.com/cloudstack-software-engineering/ CloudStack Infrastructure Supporthttp://shapeblue.com/cloudstack-infrastructure-support/ CloudStack Bootcamp Training Courseshttp://shapeblue.com/cloudstack-training/ This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: [DISCUSS] Release principles for Apache CloudStack
Hi Rajani, Happy to hear we agree on the goal and on the release process. Also great you want to join the RM effort! What remains is a discussion about the maintenance. I think the difference in our approaches, is that you say “commit to release branch, then bring it to master” and I say the opposite: “commit to master (pr) and then bring to release branch”. There are pros/cons for each. Let’s first see if we agree on more parts: - maintenance releases for x.y should only receive bug fixes, no new features - we should release often and only maintain the current x.y release (and master obviously) - upgrades should be smooth so there is no reason to stay on older releases (we should help users upgrade) I’m trying to understand how the merge workflow would look like. As I’m not a hardcore dev, I probably just miss the experience. Let me play a bit with it in order to get a good understanding of what you say. That makes it a better discussion later on. Have a nice weekend, Regards, Remi On 3 jul. 2015, at 13:01, Rajani Karuturi raj...@apache.org wrote: I agree to the goal. No regressions can be achieved either by cherry-picks or merges. Both requires developers to agree to it and has equal probability to miss. The advantage in case of merge is, lets say you do a commit a1 to 4.5 branch and forget to merge to master. now, someone else did a2 and is trying to merge. He will see both the commits and can make choice either to ask about a1 or merge both. may be if the merge has conflicts or is not a clean merge, it can be done as PR which follows the usual process. with respect to refactoring, we can always do separate PRs on master and the release branch. The merge from release branch to master in this case would be an empty merge (git merge -s ours) The problem I see with the current approach is its not always backporting the fix to release. This works if the issue reported is already known and fixed on master or any forward releases. Usually, you work on a release branch only when an issue is reported on it. For a issue reported on 4.5, first thing I do is check if the issue really exists in 4.5 and would also find the solution around the same time. Its sounds natural to me to push to 4.5(get the load off your head whether it is customer issue or security issue) and then merge to forward releases. Now, with the new process, I will have to first verify if the bug exists in the current release+1, fix in it first, then go back to release branch, verify if it exists and fix it there. Obviously there would be exceptions in merging and you may have to cherry-pick once in a while. But, making it the process is what concerns me. In addition to the goal mentioned, I would also like to find which all versions the commit is present in. For example, If a security issue is reported on 4.5, I found that this is a regression from commit abcde. Now, I would like to find which all version this is present as this security patch should be delivered on all the versions. In the current situation and also in the new process suggested, the only way to find it is to go to each and every release and see if the commit is present. In the merge route, its simple git merge --contains abcde Merging is not easy. It requires discipline. But, it has benefits. If everyone follows it, its as easy as cherry-picking as the code diff will be exactly same. Maintaining a release is the second part of the story :) I do agree with you on the release process for 4.6. Lets start with it for 4.6 release and continue discussing on how we want to maintain 4.6.x(may be we would never need one ;)) I can volunteer to be the additional RM for 4.6. ~Rajani On Fri, Jul 3, 2015 at 1:36 PM, Remi Bergsma r...@remi.nl mailto:r...@remi.nl wrote: Hi Rajani, all, I do think we have the same goal in mind: no regression and no cherry-pick mess. Just did some reading on tofu scale and see two issues: - if it doesn't happen / isn't completed we'll have regression as we've seen before. I want a working model that prevents regression by default or else we will not reach smooth upgrades without introducing old bugs again. - we are doing lots of refactoring work (probably will be doing even more) where this model also will not work because the merge fails to apply cleanly (this is what Erik mentioned yesterday). Fixing that conflict is invisible on the dev-list (as we have no PR) and error prone (not tested before commit, no LGTMs). Don’t get me wrong, I don’t want the cherry-pick mess we experienced before. What will be different: - master is stable instead of unstable - we’ll branch off x.y (say 4.6 which is up next) only after a vote on a RC passes. Instead of branching it off first, then make it stable and release it and cherry-pick stuff around. So, no cherry-picking here. - once the x.y branch is there, we should do no back ports of features. Only
RE: [DISCUSS] Release principles for Apache CloudStack
Remi, couple of questions on the branching part - when we take the Feature PR and Feature is back in Master, feel like we are potentially destabilizing Master ? I know, currently we push changes to master even before anything is tested fully - agree, we are now running the Travis test before a checkin - however, I feel those are not sufficient ? IMHO - we should take a release branch open it up for PR/checkins and once the testing is done the branch gets into Master - we take RC from the master and release it. That way no one checkins to master and constantly tested changes get into/merged to master. I remember seeing similar changes proposed by few folks... I have been little out of touch on those changes. best, Raja -Original Message- From: Rajani Karuturi [mailto:raj...@apache.org] Sent: Friday, July 3, 2015 8:00 AM To: dev@cloudstack.apache.org Subject: Re: [DISCUSS] Release principles for Apache CloudStack I do not agree to backporting aka cherry picking. I prefer forward merges(tofu scale) 4.4 to 4.5 to master etc. That way, none of the changes will be missed and git branch --contains gives a nice view of where all the changes went. On Thu, Jul 2, 2015 at 23:16 PM, Remi Bergsma r...@remi.nl wrote: Hi Daan, Indeed. I prefer committing to master first, as it will ensure everything ends up there (unless some specific use cases). Currently, we have the risk of forgetting to include a fix to a release branch back to master. When we reverse it, some bug fix that should end up in the x.y branch, is committed to master, then also applied (or reimplemented) to x.y. If you then only take one of the two steps, there is no issue as it will be in master only (and people will notice). In the other situation, when we accept a PR to x.y and forget to merge back, we possibly introduce regression bugs. I will update the diagram and wiki later tonight. While reviewing PRs, let’s be alert to see PRs not pointed towards master and at least discuss it. Regards, Remi On 2 jul. 2015, at 16:54, Daan Hoogland daan.hoogl...@gmail.com javascript:; wrote: On Thu, Jul 2, 2015 at 2:29 PM, Remi Bergsma r...@remi.nl javascript:; wrote: Since the goal is a stable master, I’d say the bug fix should go to master first. Remi, this means that merge back of the branch makes no sense anymore. -- Daan -- - Sent from Windows Phone ~Rajani
Re: [DISCUSS] Release principles for Apache CloudStack
On Jul 3, 2015, at 9:06 AM, Raja Pullela raja.pull...@citrix.com wrote: Remi, couple of questions on the branching part - when we take the Feature PR and Feature is back in Master, feel like we are potentially destabilizing Master ? I know, currently we push changes to master even before anything is tested fully - agree, we are now running the Travis test before a checkin - however, I feel those are not sufficient ? IMHO - we should take a release branch open it up for PR/checkins and once the testing is done the branch gets into Master - we take RC from the master and release it. That way no one checkins to master and constantly tested changes get into/merged to master. I remember seeing similar changes proposed by few folks... I have been little out of touch on those changes. Basically yes, we should not merge untested, unfinished features in master. best, Raja -Original Message- From: Rajani Karuturi [mailto:raj...@apache.org] Sent: Friday, July 3, 2015 8:00 AM To: dev@cloudstack.apache.org Subject: Re: [DISCUSS] Release principles for Apache CloudStack I do not agree to backporting aka cherry picking. I prefer forward merges(tofu scale) 4.4 to 4.5 to master etc. That way, none of the changes will be missed and git branch --contains gives a nice view of where all the changes went. On Thu, Jul 2, 2015 at 23:16 PM, Remi Bergsma r...@remi.nl wrote: Hi Daan, Indeed. I prefer committing to master first, as it will ensure everything ends up there (unless some specific use cases). Currently, we have the risk of forgetting to include a fix to a release branch back to master. When we reverse it, some bug fix that should end up in the x.y branch, is committed to master, then also applied (or reimplemented) to x.y. If you then only take one of the two steps, there is no issue as it will be in master only (and people will notice). In the other situation, when we accept a PR to x.y and forget to merge back, we possibly introduce regression bugs. I will update the diagram and wiki later tonight. While reviewing PRs, let’s be alert to see PRs not pointed towards master and at least discuss it. Regards, Remi On 2 jul. 2015, at 16:54, Daan Hoogland daan.hoogl...@gmail.com javascript:; wrote: On Thu, Jul 2, 2015 at 2:29 PM, Remi Bergsma r...@remi.nl javascript:; wrote: Since the goal is a stable master, I’d say the bug fix should go to master first. Remi, this means that merge back of the branch makes no sense anymore. -- Daan -- - Sent from Windows Phone ~Rajani
Re: [DISCUSS] Release principles for Apache CloudStack
On 3 jul. 2015, at 09:21, sebgoa run...@gmail.com wrote: On Jul 3, 2015, at 9:06 AM, Raja Pullela raja.pull...@citrix.com wrote: Remi, couple of questions on the branching part - when we take the Feature PR and Feature is back in Master, feel like we are potentially destabilizing Master ? I know, currently we push changes to master even before anything is tested fully - agree, we are now running the Travis test before a checkin - however, I feel those are not sufficient ? IMHO - we should take a release branch open it up for PR/checkins and once the testing is done the branch gets into Master - we take RC from the master and release it. That way no one checkins to master and constantly tested changes get into/merged to master. I remember seeing similar changes proposed by few folks... I have been little out of touch on those changes. Basically yes, we should not merge untested, unfinished features in master Exactly. Whatever is merged to master “should just work (tm)”. The purpose of having people say this LGTM, is also to have them verify / test it. In a perfect world this testing is automated. Until we’re there, we need to rely on people to do this. Regards, Remi
Re: [DISCUSS] Release principles for Apache CloudStack
Hi Rajani, all, I do think we have the same goal in mind: no regression and no cherry-pick mess. Just did some reading on tofu scale and see two issues: - if it doesn't happen / isn't completed we'll have regression as we've seen before. I want a working model that prevents regression by default or else we will not reach smooth upgrades without introducing old bugs again. - we are doing lots of refactoring work (probably will be doing even more) where this model also will not work because the merge fails to apply cleanly (this is what Erik mentioned yesterday). Fixing that conflict is invisible on the dev-list (as we have no PR) and error prone (not tested before commit, no LGTMs). Don’t get me wrong, I don’t want the cherry-pick mess we experienced before. What will be different: - master is stable instead of unstable - we’ll branch off x.y (say 4.6 which is up next) only after a vote on a RC passes. Instead of branching it off first, then make it stable and release it and cherry-pick stuff around. So, no cherry-picking here. - once the x.y branch is there, we should do no back ports of features. Only (critical?) bug fixes. Those bug fixes (that went to master first through PR and got 2x LGTM) could now also be applied to the x.y. branch (and become x.y.1 or similar) Instead of cherry-picking, I’d apply the same PR to x.y (if possible, or else a new PR that reimplements the fix for x.y in which case it would need its own 2x LGTM). In any bug fix PR we should mention the jira bug id (and PR id) in the commit message. This, you could argue, is like cherry-picking as it makes a new commit id. As the amount of bug fix commits should me minimal, I think this is ok. We should make sure upgrades are so smooth that our users can upgrade easily and we don’t have to maintain previous releases for a long time. I clarified some more things on the wiki and also updated the diagrams. Regards, Remi On 3 jul. 2015, at 04:29, Rajani Karuturi raj...@apache.org wrote: I do not agree to backporting aka cherry picking. I prefer forward merges(tofu scale) 4.4 to 4.5 to master etc. That way, none of the changes will be missed and git branch --contains gives a nice view of where all the changes went. On Thu, Jul 2, 2015 at 23:16 PM, Remi Bergsma r...@remi.nl mailto:r...@remi.nl wrote: Hi Daan, Indeed. I prefer committing to master first, as it will ensure everything ends up there (unless some specific use cases). Currently, we have the risk of forgetting to include a fix to a release branch back to master. When we reverse it, some bug fix that should end up in the x.y branch, is committed to master, then also applied (or reimplemented) to x.y. If you then only take one of the two steps, there is no issue as it will be in master only (and people will notice). In the other situation, when we accept a PR to x.y and forget to merge back, we possibly introduce regression bugs. I will update the diagram and wiki later tonight. While reviewing PRs, let’s be alert to see PRs not pointed towards master and at least discuss it. Regards, Remi On 2 jul. 2015, at 16:54, Daan Hoogland daan.hoogl...@gmail.com javascript:; wrote: On Thu, Jul 2, 2015 at 2:29 PM, Remi Bergsma r...@remi.nl mailto:r...@remi.nl javascript:; wrote: Since the goal is a stable master, I’d say the bug fix should go to master first. Remi, this means that merge back of the branch makes no sense anymore. -- Daan -- - Sent from Windows Phone ~Rajani
Re: [DISCUSS] Release principles for Apache CloudStack
On Fri, Jul 3, 2015 at 10:06 AM, Remi Bergsma r...@remi.nl wrote: Hi Rajani, all, I do think we have the same goal in mind: no regression and no cherry-pick mess. Just did some reading on tofu scale and see two issues: - if it doesn't happen / isn't completed we'll have regression as we've seen before. I want a working model that prevents regression by default or else we will not reach smooth upgrades without introducing old bugs again. - we are doing lots of refactoring work (probably will be doing even more) where this model also will not work because the merge fails to apply cleanly (this is what Erik mentioned yesterday). Fixing that conflict is invisible on the dev-list (as we have no PR) and error prone (not tested before commit, no LGTMs). Don’t get me wrong, I don’t want the cherry-pick mess we experienced before. What will be different: - master is stable instead of unstable - we’ll branch off x.y (say 4.6 which is up next) only after a vote on a RC passes. Instead of branching it off first, then make it stable and release it and cherry-pick stuff around. So, no cherry-picking here. - once the x.y branch is there, we should do no back ports of features. Only (critical?) bug fixes. Those bug fixes (that went to master first through PR and got 2x LGTM) could now also be applied to the x.y. branch (and become x.y.1 or similar) Instead of cherry-picking, I’d apply the same PR to x.y (if possible, or else a new PR that reimplements the fix for x.y in which case it would need its own 2x LGTM). In any bug fix PR we should mention the jira bug id (and PR id) in the commit message. This, you could argue, is like cherry-picking as it makes a new commit id. As the amount of bug fix commits should me minimal, I think this is ok. We should make sure upgrades are so smooth that our users can upgrade easily and we don’t have to maintain previous releases for a long time. I clarified some more things on the wiki and also updated the diagrams. I'm no hardcore dev, so take my opinions with a grain of salt, but this looks reasonable to me and a step in the right direction. +1 -- Erik
Re: [DISCUSS] Release principles for Apache CloudStack
I agree to the goal. No regressions can be achieved either by cherry-picks or merges. Both requires developers to agree to it and has equal probability to miss. The advantage in case of merge is, lets say you do a commit a1 to 4.5 branch and forget to merge to master. now, someone else did a2 and is trying to merge. He will see both the commits and can make choice either to ask about a1 or merge both. may be if the merge has conflicts or is not a clean merge, it can be done as PR which follows the usual process. with respect to refactoring, we can always do separate PRs on master and the release branch. The merge from release branch to master in this case would be an empty merge (git merge -s ours) The problem I see with the current approach is its not always backporting the fix to release. This works if the issue reported is already known and fixed on master or any forward releases. Usually, you work on a release branch only when an issue is reported on it. For a issue reported on 4.5, first thing I do is check if the issue really exists in 4.5 and would also find the solution around the same time. Its sounds natural to me to push to 4.5(get the load off your head whether it is customer issue or security issue) and then merge to forward releases. Now, with the new process, I will have to first verify if the bug exists in the current release+1, fix in it first, then go back to release branch, verify if it exists and fix it there. Obviously there would be exceptions in merging and you may have to cherry-pick once in a while. But, making it the process is what concerns me. In addition to the goal mentioned, I would also like to find which all versions the commit is present in. For example, If a security issue is reported on 4.5, I found that this is a regression from commit abcde. Now, I would like to find which all version this is present as this security patch should be delivered on all the versions. In the current situation and also in the new process suggested, the only way to find it is to go to each and every release and see if the commit is present. In the merge route, its simple git merge --contains abcde Merging is not easy. It requires discipline. But, it has benefits. If everyone follows it, its as easy as cherry-picking as the code diff will be exactly same. Maintaining a release is the second part of the story :) I do agree with you on the release process for 4.6. Lets start with it for 4.6 release and continue discussing on how we want to maintain 4.6.x(may be we would never need one ;)) I can volunteer to be the additional RM for 4.6. ~Rajani On Fri, Jul 3, 2015 at 1:36 PM, Remi Bergsma r...@remi.nl wrote: Hi Rajani, all, I do think we have the same goal in mind: no regression and no cherry-pick mess. Just did some reading on tofu scale and see two issues: - if it doesn't happen / isn't completed we'll have regression as we've seen before. I want a working model that prevents regression by default or else we will not reach smooth upgrades without introducing old bugs again. - we are doing lots of refactoring work (probably will be doing even more) where this model also will not work because the merge fails to apply cleanly (this is what Erik mentioned yesterday). Fixing that conflict is invisible on the dev-list (as we have no PR) and error prone (not tested before commit, no LGTMs). Don’t get me wrong, I don’t want the cherry-pick mess we experienced before. What will be different: - master is stable instead of unstable - we’ll branch off x.y (say 4.6 which is up next) only after a vote on a RC passes. Instead of branching it off first, then make it stable and release it and cherry-pick stuff around. So, no cherry-picking here. - once the x.y branch is there, we should do no back ports of features. Only (critical?) bug fixes. Those bug fixes (that went to master first through PR and got 2x LGTM) could now also be applied to the x.y. branch (and become x.y.1 or similar) Instead of cherry-picking, I’d apply the same PR to x.y (if possible, or else a new PR that reimplements the fix for x.y in which case it would need its own 2x LGTM). In any bug fix PR we should mention the jira bug id (and PR id) in the commit message. This, you could argue, is like cherry-picking as it makes a new commit id. As the amount of bug fix commits should me minimal, I think this is ok. We should make sure upgrades are so smooth that our users can upgrade easily and we don’t have to maintain previous releases for a long time. I clarified some more things on the wiki and also updated the diagrams. Regards, Remi On 3 jul. 2015, at 04:29, Rajani Karuturi raj...@apache.org wrote: I do not agree to backporting aka cherry picking. I prefer forward merges(tofu scale) 4.4 to 4.5 to master etc. That way, none of the changes will be missed and git branch --contains gives a nice view of where all the changes went. On Thu, Jul 2, 2015 at 23:16 PM, Remi
[DISCUSS] Release principles for Apache CloudStack
Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) Like folks suggested earlier, it would be nice to work on this with multiple people. So, feel free to join. Maybe @dahn @bhaisaab and/or others are willing to teach me some of their tricks. Regards, Remi
RE: [DISCUSS] Release principles for Apache CloudStack
Remi, Release process looks good to me. Can you also add some info about maintenance release? - After release, will the fixes go into x.y branch and merged back to master? - or bug fixes go into master and selectively merged back to x.y branch? -Original Message- From: Remi Bergsma [mailto:r...@remi.nl] Sent: 02 July 2015 17:17 To: dev@cloudstack.apache.org Subject: [DISCUSS] Release principles for Apache CloudStack Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) Like folks suggested earlier, it would be nice to work on this with multiple people. So, feel free to join. Maybe @dahn @bhaisaab and/or others are willing to teach me some of their tricks. Regards, Remi
Re: [DISCUSS] Release principles for Apache CloudStack
Hi Erik, Kishan, Since the goal is a stable master, I’d say the bug fix should go to master first. This way we make sure all upcoming releases have the fix included. I’ll add this to the wiki. Then we should decide if the fix should go back to other release branches as wel. When we get a PR that is not against master, we should be extra careful. Maybe we should ask people to specify why they want to do this. There could be a valid reason, of course. The danger is introducing old bugs again if people upgrade and I want to prevent that. In Erik’s example: The first PR is against master and fixes SomeRandomFunction(). Now we know any release branched off master has this fix. If we decide we want to fix 4.6 as well, a second PR should be created against 4.6 (reimplementation of the fix to older version as per Erik's example). In a perfect world we should be able to merge back from 4.6 to master. But for example i case of refactoring this is not possible. Regards, Remi On 2 jul. 2015, at 14:05, Erik Weber terbol...@gmail.com wrote: How about bugfix PRs that involves refactored code between versions? Example: - 4.6.0 is released - SomeRandomFunction() is refactored - 4.7.0 is released - A bug is discovered in SomeRandomFunction() and applies to both 4.6.0 and 4.7.0 - A bugfix PR is sent to the 4.6 branch I understand that the code has to be ported to both versions, but how do we ensure/enforce that it actually happens? -- Erik On Thu, Jul 2, 2015 at 2:03 PM, Kishan Kavala kishan.kav...@citrix.com wrote: Remi, Release process looks good to me. Can you also add some info about maintenance release? - After release, will the fixes go into x.y branch and merged back to master? - or bug fixes go into master and selectively merged back to x.y branch? -Original Message- From: Remi Bergsma [mailto:r...@remi.nl] Sent: 02 July 2015 17:17 To: dev@cloudstack.apache.org Subject: [DISCUSS] Release principles for Apache CloudStack Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) Like folks suggested earlier, it would be nice to work on this with multiple people. So, feel free to join. Maybe @dahn @bhaisaab and/or others are willing to teach me some of their tricks. Regards, Remi
Re: [DISCUSS] Release principles for Apache CloudStack
How about bugfix PRs that involves refactored code between versions? Example: - 4.6.0 is released - SomeRandomFunction() is refactored - 4.7.0 is released - A bug is discovered in SomeRandomFunction() and applies to both 4.6.0 and 4.7.0 - A bugfix PR is sent to the 4.6 branch I understand that the code has to be ported to both versions, but how do we ensure/enforce that it actually happens? -- Erik On Thu, Jul 2, 2015 at 2:03 PM, Kishan Kavala kishan.kav...@citrix.com wrote: Remi, Release process looks good to me. Can you also add some info about maintenance release? - After release, will the fixes go into x.y branch and merged back to master? - or bug fixes go into master and selectively merged back to x.y branch? -Original Message- From: Remi Bergsma [mailto:r...@remi.nl] Sent: 02 July 2015 17:17 To: dev@cloudstack.apache.org Subject: [DISCUSS] Release principles for Apache CloudStack Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) Like folks suggested earlier, it would be nice to work on this with multiple people. So, feel free to join. Maybe @dahn @bhaisaab and/or others are willing to teach me some of their tricks. Regards, Remi
Re: [DISCUSS] Release principles for Apache CloudStack
Thanks! Appreciate it :-) On 02 Jul 2015, at 13:51, Daan Hoogland daan.hoogl...@gmail.com wrote: be sure you're backed On Thu, Jul 2, 2015 at 1:46 PM, Remi Bergsma r...@remi.nl wrote: Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) Like folks suggested earlier, it would be nice to work on this with multiple people. So, feel free to join. Maybe @dahn @bhaisaab and/or others are willing to teach me some of their tricks. Regards, Remi -- Daan
Re: [DISCUSS] Release principles for Apache CloudStack
Hi Remi On 02.07.2015 13:46, Remi Bergsma wrote: I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) It is always good to see this release process which looks +/- identical to successful Linux kernel release process. :) The only thing I am curious about, if I get it right: Why do you branch off for RC? Why not just tag a commit in branch master as RC and branch off once it is releases from a release tag v.x.y? Because it is unlikely you make any commits on that branch anyway. Or did I miss anything? Yours René
Re: [DISCUSS] Release principles for Apache CloudStack
be sure you're backed On Thu, Jul 2, 2015 at 1:46 PM, Remi Bergsma r...@remi.nl wrote: Hi all, We already agreed contributions should always go via a PR and require two LGTM’s before we merge. Let me propose the next step on how I think we should do release management for 4.6 and on. I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) Like folks suggested earlier, it would be nice to work on this with multiple people. So, feel free to join. Maybe @dahn @bhaisaab and/or others are willing to teach me some of their tricks. Regards, Remi -- Daan
Re: [DISCUSS] Release principles for Apache CloudStack
On Thu, Jul 2, 2015 at 2:29 PM, Remi Bergsma r...@remi.nl wrote: Since the goal is a stable master, I’d say the bug fix should go to master first. Remi, this means that merge back of the branch makes no sense anymore. -- Daan
Re: [DISCUSS] Release principles for Apache CloudStack
Hi René, The reason is that I tried to stay close to how it is done now so we could reuse the scrips. You do have a valid point, as indeed no commits are expected (nor should be allowed) until the vote passes. Pinging @dahn to ask if he knows of other reasons to use a branch for RC and if he foresees any issues if we'd switch to a release tag instead (and branch off of it once vote passes). Regards, Remi On 02 Jul 2015, at 16:16, Rene Moser m...@renemoser.net wrote: Hi Remi On 02.07.2015 13:46, Remi Bergsma wrote: I talked to several people over the past weeks and wrote this wiki article: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack If you like this way of working, I volunteer to be your RM :-) It is always good to see this release process which looks +/- identical to successful Linux kernel release process. :) The only thing I am curious about, if I get it right: Why do you branch off for RC? Why not just tag a commit in branch master as RC and branch off once it is releases from a release tag v.x.y? Because it is unlikely you make any commits on that branch anyway. Or did I miss anything? Yours René
Re: [DISCUSS] Release principles for Apache CloudStack
I do not agree to backporting aka cherry picking. I prefer forward merges(tofu scale) 4.4 to 4.5 to master etc. That way, none of the changes will be missed and git branch --contains gives a nice view of where all the changes went. On Thu, Jul 2, 2015 at 23:16 PM, Remi Bergsma r...@remi.nl wrote: Hi Daan, Indeed. I prefer committing to master first, as it will ensure everything ends up there (unless some specific use cases). Currently, we have the risk of forgetting to include a fix to a release branch back to master. When we reverse it, some bug fix that should end up in the x.y branch, is committed to master, then also applied (or reimplemented) to x.y. If you then only take one of the two steps, there is no issue as it will be in master only (and people will notice). In the other situation, when we accept a PR to x.y and forget to merge back, we possibly introduce regression bugs. I will update the diagram and wiki later tonight. While reviewing PRs, let’s be alert to see PRs not pointed towards master and at least discuss it. Regards, Remi On 2 jul. 2015, at 16:54, Daan Hoogland daan.hoogl...@gmail.com javascript:; wrote: On Thu, Jul 2, 2015 at 2:29 PM, Remi Bergsma r...@remi.nl javascript:; wrote: Since the goal is a stable master, I’d say the bug fix should go to master first. Remi, this means that merge back of the branch makes no sense anymore. -- Daan -- - Sent from Windows Phone ~Rajani
Re: [DISCUSS] Release principles for Apache CloudStack
Hi Daan, Indeed. I prefer committing to master first, as it will ensure everything ends up there (unless some specific use cases). Currently, we have the risk of forgetting to include a fix to a release branch back to master. When we reverse it, some bug fix that should end up in the x.y branch, is committed to master, then also applied (or reimplemented) to x.y. If you then only take one of the two steps, there is no issue as it will be in master only (and people will notice). In the other situation, when we accept a PR to x.y and forget to merge back, we possibly introduce regression bugs. I will update the diagram and wiki later tonight. While reviewing PRs, let’s be alert to see PRs not pointed towards master and at least discuss it. Regards, Remi On 2 jul. 2015, at 16:54, Daan Hoogland daan.hoogl...@gmail.com wrote: On Thu, Jul 2, 2015 at 2:29 PM, Remi Bergsma r...@remi.nl wrote: Since the goal is a stable master, I’d say the bug fix should go to master first. Remi, this means that merge back of the branch makes no sense anymore. -- Daan
