I agree, but this release process wouldn't lead to long-lived feature branches since all development would still be done on master and merged into master. Unstable or incomplete features could potentially delay releases from going out, but I agree that feature flags could mitigate those problems. Anything we can do to keep the master branch deployable is a +1 in my book.
- Rawlin On Thu, Oct 31, 2019 at 2:46 PM Genz, Geoffrey <[email protected]> wrote: > > Just throwing out that the judicious use of feature flags could reduce the > pain from unstable or incomplete new features. Long lived feature branches > are something I believe should be avoided. > > Geoff > > On 10/31/19, 2:32 PM, "Rawlin Peters" <[email protected]> wrote: > > Alright, I think that sounds fair. In general what I'm hearing is, > every 4-6 weeks: > 1. fast-forward 4.x to the most recent "stable" commit of master > (maybe we automate this and fast-forward every day if some set of > tests pass, unless we explicitly know master is unstable for other > reasons) > 2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into > 4.*.x that are necessary for the release (e.g. bugs that break the > data plane or prevent operators from being able to do their job or run > the control plane) > > Which means the release manager needs to know/track: > - unstable commits along with their corresponding fixes > - incomplete features > - features we'd like to hold until the next release (due to current > release having enough already) > - any cherry-picks that were deemed necessary for a certain release > (so that they can either be carried forward into the next minor > release or, ideally, included as part of fast-forwarding 4.x to the > most recent stable commit on master) > > I think this will work. We gain the ability to: > 1. let master be "unstable" periodically > 2. control the size of releases > 3. make releases mirror master as closely as possible (because they > will just be a snapshot of the most recent version of master that was > considered stable -- unless absolutely necessary bug/security fixes > were cherry-picked in) > > Does that all sound right, Dave? > > - Rawlin > > > > > > > > On Thu, Oct 31, 2019 at 8:37 AM David Neuman <[email protected]> > wrote: > > > > I think a third reason would be that we are getting very close to our > > release branch date, it's a large/complex feature, and we feel like the > > release already has enough in it. In that case we could hold off on > adding > > that change to the 4.x branch until after we cut the 4.x.x branch. The > > benefit is that we can still merge that feature into master and get all > of > > our testing (both internally and externally) running against it. If > you > > stick with just branching off master and this situation arises then we > get > > in the spot where we just don't merge the PR at all until after the > release > > branch is cut (we have two examples of this right now). I personally > think > > it's better to get this into master and let our automation run against > it > > than it is to leave it out in a PR even though it is ready. > > > > As for the process, I was thinking the same thing except that you don't > > even need to cherry-pick. In your scenario above you can just rebase > > master into 4.x once E was merged. This avoids the headaches of having > to > > cherry pick each commit while still giving us the benefit of having all > of > > the changes on the release branch. We can even decide to setup > automation > > to rebase master into the release branch on an interval or on demand. > If > > we rebase in something that turns out to be broken we just roll back to > the > > previous rebase (or a certain commit) until that is fixed and then > rebase > > again. If a feature gets merged to master that we aren't ready to put > into > > the release, then we stop the automated rebase until we are ready to > get it > > into the release -- at that point we should consider ONLY cherry-picking > > bug fixes for the remainder of that release cycle to make things easier > to > > rebase again. > > > > I am sure there will be unintended consequences, and I am sure we will > have > > challenges to work through, but I still think that it's worth trying. > We > > should fail fast and be willing to change course if necessary. > > I think if we can be consistent enough with our releases then maybe we > can > > discuss branching from master again. > > > > Thanks, > > Dave > > > > > > > > On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <[email protected]> wrote: > > > > > I guess my concern is really about what possible reasons we might have > > > to not cherry-pick a specific PR that had been merged into master > > > already. The only reasons I can think of to not cherry-pick something > > > would be: > > > 1. the PR was completely broken (e.g. breaks the build, tests don't > > > pass, doesn't compile, etc). > > > 2. the PR one one of a larger multi-PR "feature", and *we know* > > > releasing some parts early could cause problems (that knowing part is > > > crucial) > > > > > > We should always strive to catch issue #1 in our CI before it merges > > > into master. When things like that merge it can block people from > > > doing development starting at that point until it's fixed. > > > I can see why issue #2 might arise due to multiple different people > > > developing one larger feature, but we should still strive to break > > > larger features up into individual pieces that can reasonably stand on > > > their own if possible. Due to time constraints or other reasons I can > > > see that might be a little difficult. > > > > > > That said, I would still strive to make the 4.x release branch a > > > snapshot of master at some "known good point" in time. Let's say we > > > cut 4.x at commit A, then a broken commit B gets merged into master. > > > If it's completely obvious that it's a broken commit, I'd wait until > > > at least a commit C gets merged to fix it. Then I would cherry-pick B > > > and C into 4.x at the same time. If the fix is really like a commit E > > > with unrelated commits C and D in between, ideally I would still wait > > > to cherry-pick C and D until a "fix" commit E was merged (to fix > > > commit B), at which point I'd cherrypick commits B through E at the > > > same time, in order, to avoid any possible merge conflicts. > > > > > > I think to make that process easier for myself I would just > > > cherry-pick everything into the 4.x branch as it gets merged into > > > master. Then, after 4-6 weeks of cherry-picking commits, if unstable > > > commits were not fixed/completed in time, I would reset the 4.x branch > > > to the last stable commit. This would mean that unstable > > > commits/incomplete features could potentially block things from going > > > into the release. I might count that as a positive because it would > > > incentivize us to improve our CI and do our best to also keep the > > > master branch as stable as possible. > > > > > > The end result would be the same as just continually jumping up the > > > master branch from known good, stable points in time to the next, > > > creating a minor release branch at each of those points. > > > > > > - Rawlin > > > > > > > > > > > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <[email protected]> wrote: > > > > > > > > Sorry, I have taken too long to respond. > > > > > > > > +1000000 on better CI and Testing, I think we all agree that needs > to get > > > > better and is a different beast than this topic. > > > > > > > > As for the release branches, I am trying to follow a process > similar to > > > > ATS, which is usually the model we try to follow with our project. > > > > While I agree that branching from master is the ideal solution, we > tried > > > > that and it didn't work. I feel like we are trying to agree to do > the > > > same > > > > thing that we already tried and didn't work. I think we should > have a > > > > little freedom to commit things into master and, if they break > things, > > > work > > > > on getting them fixed. We also might commit PRs to master that are > > > partial > > > > implementations of a bigger feature. If we are using a time based > > > > strategy to cut our releases from master then we are going to be > > > > continually worrying about making sure Master stays not broken. I > think > > > we > > > > should try to keep master deployable, but it's fine if it's not for > short > > > > periods of time. > > > > > > > > I am not trying to suggest that we are "picky" about what goes into > our > > > > release branch. I think we should try to cherry-pick as much as we > can > > > > into our release branch once it is ready to be cherry-picked. I > think we > > > > should be able to take the tip of our release branch and deploy it > > > whenever > > > > we want with some confidence since we know that features/bug-fixes > that > > > > have been cherry-picked have already been reviewed and tested > against > > > > master. Yes, its more work, but it also give us the ability to > have more > > > > confidence in what is in our release at any given time. Again, we > should > > > > be trying to cherry-pick in everything that makes sense to > cherry-pick. > > > > > > > > I think the process that I would like to see looks like this: > > > > - Cut a 4.x branch from Master > > > > - Once we think we are ready for a RC cut a 4.0.x branch from the > 4.x > > > branch > > > > - Tag the 4.0.0 release from the 4.0.x branch > > > > - New non-breaking changes and bug fixes that go into master are > also > > > > Cherry Picked to 4.x > > > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag > when we > > > > are ready for a RC. > > > > - Rinse and repeat for all 4.x releases > > > > - Once there's a need for a major release we make a 5.x branch from > > > master > > > > > > > > If we find a bug in 4.0.0 then we commit the fix to master, then > 4.x, > > > then > > > > 4.0.x which we use to create the new 4.0.1 release (assuming we > don't > > > have > > > > 4.1 yet). Yes, that is a more work than today, but I think A) it's > not > > > > really that bad and B) it gives us more control. > > > > > > > > This should allow us to deploy much quicker and with more > confidence than > > > > is possible today. We can also choose to deploy from any point in > time > > > > from the 4.x branch with relatively high confidence. > > > > > > > > I'll wrap this up by saying that I don't think there's a perfect > > > solution, > > > > but I think we can agree on something that is worth trying at least > with > > > > the 4.x release life cycle. Once we have some experience, we can > figure > > > > out where it is not meeting our needs and figure out how to adjust > as > > > > necessary. I think we can all agree our current process is not > working. > > > > > > > > Thanks, > > > > Dave > > > > > > > > > > > > > > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <[email protected]> > > > wrote: > > > > > > > > > +1 on more frequent releases > > > > > +1 on improving our CI. We need to get our CI to the point where > we're > > > > > confident deploying changes to production if the CI passes. > > > > > +1 on "trunk-based" development as opposed to GitFlow > > > > > -1 on cherry-picking new features. This is risky, tedious, and > > > > > represents unnecessary effort that would be better spent > elsewhere, > > > > > like improving our CI or adding more automated tests. > > > > > +1 on cutting minor release branches directly off master > > > > > > > > > > I think cutting minor release branches directly off master can > put us > > > > > in a win-win scenario. Our releases become smaller and more > frequent, > > > > > and we avoid the unnecessary risk and effort of cherry-picking new > > > > > features onto stale release branches. > > > > > > > > > > This is the process that I would propose: > > > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC > > > > > 2. Develop master for 4-6 weeks. > > > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC. > > > > > 4. Repeat steps 2 and 3 for new minor releases as necessary. > > > > > 5. Whenever a major (show-stopping) bug or security issue is > found, > > > > > cherry-pick the fix from master into the latest release branch > (e.g. > > > > > 4.1.x), and create a 4.1.1 RC. > > > > > > > > > > As the 4.x release manager, I really don't want to get into the > > > > > business of selecting what should and should not go into a > particular > > > > > release. Every 4-6 weeks, we should just cut a new minor release > > > > > branch off master and take everything that was contributed during > that > > > > > time, as long as everything builds fine and all the tests pass. > The > > > > > master branch, while it doesn't always have to be perfect, should > at > > > > > least always be deployable. If a new feature is merged with bugs > and > > > > > becomes a show-stopper, it should be either immediately remedied > or > > > > > reverted on master while we work on a fix. > > > > > > > > > > - Rawlin > > > > > > > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael > > > > > <[email protected]> wrote: > > > > > > > > > > > > In my opinion, in order to get to a cadence we are discussing we > > > need to > > > > > put a lot more work into the CI system. It has been failing > > > consistently, > > > > > doesn’t block PR approvals/merges and when it actually runs it > does not > > > > > test anything besides build and license headers. > > > > > > > > > > > > In the past couple of weeks we have had (off the top of my > head): > > > > > > > > > > > > * Unit tests broken > > > > > > * TO API tests broken > > > > > > * Golang vendor issues > > > > > > * TO Go build issues > > > > > > > > > > > > All were merged and broke master. > > > > > > > > > > > > Yes, having release branches sounds great and more aggressive > cadence > > > > > will minimize the amount of risk but I think that comes with the > need > > > to > > > > > improve on our automated validation and testing. > > > > > > > > > > > > We have proven that we have not kept master in a good state and > > > adding > > > > > more releases (more branches) will make it even harder to keep > that > > > > > stability we are already fighting. > > > > > > > > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> > > > wrote: > > > > > > > > > > > > Yeah, I get it. No one company should be driving release > > > > > schedules/scope. > > > > > > > > > > > > What I was really getting at is if ANY company has a recent > > > version > > > > > in a > > > > > > test or prod-like enviro (meaning the version is being > exercised > > > and > > > > > > thoroughly tested), then maybe we consider simply cutting a > > > release > > > > > from > > > > > > that version. For example, imagine company X had this > version in > > > a > > > > > > test/prod enviro: > > > > > > > > > > > > Master-10287.7e62d07 ( > > > > > > > > > > > > > > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > > > > > ) > > > > > > > > > > > > Then I would be in favor of cutting the 4.0 release from > > > > > > Master-10287.7e62d07 as we know it is proven to work based > on > > > > > company X's > > > > > > actual use of it. > > > > > > > > > > > > The truth is (imo), we don't have the bandwidth to manually > > > verify > > > > > releases > > > > > > that we are not using/have not used nor do we have the > necessary > > > > > automated > > > > > > test coverage to verify these releases. So most of our > releases > > > are > > > > > > significantly untested. > > > > > > > > > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 > <[email protected]> > > > > > wrote: > > > > > > > > > > > > > I'm really not a fan of allowing Comcast to dictate the > release > > > > > scope and > > > > > > > schedule. If cherry-picking is too messy, then we can > just cut > > > new > > > > > minor > > > > > > > releases directly from master. > > > > > > > > > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell < > > > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > I don't think it's as easy as cherry picking > (backporting) > > > > > certain > > > > > > > features > > > > > > > > into a release branch. I could be wrong but I really > don't > > > think > > > > > it is. > > > > > > > > > > > > > > > > So what I'm hearing is that 4.0 gets cut from master > and we > > > go > > > > > through > > > > > > > with > > > > > > > > our normal process of testing, validating, etc. At the > same > > > > > time, 4.1 > > > > > > > > branch is created from 4.0. Master moves on as normal. > Let's > > > say > > > > > 25 > > > > > > > commits > > > > > > > > come in to master during the next 6 weeks. During that 6 > > > weeks, > > > > > we cherry > > > > > > > > pick only the features that look interesting for 4.1? > (the > > > rest > > > > > just stay > > > > > > > > in master and will get picked up in 5.0?) > > > > > > > > > > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are > roughly 6 > > > > > weeks apart > > > > > > > and > > > > > > > > have only "interesting" features so the releases are > > > > > incrementally very > > > > > > > > small and much easier to test/validate, however, those > > > features > > > > > might > > > > > > > > depend on other commits. At some point, cherry picking > > > becomes > > > > > impossible > > > > > > > > as the foundation of the code base has shifted so > > > drastically. > > > > > Cherry > > > > > > > > picking will become very messy and introduce a high > level of > > > > > risk imo. > > > > > > > > > > > > > > > > Also, you'd have to manage the scope of these minor > releases > > > > > which we are > > > > > > > > not very good at because the designated release manager > has a > > > > > full-time > > > > > > > job > > > > > > > > to attend to and now we have even more releases to > > > > > > > > scope/test/validate/approve. I like the idea of faster > > > releases > > > > > but this > > > > > > > > sounds like more work that nobody has the bandwidth for. > > > > > > > > > > > > > > > > Ok, so rather than shooting down an idea with no > alternative > > > > > proposal, I > > > > > > > > propose this. At Comcast, we are trying to get to the > point > > > > > where we > > > > > > > > release TC to prod from the head of master every few > weeks. > > > What > > > > > if every > > > > > > > > time Comcast does an internal upgrade, they note the > commit > > > hash > > > > > and once > > > > > > > > comcast is happy with the upgrade in their prod enviro, > they > > > > > propose a > > > > > > > > release pinned to that commit hash? With this approach > you'd > > > end > > > > > up with: > > > > > > > > > > > > > > > > 1. fast releases (hopefully every few weeks) > > > > > > > > 2. well tested releases as it's in a real prod enviro > > > > > > > > 3. little/no validation work for anyone > > > > > > > > > > > > > > > > Also, this doesn't have to be Comcast. If anyone else > is out > > > > > front at the > > > > > > > > edge of master and running it in a trusted enviro, they > could > > > > > propose a > > > > > > > > release as well. > > > > > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I have been thinking about how we can get better with > our > > > > > release > > > > > > > > cadence. > > > > > > > > > I feel like we have slowed to a crawl and not been as > good > > > as > > > > > we should > > > > > > > > > about how we release. Our last Major release was in > March > > > and > > > > > we > > > > > > > haven't > > > > > > > > > had a real release since. Moving forward I would > like to > > > see > > > > > us get > > > > > > > on > > > > > > > > a > > > > > > > > > more consistent cadence and provide smaller releases > that > > > can > > > > > be more > > > > > > > > > easily digested by the community. It's my hope that > we can > > > > > get to a > > > > > > > > > cadence where we are doing releases every 4 to 6 > weeks, > > > we are > > > > > > > releasing > > > > > > > > > in such a way that not all releases all required > (e.g. if > > > you > > > > > are on > > > > > > > 4.0 > > > > > > > > > you don't need 4.1, you can just install 4.2), and we > are > > > more > > > > > > > deliberate > > > > > > > > > about what we put into a release. > > > > > > > > > > > > > > > > > > I think we can accomplish this by using our release > > > branches > > > > > better. > > > > > > > For > > > > > > > > > each major release we will cut a release branch from > master > > > > > (e.g. 4.x) > > > > > > > > and > > > > > > > > > then we will use cherry-picking to add new features > and bug > > > > > fixes to > > > > > > > that > > > > > > > > > release. This means that if 4.0 has been released > and you > > > > > want to get > > > > > > > > your > > > > > > > > > feature/bug fix in 4.1, you will first submit your PR > to > > > > > master and > > > > > > > then > > > > > > > > > cheery pick your squash merged commit to the 4.x > branch > > > which > > > > > we will > > > > > > > use > > > > > > > > > to create the 4.1 release. I think we should allow > either > > > the > > > > > > > > contributor > > > > > > > > > or the committer (who is merging the PR) to suggest > if a > > > > > feature goes > > > > > > > > into > > > > > > > > > the release branch, and if we disagree we can take it > to > > > the > > > > > list. If > > > > > > > we > > > > > > > > > decide that every PR to master also goes into 4.1 > then so > > > be > > > > > it, but at > > > > > > > > > least we are making a conscious decision of what goes > into > > > the > > > > > next > > > > > > > > > release. This will allow us to not be so worried > about > > > what > > > > > we are > > > > > > > > > merging into master and how that will affect our next > > > release. > > > > > > > According > > > > > > > > > to our 6 week cadence we will cut a new release from > the > > > > > current > > > > > > > feature > > > > > > > > > branch and put that up for a vote. These releases > will be > > > > > small and > > > > > > > > > testable enough that we can get feedback quickly and > move > > > from > > > > > RC to > > > > > > > > > release in a short time. If a release has too many > issues > > > > > then we may > > > > > > > > end > > > > > > > > > up deciding to skip that minor release in favor of > the next > > > > > one, > > > > > > > > hopefully > > > > > > > > > this does not happen very often if at all. > > > > > > > > > > > > > > > > > > Once a breaking change is introduced to master which > will > > > > > require a new > > > > > > > > > major release, we will create a new major release > branch > > > from > > > > > master > > > > > > > > (which > > > > > > > > > will mean a new release manager). We will then > repeat the > > > > > same process > > > > > > > > > with the new release branch, etc, etc. > > > > > > > > > > > > > > > > > > As for LTS we will provide support for the latest > major > > > > > release plus > > > > > > > the > > > > > > > > > one previous. So, once we release 4.0 we will > support 4.0 > > > and > > > > > 3.1. > > > > > > > Any > > > > > > > > > security issues that arise will -- if present -- be > > > applied to > > > > > each of > > > > > > > > > these versions. Once 4.1 is released we will support > 4.1 > > > and > > > > > 3.1, etc, > > > > > > > > > etc. > > > > > > > > > > > > > > > > > > Please let me know if you have any thoughts on this > plan. > > > If > > > > > there are > > > > > > > > no > > > > > > > > > major objections I would like to try this with 4.x > with the > > > > > idea that > > > > > > > we > > > > > > > > > will adjust how we do things as necessary. > > > > > > > > > > > > > > > > > > We need to get better at releasing and something > needs to > > > > > change, > > > > > > > > hopefully > > > > > > > > > this can get us going in the right direction. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dave > > > > > > > > > > > > > > > > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > > > > > cherry-picks to > > > > > > > our > > > > > > > > > release branch to control what goes into a minor > release. > > > > > Major > > > > > > > releases > > > > > > > > > will be cut from master as necessary - such as if > there is > > > a > > > > > breaking > > > > > > > > > change that is introduced. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
