I was trying to avoid using that term since it is different enough to not be exactly the same, though it is very similar.
On Wed, Oct 30, 2019 at 7:34 PM Gray, Jonathan <[email protected]> wrote: > It sounds like you're wanting to convert from Github Flow to regular > Gitflow, but change what branches do what. > https://lucamezzalira.com/2014/03/10/git-flow-vs-github-flow/ > > Jonathan G > > On 10/30/19, 5:35 PM, "Rawlin Peters" <[email protected]> wrote: > > I guess my concern is really about what possible reasons we might have > to not cherry-pick a specific PR that had been merged into master > already. The only reasons I can think of to not cherry-pick something > would be: > 1. the PR was completely broken (e.g. breaks the build, tests don't > pass, doesn't compile, etc). > 2. the PR one one of a larger multi-PR "feature", and *we know* > releasing some parts early could cause problems (that knowing part is > crucial) > > We should always strive to catch issue #1 in our CI before it merges > into master. When things like that merge it can block people from > doing development starting at that point until it's fixed. > I can see why issue #2 might arise due to multiple different people > developing one larger feature, but we should still strive to break > larger features up into individual pieces that can reasonably stand on > their own if possible. Due to time constraints or other reasons I can > see that might be a little difficult. > > That said, I would still strive to make the 4.x release branch a > snapshot of master at some "known good point" in time. Let's say we > cut 4.x at commit A, then a broken commit B gets merged into master. > If it's completely obvious that it's a broken commit, I'd wait until > at least a commit C gets merged to fix it. Then I would cherry-pick B > and C into 4.x at the same time. If the fix is really like a commit E > with unrelated commits C and D in between, ideally I would still wait > to cherry-pick C and D until a "fix" commit E was merged (to fix > commit B), at which point I'd cherrypick commits B through E at the > same time, in order, to avoid any possible merge conflicts. > > I think to make that process easier for myself I would just > cherry-pick everything into the 4.x branch as it gets merged into > master. Then, after 4-6 weeks of cherry-picking commits, if unstable > commits were not fixed/completed in time, I would reset the 4.x branch > to the last stable commit. This would mean that unstable > commits/incomplete features could potentially block things from going > into the release. I might count that as a positive because it would > incentivize us to improve our CI and do our best to also keep the > master branch as stable as possible. > > The end result would be the same as just continually jumping up the > master branch from known good, stable points in time to the next, > creating a minor release branch at each of those points. > > - Rawlin > > > > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <[email protected]> wrote: > > > > Sorry, I have taken too long to respond. > > > > +1000000 on better CI and Testing, I think we all agree that needs > to get > > better and is a different beast than this topic. > > > > As for the release branches, I am trying to follow a process similar > to > > ATS, which is usually the model we try to follow with our project. > > While I agree that branching from master is the ideal solution, we > tried > > that and it didn't work. I feel like we are trying to agree to do > the same > > thing that we already tried and didn't work. I think we should have > a > > little freedom to commit things into master and, if they break > things, work > > on getting them fixed. We also might commit PRs to master that are > partial > > implementations of a bigger feature. If we are using a time based > > strategy to cut our releases from master then we are going to be > > continually worrying about making sure Master stays not broken. I > think we > > should try to keep master deployable, but it's fine if it's not for > short > > periods of time. > > > > I am not trying to suggest that we are "picky" about what goes into > our > > release branch. I think we should try to cherry-pick as much as we > can > > into our release branch once it is ready to be cherry-picked. I > think we > > should be able to take the tip of our release branch and deploy it > whenever > > we want with some confidence since we know that features/bug-fixes > that > > have been cherry-picked have already been reviewed and tested against > > master. Yes, its more work, but it also give us the ability to have > more > > confidence in what is in our release at any given time. Again, we > should > > be trying to cherry-pick in everything that makes sense to > cherry-pick. > > > > I think the process that I would like to see looks like this: > > - Cut a 4.x branch from Master > > - Once we think we are ready for a RC cut a 4.0.x branch from the > 4.x branch > > - Tag the 4.0.0 release from the 4.0.x branch > > - New non-breaking changes and bug fixes that go into master are also > > Cherry Picked to 4.x > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag > when we > > are ready for a RC. > > - Rinse and repeat for all 4.x releases > > - Once there's a need for a major release we make a 5.x branch from > master > > > > If we find a bug in 4.0.0 then we commit the fix to master, then > 4.x, then > > 4.0.x which we use to create the new 4.0.1 release (assuming we > don't have > > 4.1 yet). Yes, that is a more work than today, but I think A) it's > not > > really that bad and B) it gives us more control. > > > > This should allow us to deploy much quicker and with more confidence > than > > is possible today. We can also choose to deploy from any point in > time > > from the 4.x branch with relatively high confidence. > > > > I'll wrap this up by saying that I don't think there's a perfect > solution, > > but I think we can agree on something that is worth trying at least > with > > the 4.x release life cycle. Once we have some experience, we can > figure > > out where it is not meeting our needs and figure out how to adjust as > > necessary. I think we can all agree our current process is not > working. > > > > Thanks, > > Dave > > > > > > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <[email protected]> > wrote: > > > > > +1 on more frequent releases > > > +1 on improving our CI. We need to get our CI to the point where > we're > > > confident deploying changes to production if the CI passes. > > > +1 on "trunk-based" development as opposed to GitFlow > > > -1 on cherry-picking new features. This is risky, tedious, and > > > represents unnecessary effort that would be better spent elsewhere, > > > like improving our CI or adding more automated tests. > > > +1 on cutting minor release branches directly off master > > > > > > I think cutting minor release branches directly off master can put > us > > > in a win-win scenario. Our releases become smaller and more > frequent, > > > and we avoid the unnecessary risk and effort of cherry-picking new > > > features onto stale release branches. > > > > > > This is the process that I would propose: > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC > > > 2. Develop master for 4-6 weeks. > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC. > > > 4. Repeat steps 2 and 3 for new minor releases as necessary. > > > 5. Whenever a major (show-stopping) bug or security issue is found, > > > cherry-pick the fix from master into the latest release branch > (e.g. > > > 4.1.x), and create a 4.1.1 RC. > > > > > > As the 4.x release manager, I really don't want to get into the > > > business of selecting what should and should not go into a > particular > > > release. Every 4-6 weeks, we should just cut a new minor release > > > branch off master and take everything that was contributed during > that > > > time, as long as everything builds fine and all the tests pass. The > > > master branch, while it doesn't always have to be perfect, should > at > > > least always be deployable. If a new feature is merged with bugs > and > > > becomes a show-stopper, it should be either immediately remedied or > > > reverted on master while we work on a fix. > > > > > > - Rawlin > > > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael > > > <[email protected]> wrote: > > > > > > > > In my opinion, in order to get to a cadence we are discussing we > need to > > > put a lot more work into the CI system. It has been failing > consistently, > > > doesn’t block PR approvals/merges and when it actually runs it > does not > > > test anything besides build and license headers. > > > > > > > > In the past couple of weeks we have had (off the top of my head): > > > > > > > > * Unit tests broken > > > > * TO API tests broken > > > > * Golang vendor issues > > > > * TO Go build issues > > > > > > > > All were merged and broke master. > > > > > > > > Yes, having release branches sounds great and more aggressive > cadence > > > will minimize the amount of risk but I think that comes with the > need to > > > improve on our automated validation and testing. > > > > > > > > We have proven that we have not kept master in a good state and > adding > > > more releases (more branches) will make it even harder to keep that > > > stability we are already fighting. > > > > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> > wrote: > > > > > > > > Yeah, I get it. No one company should be driving release > > > schedules/scope. > > > > > > > > What I was really getting at is if ANY company has a recent > version > > > in a > > > > test or prod-like enviro (meaning the version is being > exercised and > > > > thoroughly tested), then maybe we consider simply cutting a > release > > > from > > > > that version. For example, imagine company X had this > version in a > > > > test/prod enviro: > > > > > > > > Master-10287.7e62d07 ( > > > > > > > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > > > ) > > > > > > > > Then I would be in favor of cutting the 4.0 release from > > > > Master-10287.7e62d07 as we know it is proven to work based on > > > company X's > > > > actual use of it. > > > > > > > > The truth is (imo), we don't have the bandwidth to manually > verify > > > releases > > > > that we are not using/have not used nor do we have the > necessary > > > automated > > > > test coverage to verify these releases. So most of our > releases are > > > > significantly untested. > > > > > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 < > [email protected]> > > > wrote: > > > > > > > > > I'm really not a fan of allowing Comcast to dictate the > release > > > scope and > > > > > schedule. If cherry-picking is too messy, then we can just > cut new > > > minor > > > > > releases directly from master. > > > > > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell < > [email protected]> > > > wrote: > > > > > > > > > > > I don't think it's as easy as cherry picking > (backporting) > > > certain > > > > > features > > > > > > into a release branch. I could be wrong but I really > don't think > > > it is. > > > > > > > > > > > > So what I'm hearing is that 4.0 gets cut from master and > we go > > > through > > > > > with > > > > > > our normal process of testing, validating, etc. At the > same > > > time, 4.1 > > > > > > branch is created from 4.0. Master moves on as normal. > Let's say > > > 25 > > > > > commits > > > > > > come in to master during the next 6 weeks. During that 6 > weeks, > > > we cherry > > > > > > pick only the features that look interesting for 4.1? > (the rest > > > just stay > > > > > > in master and will get picked up in 5.0?) > > > > > > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are > roughly 6 > > > weeks apart > > > > > and > > > > > > have only "interesting" features so the releases are > > > incrementally very > > > > > > small and much easier to test/validate, however, those > features > > > might > > > > > > depend on other commits. At some point, cherry picking > becomes > > > impossible > > > > > > as the foundation of the code base has shifted so > drastically. > > > Cherry > > > > > > picking will become very messy and introduce a high > level of > > > risk imo. > > > > > > > > > > > > Also, you'd have to manage the scope of these minor > releases > > > which we are > > > > > > not very good at because the designated release manager > has a > > > full-time > > > > > job > > > > > > to attend to and now we have even more releases to > > > > > > scope/test/validate/approve. I like the idea of faster > releases > > > but this > > > > > > sounds like more work that nobody has the bandwidth for. > > > > > > > > > > > > Ok, so rather than shooting down an idea with no > alternative > > > proposal, I > > > > > > propose this. At Comcast, we are trying to get to the > point > > > where we > > > > > > release TC to prod from the head of master every few > weeks. What > > > if every > > > > > > time Comcast does an internal upgrade, they note the > commit hash > > > and once > > > > > > comcast is happy with the upgrade in their prod enviro, > they > > > propose a > > > > > > release pinned to that commit hash? With this approach > you'd end > > > up with: > > > > > > > > > > > > 1. fast releases (hopefully every few weeks) > > > > > > 2. well tested releases as it's in a real prod enviro > > > > > > 3. little/no validation work for anyone > > > > > > > > > > > > Also, this doesn't have to be Comcast. If anyone else is > out > > > front at the > > > > > > edge of master and running it in a trusted enviro, they > could > > > propose a > > > > > > release as well. > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > I have been thinking about how we can get better with > our > > > release > > > > > > cadence. > > > > > > > I feel like we have slowed to a crawl and not been as > good as > > > we should > > > > > > > about how we release. Our last Major release was in > March and > > > we > > > > > haven't > > > > > > > had a real release since. Moving forward I would > like to see > > > us get > > > > > on > > > > > > a > > > > > > > more consistent cadence and provide smaller releases > that can > > > be more > > > > > > > easily digested by the community. It's my hope that > we can > > > get to a > > > > > > > cadence where we are doing releases every 4 to 6 > weeks, we are > > > > > releasing > > > > > > > in such a way that not all releases all required (e.g. > if you > > > are on > > > > > 4.0 > > > > > > > you don't need 4.1, you can just install 4.2), and we > are more > > > > > deliberate > > > > > > > about what we put into a release. > > > > > > > > > > > > > > I think we can accomplish this by using our release > branches > > > better. > > > > > For > > > > > > > each major release we will cut a release branch from > master > > > (e.g. 4.x) > > > > > > and > > > > > > > then we will use cherry-picking to add new features > and bug > > > fixes to > > > > > that > > > > > > > release. This means that if 4.0 has been released and > you > > > want to get > > > > > > your > > > > > > > feature/bug fix in 4.1, you will first submit your PR > to > > > master and > > > > > then > > > > > > > cheery pick your squash merged commit to the 4.x > branch which > > > we will > > > > > use > > > > > > > to create the 4.1 release. I think we should allow > either the > > > > > > contributor > > > > > > > or the committer (who is merging the PR) to suggest if > a > > > feature goes > > > > > > into > > > > > > > the release branch, and if we disagree we can take it > to the > > > list. If > > > > > we > > > > > > > decide that every PR to master also goes into 4.1 then > so be > > > it, but at > > > > > > > least we are making a conscious decision of what goes > into the > > > next > > > > > > > release. This will allow us to not be so worried > about what > > > we are > > > > > > > merging into master and how that will affect our next > release. > > > > > According > > > > > > > to our 6 week cadence we will cut a new release from > the > > > current > > > > > feature > > > > > > > branch and put that up for a vote. These releases > will be > > > small and > > > > > > > testable enough that we can get feedback quickly and > move from > > > RC to > > > > > > > release in a short time. If a release has too many > issues > > > then we may > > > > > > end > > > > > > > up deciding to skip that minor release in favor of the > next > > > one, > > > > > > hopefully > > > > > > > this does not happen very often if at all. > > > > > > > > > > > > > > Once a breaking change is introduced to master which > will > > > require a new > > > > > > > major release, we will create a new major release > branch from > > > master > > > > > > (which > > > > > > > will mean a new release manager). We will then repeat > the > > > same process > > > > > > > with the new release branch, etc, etc. > > > > > > > > > > > > > > As for LTS we will provide support for the latest major > > > release plus > > > > > the > > > > > > > one previous. So, once we release 4.0 we will support > 4.0 and > > > 3.1. > > > > > Any > > > > > > > security issues that arise will -- if present -- be > applied to > > > each of > > > > > > > these versions. Once 4.1 is released we will support > 4.1 and > > > 3.1, etc, > > > > > > > etc. > > > > > > > > > > > > > > Please let me know if you have any thoughts on this > plan. If > > > there are > > > > > > no > > > > > > > major objections I would like to try this with 4.x > with the > > > idea that > > > > > we > > > > > > > will adjust how we do things as necessary. > > > > > > > > > > > > > > We need to get better at releasing and something needs > to > > > change, > > > > > > hopefully > > > > > > > this can get us going in the right direction. > > > > > > > > > > > > > > Thanks, > > > > > > > Dave > > > > > > > > > > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > > > cherry-picks to > > > > > our > > > > > > > release branch to control what goes into a minor > release. > > > Major > > > > > releases > > > > > > > will be cut from master as necessary - such as if > there is a > > > breaking > > > > > > > change that is introduced. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
