still -1 on on cherry-picked releases if this is what i'm hearing: 4.0 (derived from master) 4.1 (derived from 4.0 plus some cherry picks from master introduced during the last 6 weeks) 4.2 (derived from 4.1 plus some cherry picks from master introduced during the last 6 weeks) 4.3 (derived from 4.2 plus some cherry picks from master introduced during the last 6 weeks)
^^ 4.3 is a triple cherry-picked release. i would not trust that at all. in theory it might sound great but in practice i think it will cause problems like rawlin said (risky, tedious, and represents unnecessary effort that would be better spent elsewhere) jeremy On Wed, Oct 30, 2019 at 3:04 PM Gray, Jonathan <[email protected]> wrote: > Did we define why/how what we've been doing isnt' working? I see a > consensus that it's not, suggestions on alternatives, and an ambition for > faster and more stable; but not a definition of the thing is the reason why > we must change? Not saying we shouldn't, but I'm just asking so we try > not to repeat the same mistakes following new steps. Usually I've found > that release blockage is derived from not defining clear objectives and > definitions of good enough (not necessarily perfect) consistently > throughout a release cycle. It's not enough to start a release cycle with > one scope of both, yet at the end continue to revise that definition. > > Jonathan G > > > On 10/30/19, 2:36 PM, "Dave Neuman" <[email protected]> wrote: > > Sorry, I have taken too long to respond. > > +1000000 on better CI and Testing, I think we all agree that needs to > get > better and is a different beast than this topic. > > As for the release branches, I am trying to follow a process similar to > ATS, which is usually the model we try to follow with our project. > While I agree that branching from master is the ideal solution, we > tried > that and it didn't work. I feel like we are trying to agree to do the > same > thing that we already tried and didn't work. I think we should have a > little freedom to commit things into master and, if they break things, > work > on getting them fixed. We also might commit PRs to master that are > partial > implementations of a bigger feature. If we are using a time based > strategy to cut our releases from master then we are going to be > continually worrying about making sure Master stays not broken. I > think we > should try to keep master deployable, but it's fine if it's not for > short > periods of time. > > I am not trying to suggest that we are "picky" about what goes into our > release branch. I think we should try to cherry-pick as much as we can > into our release branch once it is ready to be cherry-picked. I think > we > should be able to take the tip of our release branch and deploy it > whenever > we want with some confidence since we know that features/bug-fixes that > have been cherry-picked have already been reviewed and tested against > master. Yes, its more work, but it also give us the ability to have > more > confidence in what is in our release at any given time. Again, we > should > be trying to cherry-pick in everything that makes sense to cherry-pick. > > I think the process that I would like to see looks like this: > - Cut a 4.x branch from Master > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x > branch > - Tag the 4.0.0 release from the 4.0.x branch > - New non-breaking changes and bug fixes that go into master are also > Cherry Picked to 4.x > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when > we > are ready for a RC. > - Rinse and repeat for all 4.x releases > - Once there's a need for a major release we make a 5.x branch from > master > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x, > then > 4.0.x which we use to create the new 4.0.1 release (assuming we don't > have > 4.1 yet). Yes, that is a more work than today, but I think A) it's not > really that bad and B) it gives us more control. > > This should allow us to deploy much quicker and with more confidence > than > is possible today. We can also choose to deploy from any point in > time > from the 4.x branch with relatively high confidence. > > I'll wrap this up by saying that I don't think there's a perfect > solution, > but I think we can agree on something that is worth trying at least > with > the 4.x release life cycle. Once we have some experience, we can > figure > out where it is not meeting our needs and figure out how to adjust as > necessary. I think we can all agree our current process is not > working. > > Thanks, > Dave > > > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <[email protected]> > wrote: > > > +1 on more frequent releases > > +1 on improving our CI. We need to get our CI to the point where > we're > > confident deploying changes to production if the CI passes. > > +1 on "trunk-based" development as opposed to GitFlow > > -1 on cherry-picking new features. This is risky, tedious, and > > represents unnecessary effort that would be better spent elsewhere, > > like improving our CI or adding more automated tests. > > +1 on cutting minor release branches directly off master > > > > I think cutting minor release branches directly off master can put us > > in a win-win scenario. Our releases become smaller and more frequent, > > and we avoid the unnecessary risk and effort of cherry-picking new > > features onto stale release branches. > > > > This is the process that I would propose: > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC > > 2. Develop master for 4-6 weeks. > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC. > > 4. Repeat steps 2 and 3 for new minor releases as necessary. > > 5. Whenever a major (show-stopping) bug or security issue is found, > > cherry-pick the fix from master into the latest release branch (e.g. > > 4.1.x), and create a 4.1.1 RC. > > > > As the 4.x release manager, I really don't want to get into the > > business of selecting what should and should not go into a particular > > release. Every 4-6 weeks, we should just cut a new minor release > > branch off master and take everything that was contributed during > that > > time, as long as everything builds fine and all the tests pass. The > > master branch, while it doesn't always have to be perfect, should at > > least always be deployable. If a new feature is merged with bugs and > > becomes a show-stopper, it should be either immediately remedied or > > reverted on master while we work on a fix. > > > > - Rawlin > > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael > > <[email protected]> wrote: > > > > > > In my opinion, in order to get to a cadence we are discussing we > need to > > put a lot more work into the CI system. It has been failing > consistently, > > doesn’t block PR approvals/merges and when it actually runs it does > not > > test anything besides build and license headers. > > > > > > In the past couple of weeks we have had (off the top of my head): > > > > > > * Unit tests broken > > > * TO API tests broken > > > * Golang vendor issues > > > * TO Go build issues > > > > > > All were merged and broke master. > > > > > > Yes, having release branches sounds great and more aggressive > cadence > > will minimize the amount of risk but I think that comes with the > need to > > improve on our automated validation and testing. > > > > > > We have proven that we have not kept master in a good state and > adding > > more releases (more branches) will make it even harder to keep that > > stability we are already fighting. > > > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> > wrote: > > > > > > Yeah, I get it. No one company should be driving release > > schedules/scope. > > > > > > What I was really getting at is if ANY company has a recent > version > > in a > > > test or prod-like enviro (meaning the version is being > exercised and > > > thoroughly tested), then maybe we consider simply cutting a > release > > from > > > that version. For example, imagine company X had this version > in a > > > test/prod enviro: > > > > > > Master-10287.7e62d07 ( > > > > > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > > ) > > > > > > Then I would be in favor of cutting the 4.0 release from > > > Master-10287.7e62d07 as we know it is proven to work based on > > company X's > > > actual use of it. > > > > > > The truth is (imo), we don't have the bandwidth to manually > verify > > releases > > > that we are not using/have not used nor do we have the > necessary > > automated > > > test coverage to verify these releases. So most of our > releases are > > > significantly untested. > > > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 < > [email protected]> > > wrote: > > > > > > > I'm really not a fan of allowing Comcast to dictate the > release > > scope and > > > > schedule. If cherry-picking is too messy, then we can just > cut new > > minor > > > > releases directly from master. > > > > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell < > [email protected]> > > wrote: > > > > > > > > > I don't think it's as easy as cherry picking (backporting) > > certain > > > > features > > > > > into a release branch. I could be wrong but I really don't > think > > it is. > > > > > > > > > > So what I'm hearing is that 4.0 gets cut from master and > we go > > through > > > > with > > > > > our normal process of testing, validating, etc. At the same > > time, 4.1 > > > > > branch is created from 4.0. Master moves on as normal. > Let's say > > 25 > > > > commits > > > > > come in to master during the next 6 weeks. During that 6 > weeks, > > we cherry > > > > > pick only the features that look interesting for 4.1? (the > rest > > just stay > > > > > in master and will get picked up in 5.0?) > > > > > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 > > weeks apart > > > > and > > > > > have only "interesting" features so the releases are > > incrementally very > > > > > small and much easier to test/validate, however, those > features > > might > > > > > depend on other commits. At some point, cherry picking > becomes > > impossible > > > > > as the foundation of the code base has shifted so > drastically. > > Cherry > > > > > picking will become very messy and introduce a high level > of > > risk imo. > > > > > > > > > > Also, you'd have to manage the scope of these minor > releases > > which we are > > > > > not very good at because the designated release manager > has a > > full-time > > > > job > > > > > to attend to and now we have even more releases to > > > > > scope/test/validate/approve. I like the idea of faster > releases > > but this > > > > > sounds like more work that nobody has the bandwidth for. > > > > > > > > > > Ok, so rather than shooting down an idea with no > alternative > > proposal, I > > > > > propose this. At Comcast, we are trying to get to the point > > where we > > > > > release TC to prod from the head of master every few > weeks. What > > if every > > > > > time Comcast does an internal upgrade, they note the > commit hash > > and once > > > > > comcast is happy with the upgrade in their prod enviro, > they > > propose a > > > > > release pinned to that commit hash? With this approach > you'd end > > up with: > > > > > > > > > > 1. fast releases (hopefully every few weeks) > > > > > 2. well tested releases as it's in a real prod enviro > > > > > 3. little/no validation work for anyone > > > > > > > > > > Also, this doesn't have to be Comcast. If anyone else is > out > > front at the > > > > > edge of master and running it in a trusted enviro, they > could > > propose a > > > > > release as well. > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > > [email protected]> > > > > > wrote: > > > > > > > > > > > I have been thinking about how we can get better with our > > release > > > > > cadence. > > > > > > I feel like we have slowed to a crawl and not been as > good as > > we should > > > > > > about how we release. Our last Major release was in > March and > > we > > > > haven't > > > > > > had a real release since. Moving forward I would like > to see > > us get > > > > on > > > > > a > > > > > > more consistent cadence and provide smaller releases > that can > > be more > > > > > > easily digested by the community. It's my hope that we > can > > get to a > > > > > > cadence where we are doing releases every 4 to 6 weeks, > we are > > > > releasing > > > > > > in such a way that not all releases all required (e.g. > if you > > are on > > > > 4.0 > > > > > > you don't need 4.1, you can just install 4.2), and we > are more > > > > deliberate > > > > > > about what we put into a release. > > > > > > > > > > > > I think we can accomplish this by using our release > branches > > better. > > > > For > > > > > > each major release we will cut a release branch from > master > > (e.g. 4.x) > > > > > and > > > > > > then we will use cherry-picking to add new features and > bug > > fixes to > > > > that > > > > > > release. This means that if 4.0 has been released and > you > > want to get > > > > > your > > > > > > feature/bug fix in 4.1, you will first submit your PR to > > master and > > > > then > > > > > > cheery pick your squash merged commit to the 4.x branch > which > > we will > > > > use > > > > > > to create the 4.1 release. I think we should allow > either the > > > > > contributor > > > > > > or the committer (who is merging the PR) to suggest if a > > feature goes > > > > > into > > > > > > the release branch, and if we disagree we can take it to > the > > list. If > > > > we > > > > > > decide that every PR to master also goes into 4.1 then > so be > > it, but at > > > > > > least we are making a conscious decision of what goes > into the > > next > > > > > > release. This will allow us to not be so worried about > what > > we are > > > > > > merging into master and how that will affect our next > release. > > > > According > > > > > > to our 6 week cadence we will cut a new release from the > > current > > > > feature > > > > > > branch and put that up for a vote. These releases will > be > > small and > > > > > > testable enough that we can get feedback quickly and > move from > > RC to > > > > > > release in a short time. If a release has too many > issues > > then we may > > > > > end > > > > > > up deciding to skip that minor release in favor of the > next > > one, > > > > > hopefully > > > > > > this does not happen very often if at all. > > > > > > > > > > > > Once a breaking change is introduced to master which will > > require a new > > > > > > major release, we will create a new major release branch > from > > master > > > > > (which > > > > > > will mean a new release manager). We will then repeat > the > > same process > > > > > > with the new release branch, etc, etc. > > > > > > > > > > > > As for LTS we will provide support for the latest major > > release plus > > > > the > > > > > > one previous. So, once we release 4.0 we will support > 4.0 and > > 3.1. > > > > Any > > > > > > security issues that arise will -- if present -- be > applied to > > each of > > > > > > these versions. Once 4.1 is released we will support > 4.1 and > > 3.1, etc, > > > > > > etc. > > > > > > > > > > > > Please let me know if you have any thoughts on this > plan. If > > there are > > > > > no > > > > > > major objections I would like to try this with 4.x with > the > > idea that > > > > we > > > > > > will adjust how we do things as necessary. > > > > > > > > > > > > We need to get better at releasing and something needs to > > change, > > > > > hopefully > > > > > > this can get us going in the right direction. > > > > > > > > > > > > Thanks, > > > > > > Dave > > > > > > > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > > cherry-picks to > > > > our > > > > > > release branch to control what goes into a minor release. > > Major > > > > releases > > > > > > will be cut from master as necessary - such as if there > is a > > breaking > > > > > > change that is introduced. > > > > > > > > > > > > > > > > > > > > > > > > > >
