> What I was really getting at is if ANY company has a recent version in a > test or prod-like enviro (meaning the version is being exercised and > thoroughly tested), then maybe we consider simply cutting a release from > that version
I think the idea s that companies should stop doing their own versioning - which was totally necessary because it's unreasonable to wait half a year for an Apache release - and start deploying official releases. That way the releases ARE thoroughly exercised and tested, just that the commit hashes aren't dictated by the companies. Companies could refuse, of course, but that only makes things harder for them IMO. At Comcast it's taken notoriously long to validate releases, and making them smaller will help that. > In my opinion, in order to get to a cadence we are discussing we need to put a lot more work into the CI system. I agree, it's all but worthless at the moment and testing is vital to providing stable releases. I know GitHub Actions have been discussed before, but they are coming out of beta in a couple of weeks and if the problem with CI is capacity then maybe we could disable the Jenkins jobs and set up Actions in their stead. On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael <michael_hop...@comcast.com> wrote: > In my opinion, in order to get to a cadence we are discussing we need to > put a lot more work into the CI system. It has been failing consistently, > doesn’t block PR approvals/merges and when it actually runs it does not > test anything besides build and license headers. > > In the past couple of weeks we have had (off the top of my head): > > * Unit tests broken > * TO API tests broken > * Golang vendor issues > * TO Go build issues > > All were merged and broke master. > > Yes, having release branches sounds great and more aggressive cadence will > minimize the amount of risk but I think that comes with the need to improve > on our automated validation and testing. > > We have proven that we have not kept master in a good state and adding > more releases (more branches) will make it even harder to keep that > stability we are already fighting. > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mitchell...@gmail.com> wrote: > > Yeah, I get it. No one company should be driving release > schedules/scope. > > What I was really getting at is if ANY company has a recent version in > a > test or prod-like enviro (meaning the version is being exercised and > thoroughly tested), then maybe we consider simply cutting a release > from > that version. For example, imagine company X had this version in a > test/prod enviro: > > Master-10287.7e62d07 ( > > https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master > ) > > Then I would be in favor of cutting the 4.0 release from > Master-10287.7e62d07 as we know it is proven to work based on company > X's > actual use of it. > > The truth is (imo), we don't have the bandwidth to manually verify > releases > that we are not using/have not used nor do we have the necessary > automated > test coverage to verify these releases. So most of our releases are > significantly untested. > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <ocket8...@gmail.com> > wrote: > > > I'm really not a fan of allowing Comcast to dictate the release > scope and > > schedule. If cherry-picking is too messy, then we can just cut new > minor > > releases directly from master. > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <mitchell...@gmail.com> > wrote: > > > > > I don't think it's as easy as cherry picking (backporting) certain > > features > > > into a release branch. I could be wrong but I really don't think > it is. > > > > > > So what I'm hearing is that 4.0 gets cut from master and we go > through > > with > > > our normal process of testing, validating, etc. At the same time, > 4.1 > > > branch is created from 4.0. Master moves on as normal. Let's say 25 > > commits > > > come in to master during the next 6 weeks. During that 6 weeks, we > cherry > > > pick only the features that look interesting for 4.1? (the rest > just stay > > > in master and will get picked up in 5.0?) > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks > apart > > and > > > have only "interesting" features so the releases are incrementally > very > > > small and much easier to test/validate, however, those features > might > > > depend on other commits. At some point, cherry picking becomes > impossible > > > as the foundation of the code base has shifted so drastically. > Cherry > > > picking will become very messy and introduce a high level of risk > imo. > > > > > > Also, you'd have to manage the scope of these minor releases which > we are > > > not very good at because the designated release manager has a > full-time > > job > > > to attend to and now we have even more releases to > > > scope/test/validate/approve. I like the idea of faster releases > but this > > > sounds like more work that nobody has the bandwidth for. > > > > > > Ok, so rather than shooting down an idea with no alternative > proposal, I > > > propose this. At Comcast, we are trying to get to the point where > we > > > release TC to prod from the head of master every few weeks. What > if every > > > time Comcast does an internal upgrade, they note the commit hash > and once > > > comcast is happy with the upgrade in their prod enviro, they > propose a > > > release pinned to that commit hash? With this approach you'd end > up with: > > > > > > 1. fast releases (hopefully every few weeks) > > > 2. well tested releases as it's in a real prod enviro > > > 3. little/no validation work for anyone > > > > > > Also, this doesn't have to be Comcast. If anyone else is out front > at the > > > edge of master and running it in a trusted enviro, they could > propose a > > > release as well. > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman < > david.neuma...@gmail.com> > > > wrote: > > > > > > > I have been thinking about how we can get better with our release > > > cadence. > > > > I feel like we have slowed to a crawl and not been as good as we > should > > > > about how we release. Our last Major release was in March and we > > haven't > > > > had a real release since. Moving forward I would like to see > us get > > on > > > a > > > > more consistent cadence and provide smaller releases that can be > more > > > > easily digested by the community. It's my hope that we can get > to a > > > > cadence where we are doing releases every 4 to 6 weeks, we are > > releasing > > > > in such a way that not all releases all required (e.g. if you > are on > > 4.0 > > > > you don't need 4.1, you can just install 4.2), and we are more > > deliberate > > > > about what we put into a release. > > > > > > > > I think we can accomplish this by using our release branches > better. > > For > > > > each major release we will cut a release branch from master > (e.g. 4.x) > > > and > > > > then we will use cherry-picking to add new features and bug > fixes to > > that > > > > release. This means that if 4.0 has been released and you want > to get > > > your > > > > feature/bug fix in 4.1, you will first submit your PR to master > and > > then > > > > cheery pick your squash merged commit to the 4.x branch which we > will > > use > > > > to create the 4.1 release. I think we should allow either the > > > contributor > > > > or the committer (who is merging the PR) to suggest if a feature > goes > > > into > > > > the release branch, and if we disagree we can take it to the > list. If > > we > > > > decide that every PR to master also goes into 4.1 then so be it, > but at > > > > least we are making a conscious decision of what goes into the > next > > > > release. This will allow us to not be so worried about what we > are > > > > merging into master and how that will affect our next release. > > According > > > > to our 6 week cadence we will cut a new release from the current > > feature > > > > branch and put that up for a vote. These releases will be small > and > > > > testable enough that we can get feedback quickly and move from > RC to > > > > release in a short time. If a release has too many issues then > we may > > > end > > > > up deciding to skip that minor release in favor of the next one, > > > hopefully > > > > this does not happen very often if at all. > > > > > > > > Once a breaking change is introduced to master which will > require a new > > > > major release, we will create a new major release branch from > master > > > (which > > > > will mean a new release manager). We will then repeat the same > process > > > > with the new release branch, etc, etc. > > > > > > > > As for LTS we will provide support for the latest major release > plus > > the > > > > one previous. So, once we release 4.0 we will support 4.0 and > 3.1. > > Any > > > > security issues that arise will -- if present -- be applied to > each of > > > > these versions. Once 4.1 is released we will support 4.1 and > 3.1, etc, > > > > etc. > > > > > > > > Please let me know if you have any thoughts on this plan. If > there are > > > no > > > > major objections I would like to try this with 4.x with the idea > that > > we > > > > will adjust how we do things as necessary. > > > > > > > > We need to get better at releasing and something needs to change, > > > hopefully > > > > this can get us going in the right direction. > > > > > > > > Thanks, > > > > Dave > > > > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using > cherry-picks to > > our > > > > release branch to control what goes into a minor release. Major > > releases > > > > will be cut from master as necessary - such as if there is a > breaking > > > > change that is introduced. > > > > > > > > > > > >
