In my opinion, in order to get to a cadence we are discussing we need to put a lot more work into the CI system. It has been failing consistently, doesn’t block PR approvals/merges and when it actually runs it does not test anything besides build and license headers.
In the past couple of weeks we have had (off the top of my head): * Unit tests broken * TO API tests broken * Golang vendor issues * TO Go build issues All were merged and broke master. Yes, having release branches sounds great and more aggressive cadence will minimize the amount of risk but I think that comes with the need to improve on our automated validation and testing. We have proven that we have not kept master in a good state and adding more releases (more branches) will make it even harder to keep that stability we are already fighting. On 10/30/19, 9:17 AM, "Jeremy Mitchell" <[email protected]> wrote: Yeah, I get it. No one company should be driving release schedules/scope. What I was really getting at is if ANY company has a recent version in a test or prod-like enviro (meaning the version is being exercised and thoroughly tested), then maybe we consider simply cutting a release from that version. For example, imagine company X had this version in a test/prod enviro: Master-10287.7e62d07 ( https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master) Then I would be in favor of cutting the 4.0 release from Master-10287.7e62d07 as we know it is proven to work based on company X's actual use of it. The truth is (imo), we don't have the bandwidth to manually verify releases that we are not using/have not used nor do we have the necessary automated test coverage to verify these releases. So most of our releases are significantly untested. On Wed, Oct 30, 2019 at 8:25 AM ocket 8888 <[email protected]> wrote: > I'm really not a fan of allowing Comcast to dictate the release scope and > schedule. If cherry-picking is too messy, then we can just cut new minor > releases directly from master. > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <[email protected]> wrote: > > > I don't think it's as easy as cherry picking (backporting) certain > features > > into a release branch. I could be wrong but I really don't think it is. > > > > So what I'm hearing is that 4.0 gets cut from master and we go through > with > > our normal process of testing, validating, etc. At the same time, 4.1 > > branch is created from 4.0. Master moves on as normal. Let's say 25 > commits > > come in to master during the next 6 weeks. During that 6 weeks, we cherry > > pick only the features that look interesting for 4.1? (the rest just stay > > in master and will get picked up in 5.0?) > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly 6 weeks apart > and > > have only "interesting" features so the releases are incrementally very > > small and much easier to test/validate, however, those features might > > depend on other commits. At some point, cherry picking becomes impossible > > as the foundation of the code base has shifted so drastically. Cherry > > picking will become very messy and introduce a high level of risk imo. > > > > Also, you'd have to manage the scope of these minor releases which we are > > not very good at because the designated release manager has a full-time > job > > to attend to and now we have even more releases to > > scope/test/validate/approve. I like the idea of faster releases but this > > sounds like more work that nobody has the bandwidth for. > > > > Ok, so rather than shooting down an idea with no alternative proposal, I > > propose this. At Comcast, we are trying to get to the point where we > > release TC to prod from the head of master every few weeks. What if every > > time Comcast does an internal upgrade, they note the commit hash and once > > comcast is happy with the upgrade in their prod enviro, they propose a > > release pinned to that commit hash? With this approach you'd end up with: > > > > 1. fast releases (hopefully every few weeks) > > 2. well tested releases as it's in a real prod enviro > > 3. little/no validation work for anyone > > > > Also, this doesn't have to be Comcast. If anyone else is out front at the > > edge of master and running it in a trusted enviro, they could propose a > > release as well. > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <[email protected]> > > wrote: > > > > > I have been thinking about how we can get better with our release > > cadence. > > > I feel like we have slowed to a crawl and not been as good as we should > > > about how we release. Our last Major release was in March and we > haven't > > > had a real release since. Moving forward I would like to see us get > on > > a > > > more consistent cadence and provide smaller releases that can be more > > > easily digested by the community. It's my hope that we can get to a > > > cadence where we are doing releases every 4 to 6 weeks, we are > releasing > > > in such a way that not all releases all required (e.g. if you are on > 4.0 > > > you don't need 4.1, you can just install 4.2), and we are more > deliberate > > > about what we put into a release. > > > > > > I think we can accomplish this by using our release branches better. > For > > > each major release we will cut a release branch from master (e.g. 4.x) > > and > > > then we will use cherry-picking to add new features and bug fixes to > that > > > release. This means that if 4.0 has been released and you want to get > > your > > > feature/bug fix in 4.1, you will first submit your PR to master and > then > > > cheery pick your squash merged commit to the 4.x branch which we will > use > > > to create the 4.1 release. I think we should allow either the > > contributor > > > or the committer (who is merging the PR) to suggest if a feature goes > > into > > > the release branch, and if we disagree we can take it to the list. If > we > > > decide that every PR to master also goes into 4.1 then so be it, but at > > > least we are making a conscious decision of what goes into the next > > > release. This will allow us to not be so worried about what we are > > > merging into master and how that will affect our next release. > According > > > to our 6 week cadence we will cut a new release from the current > feature > > > branch and put that up for a vote. These releases will be small and > > > testable enough that we can get feedback quickly and move from RC to > > > release in a short time. If a release has too many issues then we may > > end > > > up deciding to skip that minor release in favor of the next one, > > hopefully > > > this does not happen very often if at all. > > > > > > Once a breaking change is introduced to master which will require a new > > > major release, we will create a new major release branch from master > > (which > > > will mean a new release manager). We will then repeat the same process > > > with the new release branch, etc, etc. > > > > > > As for LTS we will provide support for the latest major release plus > the > > > one previous. So, once we release 4.0 we will support 4.0 and 3.1. > Any > > > security issues that arise will -- if present -- be applied to each of > > > these versions. Once 4.1 is released we will support 4.1 and 3.1, etc, > > > etc. > > > > > > Please let me know if you have any thoughts on this plan. If there are > > no > > > major objections I would like to try this with 4.x with the idea that > we > > > will adjust how we do things as necessary. > > > > > > We need to get better at releasing and something needs to change, > > hopefully > > > this can get us going in the right direction. > > > > > > Thanks, > > > Dave > > > > > > > > > TL;DR - I am proposing a 6 week release cycle using cherry-picks to > our > > > release branch to control what goes into a minor release. Major > releases > > > will be cut from master as necessary - such as if there is a breaking > > > change that is introduced. > > > > > >
