Just throwing out that the judicious use of feature flags could reduce the pain
from unstable or incomplete new features. Long lived feature branches are
something I believe should be avoided.
Geoff
On 10/31/19, 2:32 PM, "Rawlin Peters" <raw...@apache.org> wrote:
Alright, I think that sounds fair. In general what I'm hearing is,
every 4-6 weeks:
1. fast-forward 4.x to the most recent "stable" commit of master
(maybe we automate this and fast-forward every day if some set of
tests pass, unless we explicitly know master is unstable for other
reasons)
2. cut 4.*.x off 4.x, only cherry-pick new bug/security fixes into
4.*.x that are necessary for the release (e.g. bugs that break the
data plane or prevent operators from being able to do their job or run
the control plane)
Which means the release manager needs to know/track:
- unstable commits along with their corresponding fixes
- incomplete features
- features we'd like to hold until the next release (due to current
release having enough already)
- any cherry-picks that were deemed necessary for a certain release
(so that they can either be carried forward into the next minor
release or, ideally, included as part of fast-forwarding 4.x to the
most recent stable commit on master)
I think this will work. We gain the ability to:
1. let master be "unstable" periodically
2. control the size of releases
3. make releases mirror master as closely as possible (because they
will just be a snapshot of the most recent version of master that was
considered stable -- unless absolutely necessary bug/security fixes
were cherry-picked in)
Does that all sound right, Dave?
- Rawlin
On Thu, Oct 31, 2019 at 8:37 AM David Neuman <david.neuma...@gmail.com>
wrote:
>
> I think a third reason would be that we are getting very close to our
> release branch date, it's a large/complex feature, and we feel like the
> release already has enough in it. In that case we could hold off on
adding
> that change to the 4.x branch until after we cut the 4.x.x branch. The
> benefit is that we can still merge that feature into master and get all of
> our testing (both internally and externally) running against it. If you
> stick with just branching off master and this situation arises then we get
> in the spot where we just don't merge the PR at all until after the
release
> branch is cut (we have two examples of this right now). I personally
think
> it's better to get this into master and let our automation run against it
> than it is to leave it out in a PR even though it is ready.
>
> As for the process, I was thinking the same thing except that you don't
> even need to cherry-pick. In your scenario above you can just rebase
> master into 4.x once E was merged. This avoids the headaches of having to
> cherry pick each commit while still giving us the benefit of having all of
> the changes on the release branch. We can even decide to setup automation
> to rebase master into the release branch on an interval or on demand. If
> we rebase in something that turns out to be broken we just roll back to
the
> previous rebase (or a certain commit) until that is fixed and then rebase
> again. If a feature gets merged to master that we aren't ready to put
into
> the release, then we stop the automated rebase until we are ready to get
it
> into the release -- at that point we should consider ONLY cherry-picking
> bug fixes for the remainder of that release cycle to make things easier to
> rebase again.
>
> I am sure there will be unintended consequences, and I am sure we will
have
> challenges to work through, but I still think that it's worth trying. We
> should fail fast and be willing to change course if necessary.
> I think if we can be consistent enough with our releases then maybe we can
> discuss branching from master again.
>
> Thanks,
> Dave
>
>
>
> On Wed, Oct 30, 2019 at 5:34 PM Rawlin Peters <raw...@apache.org> wrote:
>
> > I guess my concern is really about what possible reasons we might have
> > to not cherry-pick a specific PR that had been merged into master
> > already. The only reasons I can think of to not cherry-pick something
> > would be:
> > 1. the PR was completely broken (e.g. breaks the build, tests don't
> > pass, doesn't compile, etc).
> > 2. the PR one one of a larger multi-PR "feature", and *we know*
> > releasing some parts early could cause problems (that knowing part is
> > crucial)
> >
> > We should always strive to catch issue #1 in our CI before it merges
> > into master. When things like that merge it can block people from
> > doing development starting at that point until it's fixed.
> > I can see why issue #2 might arise due to multiple different people
> > developing one larger feature, but we should still strive to break
> > larger features up into individual pieces that can reasonably stand on
> > their own if possible. Due to time constraints or other reasons I can
> > see that might be a little difficult.
> >
> > That said, I would still strive to make the 4.x release branch a
> > snapshot of master at some "known good point" in time. Let's say we
> > cut 4.x at commit A, then a broken commit B gets merged into master.
> > If it's completely obvious that it's a broken commit, I'd wait until
> > at least a commit C gets merged to fix it. Then I would cherry-pick B
> > and C into 4.x at the same time. If the fix is really like a commit E
> > with unrelated commits C and D in between, ideally I would still wait
> > to cherry-pick C and D until a "fix" commit E was merged (to fix
> > commit B), at which point I'd cherrypick commits B through E at the
> > same time, in order, to avoid any possible merge conflicts.
> >
> > I think to make that process easier for myself I would just
> > cherry-pick everything into the 4.x branch as it gets merged into
> > master. Then, after 4-6 weeks of cherry-picking commits, if unstable
> > commits were not fixed/completed in time, I would reset the 4.x branch
> > to the last stable commit. This would mean that unstable
> > commits/incomplete features could potentially block things from going
> > into the release. I might count that as a positive because it would
> > incentivize us to improve our CI and do our best to also keep the
> > master branch as stable as possible.
> >
> > The end result would be the same as just continually jumping up the
> > master branch from known good, stable points in time to the next,
> > creating a minor release branch at each of those points.
> >
> > - Rawlin
> >
> >
> >
> > On Wed, Oct 30, 2019 at 2:36 PM Dave Neuman <neu...@apache.org> wrote:
> > >
> > > Sorry, I have taken too long to respond.
> > >
> > > +1000000 on better CI and Testing, I think we all agree that needs to
get
> > > better and is a different beast than this topic.
> > >
> > > As for the release branches, I am trying to follow a process similar
to
> > > ATS, which is usually the model we try to follow with our project.
> > > While I agree that branching from master is the ideal solution, we
tried
> > > that and it didn't work. I feel like we are trying to agree to do the
> > same
> > > thing that we already tried and didn't work. I think we should have a
> > > little freedom to commit things into master and, if they break things,
> > work
> > > on getting them fixed. We also might commit PRs to master that are
> > partial
> > > implementations of a bigger feature. If we are using a time based
> > > strategy to cut our releases from master then we are going to be
> > > continually worrying about making sure Master stays not broken. I
think
> > we
> > > should try to keep master deployable, but it's fine if it's not for
short
> > > periods of time.
> > >
> > > I am not trying to suggest that we are "picky" about what goes into
our
> > > release branch. I think we should try to cherry-pick as much as we
can
> > > into our release branch once it is ready to be cherry-picked. I
think we
> > > should be able to take the tip of our release branch and deploy it
> > whenever
> > > we want with some confidence since we know that features/bug-fixes
that
> > > have been cherry-picked have already been reviewed and tested against
> > > master. Yes, its more work, but it also give us the ability to have
more
> > > confidence in what is in our release at any given time. Again, we
should
> > > be trying to cherry-pick in everything that makes sense to
cherry-pick.
> > >
> > > I think the process that I would like to see looks like this:
> > > - Cut a 4.x branch from Master
> > > - Once we think we are ready for a RC cut a 4.0.x branch from the 4.x
> > branch
> > > - Tag the 4.0.0 release from the 4.0.x branch
> > > - New non-breaking changes and bug fixes that go into master are also
> > > Cherry Picked to 4.x
> > > - After 6 weeks we make a 4.1.x branch from 4.x with a 4.1.0 tag when
we
> > > are ready for a RC.
> > > - Rinse and repeat for all 4.x releases
> > > - Once there's a need for a major release we make a 5.x branch from
> > master
> > >
> > > If we find a bug in 4.0.0 then we commit the fix to master, then 4.x,
> > then
> > > 4.0.x which we use to create the new 4.0.1 release (assuming we don't
> > have
> > > 4.1 yet). Yes, that is a more work than today, but I think A) it's
not
> > > really that bad and B) it gives us more control.
> > >
> > > This should allow us to deploy much quicker and with more confidence
than
> > > is possible today. We can also choose to deploy from any point in
time
> > > from the 4.x branch with relatively high confidence.
> > >
> > > I'll wrap this up by saying that I don't think there's a perfect
> > solution,
> > > but I think we can agree on something that is worth trying at least
with
> > > the 4.x release life cycle. Once we have some experience, we can
figure
> > > out where it is not meeting our needs and figure out how to adjust as
> > > necessary. I think we can all agree our current process is not
working.
> > >
> > > Thanks,
> > > Dave
> > >
> > >
> > >
> > > On Wed, Oct 30, 2019 at 11:01 AM Rawlin Peters <raw...@apache.org>
> > wrote:
> > >
> > > > +1 on more frequent releases
> > > > +1 on improving our CI. We need to get our CI to the point where
we're
> > > > confident deploying changes to production if the CI passes.
> > > > +1 on "trunk-based" development as opposed to GitFlow
> > > > -1 on cherry-picking new features. This is risky, tedious, and
> > > > represents unnecessary effort that would be better spent elsewhere,
> > > > like improving our CI or adding more automated tests.
> > > > +1 on cutting minor release branches directly off master
> > > >
> > > > I think cutting minor release branches directly off master can put
us
> > > > in a win-win scenario. Our releases become smaller and more
frequent,
> > > > and we avoid the unnecessary risk and effort of cherry-picking new
> > > > features onto stale release branches.
> > > >
> > > > This is the process that I would propose:
> > > > 1. Cut a 4.0.x branch off master which becomes the 4.0 RC
> > > > 2. Develop master for 4-6 weeks.
> > > > 3. Cut a 4.1.x branch off master which becomes the 4.1 RC.
> > > > 4. Repeat steps 2 and 3 for new minor releases as necessary.
> > > > 5. Whenever a major (show-stopping) bug or security issue is found,
> > > > cherry-pick the fix from master into the latest release branch (e.g.
> > > > 4.1.x), and create a 4.1.1 RC.
> > > >
> > > > As the 4.x release manager, I really don't want to get into the
> > > > business of selecting what should and should not go into a
particular
> > > > release. Every 4-6 weeks, we should just cut a new minor release
> > > > branch off master and take everything that was contributed during
that
> > > > time, as long as everything builds fine and all the tests pass. The
> > > > master branch, while it doesn't always have to be perfect, should at
> > > > least always be deployable. If a new feature is merged with bugs and
> > > > becomes a show-stopper, it should be either immediately remedied or
> > > > reverted on master while we work on a fix.
> > > >
> > > > - Rawlin
> > > >
> > > > On Wed, Oct 30, 2019 at 9:30 AM Hoppal, Michael
> > > > <michael_hop...@comcast.com> wrote:
> > > > >
> > > > > In my opinion, in order to get to a cadence we are discussing we
> > need to
> > > > put a lot more work into the CI system. It has been failing
> > consistently,
> > > > doesn’t block PR approvals/merges and when it actually runs it does
not
> > > > test anything besides build and license headers.
> > > > >
> > > > > In the past couple of weeks we have had (off the top of my head):
> > > > >
> > > > > * Unit tests broken
> > > > > * TO API tests broken
> > > > > * Golang vendor issues
> > > > > * TO Go build issues
> > > > >
> > > > > All were merged and broke master.
> > > > >
> > > > > Yes, having release branches sounds great and more aggressive
cadence
> > > > will minimize the amount of risk but I think that comes with the
need
> > to
> > > > improve on our automated validation and testing.
> > > > >
> > > > > We have proven that we have not kept master in a good state and
> > adding
> > > > more releases (more branches) will make it even harder to keep that
> > > > stability we are already fighting.
> > > > >
> > > > > On 10/30/19, 9:17 AM, "Jeremy Mitchell" <mitchell...@gmail.com>
> > wrote:
> > > > >
> > > > > Yeah, I get it. No one company should be driving release
> > > > schedules/scope.
> > > > >
> > > > > What I was really getting at is if ANY company has a recent
> > version
> > > > in a
> > > > > test or prod-like enviro (meaning the version is being
exercised
> > and
> > > > > thoroughly tested), then maybe we consider simply cutting a
> > release
> > > > from
> > > > > that version. For example, imagine company X had this version
in
> > a
> > > > > test/prod enviro:
> > > > >
> > > > > Master-10287.7e62d07 (
> > > > >
> > > >
> >
https://protect2.fireeye.com/url?k=6592a637da71d1f5.65928183-ae2adcde1216f77b&u=https://github.com/apache/trafficcontrol/commits/master
> > > > )
> > > > >
> > > > > Then I would be in favor of cutting the 4.0 release from
> > > > > Master-10287.7e62d07 as we know it is proven to work based on
> > > > company X's
> > > > > actual use of it.
> > > > >
> > > > > The truth is (imo), we don't have the bandwidth to manually
> > verify
> > > > releases
> > > > > that we are not using/have not used nor do we have the
necessary
> > > > automated
> > > > > test coverage to verify these releases. So most of our
releases
> > are
> > > > > significantly untested.
> > > > >
> > > > > On Wed, Oct 30, 2019 at 8:25 AM ocket 8888
<ocket8...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I'm really not a fan of allowing Comcast to dictate the
release
> > > > scope and
> > > > > > schedule. If cherry-picking is too messy, then we can just
cut
> > new
> > > > minor
> > > > > > releases directly from master.
> > > > > >
> > > > > > On Tue, Oct 29, 2019, 15:59 Jeremy Mitchell <
> > mitchell...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > I don't think it's as easy as cherry picking (backporting)
> > > > certain
> > > > > > features
> > > > > > > into a release branch. I could be wrong but I really don't
> > think
> > > > it is.
> > > > > > >
> > > > > > > So what I'm hearing is that 4.0 gets cut from master and
we
> > go
> > > > through
> > > > > > with
> > > > > > > our normal process of testing, validating, etc. At the
same
> > > > time, 4.1
> > > > > > > branch is created from 4.0. Master moves on as normal.
Let's
> > say
> > > > 25
> > > > > > commits
> > > > > > > come in to master during the next 6 weeks. During that 6
> > weeks,
> > > > we cherry
> > > > > > > pick only the features that look interesting for 4.1? (the
> > rest
> > > > just stay
> > > > > > > in master and will get picked up in 5.0?)
> > > > > > >
> > > > > > > In theory, it sounds great. 4.1, 4.2, 4.3 all are roughly
6
> > > > weeks apart
> > > > > > and
> > > > > > > have only "interesting" features so the releases are
> > > > incrementally very
> > > > > > > small and much easier to test/validate, however, those
> > features
> > > > might
> > > > > > > depend on other commits. At some point, cherry picking
> > becomes
> > > > impossible
> > > > > > > as the foundation of the code base has shifted so
> > drastically.
> > > > Cherry
> > > > > > > picking will become very messy and introduce a high level
of
> > > > risk imo.
> > > > > > >
> > > > > > > Also, you'd have to manage the scope of these minor
releases
> > > > which we are
> > > > > > > not very good at because the designated release manager
has a
> > > > full-time
> > > > > > job
> > > > > > > to attend to and now we have even more releases to
> > > > > > > scope/test/validate/approve. I like the idea of faster
> > releases
> > > > but this
> > > > > > > sounds like more work that nobody has the bandwidth for.
> > > > > > >
> > > > > > > Ok, so rather than shooting down an idea with no
alternative
> > > > proposal, I
> > > > > > > propose this. At Comcast, we are trying to get to the
point
> > > > where we
> > > > > > > release TC to prod from the head of master every few
weeks.
> > What
> > > > if every
> > > > > > > time Comcast does an internal upgrade, they note the
commit
> > hash
> > > > and once
> > > > > > > comcast is happy with the upgrade in their prod enviro,
they
> > > > propose a
> > > > > > > release pinned to that commit hash? With this approach
you'd
> > end
> > > > up with:
> > > > > > >
> > > > > > > 1. fast releases (hopefully every few weeks)
> > > > > > > 2. well tested releases as it's in a real prod enviro
> > > > > > > 3. little/no validation work for anyone
> > > > > > >
> > > > > > > Also, this doesn't have to be Comcast. If anyone else is
out
> > > > front at the
> > > > > > > edge of master and running it in a trusted enviro, they
could
> > > > propose a
> > > > > > > release as well.
> > > > > > >
> > > > > > > Jeremy
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Oct 29, 2019 at 3:06 PM David Neuman <
> > > > david.neuma...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have been thinking about how we can get better with
our
> > > > release
> > > > > > > cadence.
> > > > > > > > I feel like we have slowed to a crawl and not been as
good
> > as
> > > > we should
> > > > > > > > about how we release. Our last Major release was in
March
> > and
> > > > we
> > > > > > haven't
> > > > > > > > had a real release since. Moving forward I would like
to
> > see
> > > > us get
> > > > > > on
> > > > > > > a
> > > > > > > > more consistent cadence and provide smaller releases
that
> > can
> > > > be more
> > > > > > > > easily digested by the community. It's my hope that we
can
> > > > get to a
> > > > > > > > cadence where we are doing releases every 4 to 6 weeks,
> > we are
> > > > > > releasing
> > > > > > > > in such a way that not all releases all required (e.g.
if
> > you
> > > > are on
> > > > > > 4.0
> > > > > > > > you don't need 4.1, you can just install 4.2), and we
are
> > more
> > > > > > deliberate
> > > > > > > > about what we put into a release.
> > > > > > > >
> > > > > > > > I think we can accomplish this by using our release
> > branches
> > > > better.
> > > > > > For
> > > > > > > > each major release we will cut a release branch from
master
> > > > (e.g. 4.x)
> > > > > > > and
> > > > > > > > then we will use cherry-picking to add new features and
bug
> > > > fixes to
> > > > > > that
> > > > > > > > release. This means that if 4.0 has been released and
you
> > > > want to get
> > > > > > > your
> > > > > > > > feature/bug fix in 4.1, you will first submit your PR to
> > > > master and
> > > > > > then
> > > > > > > > cheery pick your squash merged commit to the 4.x branch
> > which
> > > > we will
> > > > > > use
> > > > > > > > to create the 4.1 release. I think we should allow
either
> > the
> > > > > > > contributor
> > > > > > > > or the committer (who is merging the PR) to suggest if a
> > > > feature goes
> > > > > > > into
> > > > > > > > the release branch, and if we disagree we can take it to
> > the
> > > > list. If
> > > > > > we
> > > > > > > > decide that every PR to master also goes into 4.1 then
so
> > be
> > > > it, but at
> > > > > > > > least we are making a conscious decision of what goes
into
> > the
> > > > next
> > > > > > > > release. This will allow us to not be so worried about
> > what
> > > > we are
> > > > > > > > merging into master and how that will affect our next
> > release.
> > > > > > According
> > > > > > > > to our 6 week cadence we will cut a new release from the
> > > > current
> > > > > > feature
> > > > > > > > branch and put that up for a vote. These releases will
be
> > > > small and
> > > > > > > > testable enough that we can get feedback quickly and
move
> > from
> > > > RC to
> > > > > > > > release in a short time. If a release has too many
issues
> > > > then we may
> > > > > > > end
> > > > > > > > up deciding to skip that minor release in favor of the
next
> > > > one,
> > > > > > > hopefully
> > > > > > > > this does not happen very often if at all.
> > > > > > > >
> > > > > > > > Once a breaking change is introduced to master which
will
> > > > require a new
> > > > > > > > major release, we will create a new major release branch
> > from
> > > > master
> > > > > > > (which
> > > > > > > > will mean a new release manager). We will then repeat
the
> > > > same process
> > > > > > > > with the new release branch, etc, etc.
> > > > > > > >
> > > > > > > > As for LTS we will provide support for the latest major
> > > > release plus
> > > > > > the
> > > > > > > > one previous. So, once we release 4.0 we will support
4.0
> > and
> > > > 3.1.
> > > > > > Any
> > > > > > > > security issues that arise will -- if present -- be
> > applied to
> > > > each of
> > > > > > > > these versions. Once 4.1 is released we will support
4.1
> > and
> > > > 3.1, etc,
> > > > > > > > etc.
> > > > > > > >
> > > > > > > > Please let me know if you have any thoughts on this
plan.
> > If
> > > > there are
> > > > > > > no
> > > > > > > > major objections I would like to try this with 4.x with
the
> > > > idea that
> > > > > > we
> > > > > > > > will adjust how we do things as necessary.
> > > > > > > >
> > > > > > > > We need to get better at releasing and something needs
to
> > > > change,
> > > > > > > hopefully
> > > > > > > > this can get us going in the right direction.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Dave
> > > > > > > >
> > > > > > > >
> > > > > > > > TL;DR - I am proposing a 6 week release cycle using
> > > > cherry-picks to
> > > > > > our
> > > > > > > > release branch to control what goes into a minor
release.
> > > > Major
> > > > > > releases
> > > > > > > > will be cut from master as necessary - such as if there
is
> > a
> > > > breaking
> > > > > > > > change that is introduced.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> >
