Thanks Ben! I wonder if the new guidelines belong in the "Committing" docs instead? http://mesos.apache.org/documentation/latest/committing/
On Thu, Jul 26, 2018 at 2:18 PM, Benjamin Mahler <bmah...@apache.org> wrote: > Thanks to everyone who chimed in. To Gilbert's questions, I think that's > outside the territory of what a policy or guidelines will cover, and will > be up to the judgement of the committer. > > My initial search for "backport" was insufficient and I missed the > following existing comment in a different section of versioning.md lays out > a policy of only critical issues: > > <start quote> > At any given time, 3 releases are supported: ... Support means fixing of > critical issues that affect the release. Once an issue is deemed critical, > it will be fixed in only those affected releases that are still supported. > ... > > Which issues are considered critical? > > Security fixes > Compatibility regressions > Functional regressions > Performance regressions > Fixes for 3rd party integration (e.g., Docker remote API) > > Whether an issue is considered critical or not is sometimes subjective. In > some cases it is obvious and sometimes it is fuzzy. Users should work with > committers to figure out the criticality of an issue and get agreement and > commitment for support. > <end quote> > > From the discussion here, it sounds like we should add the guidelines > similar to what Greg suggested above to clarify that there's some > subjectivity here, as well as keeping some of the clear cases (like what > the existing documentation does). I can send out a change sometime soon. > > On Wed, Jul 18, 2018 at 10:37 AM, Gilbert Song <gilb...@mesosphere.io> > wrote: > >> Thanks for clarifying the backporting policy, BenM! >> >> I totally agree with the changes proposed for the backporting policy, but >> I realize two more scenarios that are more clear to me yet: >> >> - There are some bugs that are not fixable (due to legacy technical >> decisions), and we end up with fixing the issue by a semantic/behavior >> change in a new release. Do we expect this semantic/behavior change being >> backported? >> - There might be some bugs that root cause is unknown yet, but it did >> impact on a couple releases. If we decide to add some commits for >> debugging >> purpose (e.g., a new debugging endpoint, or more logging), should we also >> allow these patches to be backported? >> >> For #2, I think we should do the backporting, but for #1, maybe more >> discussion is needed since it relates to whether the user has to upgrade or >> not. >> >> Cheers, >> Gilbert >> >> On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <larry...@mac.com> wrote: >> >>> I don’t have a big stake in, however, one opinion is if a large >>> commercial enterprise was using a specific release that is working the >>> desire is often to only upgrade if necessary. Necessary can be for a >>> number of reasons including new features; however if a new feature is not >>> needed the compelling reason to upgrade is to fix a specific problem that >>> is causing issues. Thus keeping a maintenance release stable is very >>> important and reducing the chance for, while fixing one problem, >>> introducing another. >>> >>> Often a clear classification of severity of the problem would dictate >>> the need to make a change. (yes these can be subjective, but some guidance >>> would be better than nothing). >>> >>> It might be good to give committers guidance on back porting things that >>> have a high impact on improving a problem. Fixing a crashing bug, fixing a >>> degenerative performance issue, etc, where these issues have no easy/viable >>> work around. Nice to have fixes aren’t, always, worth updating to. >>> >>> There can be an argument to respond with a “then don’t upgrade” but if >>> changing the release with “nice to have’s” and several point releases later >>> when a critical bug is fixed then the org if forced to accept the risk of >>> the nice to have’s. >>> >>> just an opinion. >>> …larry >>> >>> >>> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <chhs...@mesosphere.io> >>> wrote: >>> >>> I just have a comment on a special case: >>> If a fix for a flaky test is easy to backport, >>> IMO we probably should backport it, >>> otherwise if someone backports another critical fix in the future, >>> it would take them extra effort to check all CI failures. >>> >>> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vinodk...@apache.org> >>> wrote: >>> >>>> I like how you summarized it Greg and I would vote for leaving the >>>> decision >>>> to the committer too. In addition to what others mentioned, I think >>>> committer should've the responsibility because if things break in a >>>> point >>>> release (after it is released), it is the committer and contributor who >>>> are >>>> on the hook to triage and fix it and not the release manager. >>>> >>>> Having said that, if "during" the release process (i.e., cutting an RC) >>>> these backports cause delays for a release manager in getting the >>>> release >>>> out (e.g., CI flakiness introduced due to backports), release manager >>>> could >>>> be the ultimate arbiter on whether such a backport should be reverted or >>>> fixed by the committer/contributor. Hopefully such issues are caught >>>> much >>>> before a release process is started (e.g., CI running against release >>>> branches). >>>> >>>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yujie....@gmail.com> wrote: >>>> >>>> > Greg, I like your idea of adding a prescriptive "policy" when >>>> evaluating >>>> > whether a bug fix should be backported, and leave the decision to >>>> committer >>>> > (because they have the most context, and avoid a bottleneck in the >>>> > process). >>>> > >>>> > - Jie >>>> > >>>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <g...@mesosphere.io> >>>> wrote: >>>> > >>>> > > My impression is that we have two opposing schools of thought here: >>>> > > >>>> > > 1. Backport as little as possible, to avoid unforeseen >>>> consequences >>>> > > 2. Backport as much as proves practical, to eliminate bugs in >>>> > > supported versions >>>> > > >>>> > > Do other people agree with this assessment? >>>> > > >>>> > > If so, how can we find common ground? One possible solution would >>>> be to >>>> > > leave the decision on backporting up to the committer, without >>>> > specifying a >>>> > > project-wide policy. This seems to be the status quo, and would >>>> lead to >>>> > > some variation across committers regarding what types of fixes are >>>> > > backported. We could also choose to delegate the decision to the >>>> release >>>> > > manager; I favor leaving the decision with the committer, to >>>> eliminate >>>> > the >>>> > > burden on release managers. >>>> > > >>>> > > Here's a thought: rather than defining a prescriptive "policy" that >>>> we >>>> > > expect committers to abide by, we could enumerate in the >>>> documentation >>>> > the >>>> > > competing concerns that we expect committers to consider when making >>>> > > decisions on backports. The committing docs could read something >>>> like: >>>> > > >>>> > > "When bug fixes are committed to master, the committer should >>>> evaluate >>>> > the >>>> > > fix to determine whether or not it should be backported to supported >>>> > > versions. This is left to the committer, but they are expected to >>>> weigh >>>> > the >>>> > > following concerns when making the decision: >>>> > > >>>> > > - Every backported change comes with a risk of unintended >>>> > > consequences. The change should be carefully evaluated to ensure >>>> that >>>> > such >>>> > > side-effects are highly unlikely. >>>> > > - As the complexity of applying a backport increases due to merge >>>> > > conflicts, the likelihood of unintended consequences also >>>> increases. >>>> > Bug >>>> > > fixes which require extensive rebasing should only be backported >>>> when >>>> > the >>>> > > bug is critical enough to warrant the risk. >>>> > > - Users of supported versions benefit greatly from the >>>> resolution of >>>> > > bugs in point releases. Thus, whenever concerns #1 and #2 can be >>>> > allayed >>>> > > for a given bug fix, it should be backported." >>>> > > >>>> > > >>>> > > Cheers, >>>> > > Greg >>>> > > >>>> > > >>>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov < >>>> a...@mesosphere.com> >>>> > > wrote: >>>> > > >>>> > >> Back porting as little as possible is the ultimate goal for me. My >>>> > >> reasons are closely aligned with what Andrew wrote above. >>>> > >> >>>> > >> If we agree on this strategy, the next question is how to enforce >>>> it. My >>>> > >> intuition is that committers will lean towards back porting their >>>> > patches >>>> > >> in arguable cases, because humans tend to overestimate the >>>> importance of >>>> > >> their personal work. Delegating the decision in such cases to a >>>> release >>>> > >> manager in my opinion will help us enforce the strategy of minimal >>>> > number >>>> > >> backports. As a bonus, the release manager will have a much better >>>> > >> understanding of what's going on with the release, keyword: "more >>>> > >> ownership". >>>> > >> >>>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer < >>>> > >> and...@schwartzmeyer.com> wrote: >>>> > >> >>>> > >>> I believe I fall somewhere between Alex and Ben. >>>> > >>> >>>> > >>> As for deciding what to backport or not, I lean toward Alex's >>>> view of >>>> > >>> backporting as little as possible (and agree with his criteria). >>>> My >>>> > >>> reasoning is that all changes can have unforeseen consequences, >>>> which I >>>> > >>> believe is something to be actively avoided in already released >>>> > versions. >>>> > >>> The reason for backporting patches to fix regressions is the same >>>> as >>>> > the >>>> > >>> reason to avoid backporting as much as possible: keep behavior >>>> > consistent >>>> > >>> (and safe) within a release. With that as the goal of a branch in >>>> > >>> maintenance mode, it makes sense to fix regressions, and make >>>> > exceptions to >>>> > >>> fix CVEs and other critical/blocking issues. >>>> > >>> >>>> > >>> As for who should decide what to backport, I lean toward Ben's >>>> view of >>>> > >>> the burden being on the committer. I don't think we should add >>>> more >>>> > work >>>> > >>> for release managers, and I think the committer/shepherd >>>> obviously has >>>> > the >>>> > >>> most understanding of the context around changes proposed for >>>> backport. >>>> > >>> >>>> > >>> Here's an example of a recent bugfix which I backported: >>>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790) >>>> > >>> >>>> > >>> While normally I believe this change falls under "avoid due to >>>> > >>> unforeseen consequences," I made an exception as the bug was old, >>>> circa >>>> > >>> 2015, (indicating it had been an issue for others), and was >>>> causing >>>> > >>> recurring failures in testing. The fix itself was very small, >>>> meaning >>>> > it >>>> > >>> was easier to evaluate for possible side effects, so I felt a >>>> little >>>> > safer >>>> > >>> in that regard. The effect of not having the fix was a fatal and >>>> > undesired >>>> > >>> crash, which furthermore left troublesome side effects on the >>>> system >>>> > (you >>>> > >>> couldn't bring the agent back up). And lastly, a dependent project >>>> > (DC/OS) >>>> > >>> wanted it in their next bump, which necessitated backporting to >>>> the >>>> > release >>>> > >>> they were pulling in. >>>> > >>> >>>> > >>> I think in general we should backport only as necessary, and >>>> leave it >>>> > on >>>> > >>> the committers to decide if backporting a particular change is >>>> > necessary. >>>> > >>> >>>> > >>> >>>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote: >>>> > >>> >>>> > >>>> This is exactly where our views differ, Ben : ) >>>> > >>>> >>>> > >>>> Ideally, I would like a release manager to have more ownership >>>> and >>>> > less >>>> > >>>> manual work. In my imagination, a release manager has more power >>>> and >>>> > >>>> control about dates, features, backports and everything that is >>>> > related >>>> > >>>> to >>>> > >>>> "their" branch. I would also like us to back port as little as >>>> > >>>> possible, to >>>> > >>>> simplify testing and releasing patch versions. >>>> > >>>> >>>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler < >>>> bmah...@apache.org> >>>> > >>>> wrote: >>>> > >>>> >>>> > >>>> +user, I probably it would be good to hear from users as well. >>>> > >>>>> >>>> > >>>>> Please see the original proposal as well as Alex's proposal and >>>> let >>>> > us >>>> > >>>>> know >>>> > >>>>> your thoughts. >>>> > >>>>> >>>> > >>>>> To continue the discussion from where Alex left off: >>>> > >>>>> >>>> > >>>>> > Other bugs and significant improvements, e.g., performance, >>>> may be >>>> > >>>>> back >>>> > >>>>> ported, >>>> > >>>>> the release manager should ideally be the one who decides on >>>> this. >>>> > >>>>> >>>> > >>>>> I'm a little puzzled by this, why is the release manager >>>> involved? As >>>> > >>>>> we >>>> > >>>>> already document, backports occur when the bug is fixed, so this >>>> > >>>>> happens in >>>> > >>>>> the steady state of development, not at release time. The >>>> release >>>> > >>>>> manager >>>> > >>>>> only comes in at the time of the release itself, at which point >>>> all >>>> > >>>>> backports have already happened and the release manager handles >>>> the >>>> > >>>>> release >>>> > >>>>> process. Only blocker level issues can stop the release and >>>> while the >>>> > >>>>> release manager has a strong say, we should generally agree on >>>> what >>>> > >>>>> consists of a release blocking issue. >>>> > >>>>> >>>> > >>>>> Just to clarify my workflow, I generally backport every bug fix >>>> I >>>> > >>>>> commit >>>> > >>>>> that applies cleanly, right after I commit it to master (with >>>> the >>>> > >>>>> exceptions I listed below). >>>> > >>>>> >>>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov < >>>> a...@mesosphere.com >>>> > > >>>> > >>>>> wrote: >>>> > >>>>> >>>> > >>>>> > I would like to back port as little as possible. I suggest the >>>> > >>>>> following >>>> > >>>>> > criteria: >>>> > >>>>> > >>>> > >>>>> > * By default, regressions are back ported to existing release >>>> > >>>>> branches. A >>>> > >>>>> > bug is considered a regression if the functionality is >>>> present in >>>> > the >>>> > >>>>> > previous minor or patch version and is not affected by the bug >>>> > there. >>>> > >>>>> > >>>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be back >>>> ported. >>>> > >>>>> > >>>> > >>>>> > * Other bugs and significant improvements, e.g., performance, >>>> may >>>> > be >>>> > >>>>> back >>>> > >>>>> > ported, the release manager should ideally be the one who >>>> decides >>>> > on >>>> > >>>>> this. >>>> > >>>>> > >>>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone < >>>> vinodk...@apache.org >>>> > > >>>> > >>>>> wrote: >>>> > >>>>> > >>>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement with the >>>> > >>>>> points you >>>> > >>>>> > > made. >>>> > >>>>> > > >>>> > >>>>> > > Once we have consensus, would you mind updating the doc? >>>> > >>>>> > > >>>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler < >>>> > >>>>> bmah...@apache.org> >>>> > >>>>> > > wrote: >>>> > >>>>> > > >>>> > >>>>> > > > I realized recently that we aren't all on the same page >>>> with >>>> > >>>>> > backporting. >>>> > >>>>> > > > We currently only document the following: >>>> > >>>>> > > > >>>> > >>>>> > > > "Typically the fix for an issue that is affecting >>>> supported >>>> > >>>>> releases >>>> > >>>>> > > lands >>>> > >>>>> > > > on the master branch and is then backported to the release >>>> > >>>>> branch(es). >>>> > >>>>> > In >>>> > >>>>> > > > rare cases, the fix might directly go into a release >>>> branch >>>> > >>>>> without >>>> > >>>>> > > landing >>>> > >>>>> > > > on master (e.g., fix / issue is not applicable to >>>> master)." [1] >>>> > >>>>> > > > >>>> > >>>>> > > > This leaves room for interpretation about what lies >>>> outside of >>>> > >>>>> > "typical". >>>> > >>>>> > > > Here's the simplest way I can explain what I stick to, >>>> and I'd >>>> > >>>>> like >>>> > >>>>> to >>>> > >>>>> > > hear >>>> > >>>>> > > > what others have in mind: >>>> > >>>>> > > > >>>> > >>>>> > > > * By default, bug fixes at any level should be backported >>>> to >>>> > >>>>> existing >>>> > >>>>> > > > release branches if it affects those releases. Especially >>>> > >>>>> important: >>>> > >>>>> > > > crashes, bugs in non-experimental features. >>>> > >>>>> > > > >>>> > >>>>> > > > * Exceptional cases that can omit backporting: difficult >>>> to >>>> > >>>>> backport >>>> > >>>>> > > fixes >>>> > >>>>> > > > (especially if the bugs are deemed of low priority), bugs >>>> in >>>> > >>>>> > experimental >>>> > >>>>> > > > features. >>>> > >>>>> > > > >>>> > >>>>> > > > * Exceptional non-bug cases that can be backported: >>>> performance >>>> > >>>>> > > > improvements. >>>> > >>>>> > > > >>>> > >>>>> > > > I realize that there is a ton of subtlety here (even in >>>> terms >>>> > of >>>> > >>>>> which >>>> > >>>>> > > > things are defined as bugs). But I hope we can lay down a >>>> > policy >>>> > >>>>> that >>>> > >>>>> > > gives >>>> > >>>>> > > > everyone the right mindset for common cases and then >>>> discuss >>>> > >>>>> corner >>>> > >>>>> > cases >>>> > >>>>> > > > on-demand in the future. >>>> > >>>>> > > > >>>> > >>>>> > > > [1] http://mesos.apache.org/docume >>>> ntation/latest/versioning/ >>>> > >>>>> > > > >>>> > >>>>> > > >>>> > >>>>> > >>>> > >>>>> >>>> > >>>>> >>>> > >>> >>>> > >> >>>> > > >>>> > >>>> >>> >>> >> >
