Share the same concern as Russell here. Not great for the project for everyone to go "private branch" approach.
On Tue, Dec 11, 2012 at 8:33 AM, Russell Jurney <[email protected]>wrote: > Wait. Ack. Do we want everyone to do this? This sounds like fragmentation. > :( > > Russell Jurney twitter.com/rjurney > > > On Dec 10, 2012, at 3:24 PM, Olga Natkovich <[email protected]> wrote: > > > If everybody is using a private branch then > > > > (1) We are not serving a significant part of our community > > (2) There is no motivation to contribute those patches to branches (only > to trunk). > > > > Yahoo has been trying hard to work of the Apache branches but if we > increase the scope of what is going into branches, we will go with private > branch approach as well. > > > > Olga > > > > > > ________________________________ > > From: Julien Le Dem <[email protected]> > > To: Olga Natkovich <[email protected]> > > Cc: "[email protected]" <[email protected]>; Santhosh M S < > [email protected]>; "[email protected]" <[email protected]> > > Sent: Friday, December 7, 2012 3:54 PM > > Subject: Re: Our release process > > > > Here's my criteria for inclusion in a release branch: > > - no new feature. Only bug fixes. > > - The criteria is more about stability than priority. The person/group > > asking for it has a good reason for wanting it in the branch. If > commiters > > think the patch is reasonable and won't make the branch unstable then we > > should check it in. If it breaks something anyway, we revert it. > > > > For what it's worth we (at Twitter) maintain an internal branch where we > > add patches we need and I would suggest anybody that wants to be able to > > make emergency fixes to their own deployment to do the same. We do keep > > that branch as close to apache as we can but it has a few patches that > are > > in trunk only and do not satisfy the no new feature criteria. > > > > What does the PMC think ? > > > > Julien > > > > > > > > > > On Tue, Dec 4, 2012 at 12:46 PM, Olga Natkovich <[email protected] > >wrote: > > > >> I am ok with tests running nightly and reverting patches that cause > >> failures. We used to have that. Does anybody know what happened? Is > anybody > >> volunteering to make it work again? > >> > >> I would like to see specific criteria for what goes into the branch been > >> published (rather than case-by-case). This way each team can decided if > the > >> criteria stringent enough of if they need to run a private branch. > >> > >> Olga > >> > >> ------------------------------ > >> *From:* Santhosh M S <[email protected]> > >> *To:* Julien Le Dem <[email protected]>; "[email protected]" < > >> [email protected]> > >> *Cc:* "[email protected]" <[email protected]> > >> *Sent:* Friday, November 30, 2012 11:46 PM > >> > >> *Subject:* Re: Our release process > >> > >> HI Julien, > >> > >> You are making most of the points that I did on this thread (CI for e2e, > >> not burdening clean e2e prior to every commit for a release branch). The > >> only point on which there is no clear agreement is the definition of a > bug > >> that can be included in a previously released branch. I am fine with a > case > >> by case inclusion. > >> > >> Hi Olga, > >> > >> Are you fine with Julien's proposal as it stands - bugs that are > included > >> will be determined at the time of inclusion instead of doing it now. > >> > >> Santhosh > >> > >> > >> ________________________________ > >> From: Julien Le Dem <[email protected]> > >> To: [email protected]; Santhosh M S <[email protected]> > >> Cc: "[email protected]" <[email protected]> > >> Sent: Friday, November 30, 2012 5:37 PM > >> Subject: Re: Our release process > >> > >> Proposed criteria: > >> - it makes the tests fail. targets test-commit + test + e2e tests > >> - a critical bug is reported in a short time frame (definition of > >> critical not needed as it is rare and can be decided on a case by case > >> basis) > >> > >> That raises another question: what are the existing CI servers running > >> the tests? > >> - the Apache CI runs test-commit and test (is it more stable now?) > >> and not e2e. It would be great if it did. > >> - we have a Jenkins build at Twitter where we run test-commit and > >> test, we could not run e2e easily in our environment. > >> - I understand there's a Yahoo/Hortonworks build (test-commit + test + > e2e > >> ???) > >> > >> Whenever those builds fail we should open or reopen JIRAS and fix it. > >> > >> The time it takes to run the full > >> test suite makes it impractical to > >> run on a desktop/laptop. > >> > >> For the release Pig-0.11.0 we need to get this list of JIRAs down to 0 > >> and publish the jar. > >> > >> > https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+PIG+AND+fixVersion+%3D+%220.11%22+AND+resolution+%3D+Unresolved+ORDER+BY+updated+DESC%2C+due+ASC%2C+priority+DESC > >> > >> Julien > >> > >> On Thu, Nov 29, 2012 at 11:16 PM, Santhosh M S > >> <[email protected]> wrote: > >>> Looks like everyone is interested in having frequent releases - I don't > >> see anyone disagreeing with that. > >>> > >>> Regarding "If a patch > >> makes the release branch unstable, we revert it" - what are the > criteria? > >> If we can't decide on the criteria on this thread (already pretty long) > >> then lets get the release trains going. We can revisit the criteria for > >> inclusion of bug fixes when that happens. > >>> > >>> Santhosh > >>> > >>> > >>> ________________________________ > >>> From: Julien Le Dem <[email protected]> > >>> To: [email protected]; Santhosh M S <[email protected]> > >>> Cc: "[email protected]" <[email protected]> > >>> Sent: > >> Thursday, November 29, 2012 9:45 AM > >>> Subject: Re: Our release process > >>> > >>> The release branch receives only bug fixes. Patch level releases (3rd > >>> version number) are issued out of the release branch and introduce > >>> only bug fixes and no new features. > >>> Deciding whether a patch is applied to the release branch is based on > >>> preserving stability (as Bill said). If a patch makes the release > >>> branch unstable, we revert it. > >>> New features are added to trunk where new major and minor releases will > >> happen. > >>> If we need a new feature out then we make a new minor release. > >>> Doing frequent releases is the industry standard and will resolve > >>> conflicts around what should go in a release branch. > >>> > >>> Making a new release is currently painful *because* we wait so long in > >>> between two releases. Let's fix that. > >>> > >>> Julien > >>> > >>> On Wed, Nov 28, 2012 at > >> 10:09 PM, Santhosh M S > >>> <[email protected]> wrote: > >>>> Since releasing a major version once a month is agressive and we have > >> not released on a quarterly basis, we should allow commits to a released > >> branch to facilitate dot releases. > >>>> > >>>> If we are allowing commits to a released branch, the criteria for > >> inclusion can be created anew or we use the industry standards for > severity > >> (or priority). It could be painful for a few folks but I don't see > better > >> alternatives. > >>>> > >>>> Regarding reverting commits based on e2e tests breaking: > >>>> 1. Who is running the tests? > >>>> 2. How often are they run? > >>>> If we have nightly e2e runs then its easier to catch these errors > >> early. If not the barrier for inclusion is pretty high and time > >> consuming making it harder to develop. > >>>> > >>>> Santhosh > >>>> > >>>> > >>>> ________________________________ > >>>> From: Bill Graham <[email protected]> > >>>> To: [email protected] > >>>> Sent: Wednesday, November 28, 2012 11:39 AM > >>>> Subject: Re: Our release process > >>>> > >>>> I agree releasing often is ideal, but releasing major versions once a > >> month > >>>> would be a bit agressive. > >>>> > >>>> +1 to Olga's initial definition of how Yahoo! determines what goes > into > >> a > >>>> released branch. Basically is something broken without a workaround or > >> is > >>>> there potential silent data loss. Trying to get a more granular > >> definition > >>>> than that (i.e. P1, P2, severity, etc) will be > >> painful. The reality in that > >>>> case is that for whomever is blocked by the bug will consider it a P1. > >>>> > >>>> Fixes need to be relatively low-risk though to keep stability, but > this > >> is > >>>> also subjective. For this I'm in favor of relying on developer and > >> reviewer > >>>> judgement to make that call and I'm +1 to Alan's proposal of rolling > >> back > >>>> patches that break the e2e tests or anything else. > >>>> > >>>> I think our policy should avoid time-based consideration on how many > >>>> quarters away are we from the next major release since that's also > >>>> impossible to quantify. Plus, if the answer to the question is that > >> we're > >>>> more than 1-2 quarters from the next release is "yes" then we should > be > >>>> fixing that release problem. > >>>> > >>>> > >>>> On Wed, Nov 28, 2012 at 10:22 AM, Julien Le Dem <[email protected]> > >> wrote: > >>>> > >>>>> I would really like to see us doing frequent releases (at least once > >>>>> per quarter if not once a month). > >>>>> I think the whole notion of priority or being a "blocker" is > >> subjective. > >>>>> Releasing infrequently pressures us to push more changes than we > would > >>>>> want to the release branch. > >>>>> We should focus on keeping TRUNK stable as well so that it is easier > >>>>> to release and users can do more frequent and smaller upgrades. > >>>>> > >>>>> There should be a small enough number of patches going in the release > >>>>> branch so that we can get agreement on whether we check them in or > >>>>> not. > >>>>> I like Alan's proposal of reverting quickly when there's a problem. > >>>>> Again, this becomes less of a problem if we release more > >> often. > >>>>> > >>>>> Which leads me to my next question: what are the next steps for > >>>>> releasing pig 0.11 ? > >>>>> > >>>>> Julien > >>>>> > >>>>> On Tue, Nov 27, 2012 at 10:22 PM, Santhosh M S > >>>>> <[email protected]> wrote: > >>>>>> Hi Olga, > >>>>>> > >>>>>> For a moment, I will move away from P1 and P2 which are related to > >>>>> priorities and use the Severity definitions. > >>>>>> > >>>>>> The standard bugzilla definitions for severity are: > >>>>>> > >>>>>> Blocker - Blocks development and/or testing work. > >>>>>> Critical - Crashes, loss of data, severe memory leak. > >>>>>> Major - Major loss of function. > >>>>>> > >>>>>> I am > >> skipping the other levels (normal, minor and trivial) for this > >>>>> discussion. > >>>>>> > >>>>>> Coming back to priorities, the proposed definitions map P1 to > Blocker > >>>>> and Critical. I am proposing mapping P2 to Major even when there are > >> known > >>>>> workarounds. We are doing this since JIRA does not have severity by > >> default > >>>>> (see: > >> https://confluence.atlassian.com/pages/viewpage.action?pageId=192840 > >>>>> ) > >>>>>> > >>>>>> I am proposing that P2s be included in the released branch only when > >>>>> trunk or unreleased versions are known to be backward incompatible or > >> if > >>>>> the release is more than a quarter (or two) away. > >>>>>> > >>>>>> Thanks, > >>>>>> Santhosh > >>> > >>>>>> ________________________________ > >>>>>> From: Olga Natkovich <[email protected]> > >>>>>> To: "[email protected]" <[email protected]>; Santhosh M S < > >>>>> [email protected]> > >>>>>> Sent: Tuesday, November 27, 2012 10:41 AM > >>>>>> Subject: Re: Our release process > >>>>>> > >>>>>> Hi Santhosh, > >>>>>> > >>>>>> What is your definition of P2s? > >>>>>> > >>>>>> Olga > >>>>>> > >>>>>> > >>>>>> ----- Original > >> Message ----- > >>>>>> From: Santhosh M S <[email protected]> > >>>>>> To: "[email protected]" <[email protected]>; Olga Natkovich < > >>>>> [email protected]> > >>>>>> Cc: > >>>>>> Sent: Monday, November 26, 2012 11:49 PM > >>>>>> Subject: Re: Our release process > >>>>>> > >>>>>> Hi Olga, > >>>>>> > >>>>>> I agree that we cannot guarantee backward compatibility upfront. > With > >>>>> that knowledge, I am proposing a small modification to your proposal. > >>> > >>>>>> 1. If the trunk or unreleased version is known to be backwards > >>>>> compatible then only P1 issues go into the released branch. > >>>>>> 2. If the the trunk or unreleased version is known to be backwards > >>>>> incompatible or the release is a long ways off (two quarters?) then > we > >>>>> should allow for dot releases on the branch, i.e., P1 and P2 issues. > >>>>>> > >>>>>> I am hoping that should provide an incentive for users to move to a > >>>>> higher release and at the same time allow developers to fix issues of > >>>>> significance without impacting stability. > >>>>>> > >>>>>> Thanks, > >>>>>> Santhosh > >>>>>> > >>>>>> > >>>>>> ________________________________ > >>>>>> From: Olga Natkovich <[email protected]> > >>>>>> To: "[email protected]" <[email protected]> > >>>>>> Sent: Monday, November 26, 2012 9:38 AM > >>>>>> Subject: Re: Our release process > >>>>>> > >>>>>> Hi Santhosh, > >>>>>> > >>>>>> I understand the compatibility issue though I am not sure we can > >>>>> guarantee it for all releases upfront but agree that we should make > an > >>>>> effort. > >>>>>> > >>>>>> On the e2e tests, part of the proposal is only do make P1 type of > >>>>> changes to the branch after the initial release so they should be > rare. > >>>>>> > >>>>>> Olga > >>> > >>>>>> ________________________________ > >>>>>> From: Santhosh M S <[email protected]> > >>>>>> To: Olga Natkovich <[email protected]>; "[email protected]" < > >>>>> [email protected]> > >>>>>> Sent: Monday, November 26, 2012 12:00 AM > >>>>>> Subject: Re: Our release process > >>>>>> > >>>>>> > >>>>>> It takes too long to run. If the e2e tests are run every night or a > >>>>> reasonable timeframe then it will reduce the barrier for submitting > >>>>> patches. The context for this: > >> the reluctance of folks to move to a higher > >>>>> version when the higher version is not backward compatible. > >>>>>> > >>>>>> Santhosh > >>>>>> > >>>>>> > >>>>>> ________________________________ > >>>>>> From: Olga Natkovich <[email protected]> > >>>>>> To: "[email protected]" <[email protected]>; Santhosh M S < > >>>>> [email protected]> > >>>>>> Sent: Sunday, November 25, 2012 5:56 PM > >>>>>> Subject: Re: Our release process > >>>>>> > >>>>>> Hi > >> Santhosh, > >>>>>> > >>>>>> Can you clarify why running e2e tests on every checking is a > problem? > >>>>>> > >>>>>> Olga > >>>>>> > >>>>>> > >>>>>> ________________________________ > >>>>>> From: Santhosh M S <[email protected]> > >>>>>> To: "[email protected]" <[email protected]> > >>>>>> Sent: Monday, November 19, 2012 3:48 PM > >>>>>> Subject: Re: Our release process > >>>>>> > >>>>>> The push for an upgrade will work only if the higher release is > >> backward > >>>>> compatible with the lower release. If not, folks will tend to use > >> private > >>>>> branches. Having a stable branch on a large deployment is a good > >> indicator > >>>>> of stability. However, please note that there have been instances > where > >>>>> some releases were never adopted. I will be extremely careful in > >> applying > >>>>> the rule of > >>>>>> running e2e tests for every commit to a released branch. > >>>>>> > >>>>>> If we release every quarter (hopefully) and preserve backward > >>>>> compatibility then I am +1 to the proposal. If the backward > >> compatibility > >>>>> is not preserved then I am -1 for having to run e2e for every commit > >> to a > >>>>> released branch. > >>>>>> > >>>>>> Santhosh > >>>>>> > >>>>>> > >>>>>> ________________________________ > >>>>>> From: Jonathan Coveney <[email protected]> > >>>>>> To: "[email protected]" <[email protected]> > >>>>>> Sent: Tuesday, November 6, 2012 6:34 PM > >>>>>> Subject: Re: Our release process > >>>>>> > >>>>>> I think it might be good to clarify (for me) a couple of cases: > >>>>>> > >>>>>> 1. we have branched a new release > >>>>>> 2. an existing release > >>>>>> > >>>>>> The way I understand things, in the case of 1, we have > >>>>>> a backlog of patches > >>>>>> (not all of which are P1 bugs), and that's ok. If a new bad bug > >> comes in > >>>>>> (the subject of debate here), then it goes in anyway (and in some > >> cases, > >>>>>> would go into 0.9 etc). > >>>>>> > >>>>>> Olga is saying that for existing release (0.9, 0.10), we should only > >>>>> commit > >>>>>> P1 bug fixes there. This makes sense to me, as we're fixing the > >> "official > >>>>>> release" in place. > >>>>>> > >>>>>> IMHO, this would encourage people to use newer release (as this is > >> where > >>>>>> the latest and greatest stuff is, including non-critical bug fixes). > >>>>> Olga's > >>>>>> criteria is a pretty clear barrier for inclusion into these > releases. > >>>>> With > >>>>>> old releases, I think the key is really that they keep doing what > >> they > >>>>> have > >>>>>> always done. Most bugs are well understood by now, and the ones that > >>>>> aren't > >>>>>> will no doubt be P1. > >>> I'm not decided (thus no formal +1 or whatnot), but Olga's point seems > >>>>>> pretty reasonable to me, especially given that trunk has pretty > >>>>>> liberal > >>>>>> development. Once it gets tidied up, I can understand not wanting to > >>>>> jostle > >>>>>> it. > >>>>>> > >>>>>> > >>>>>> 2012/11/5 Alan Gates <[email protected]> > >>>>>> > >>>>>>> Jonathan, for clarity, are you saying you agree that we should only > >> put > >>>>>>> bug fixes in branches or we should only put high priority bug fixes > >> in > >>>>>>> branches? I think we all agree on the former, but there appear to > >> be > >>>>>>> different views on the latter. > >>>>>>> > >>>>>>> Alan. > >>>> > >>>>>>> On Nov 5, 2012, at 4:53 PM, Jonathan Coveney wrote: > >>>>>>> > >>>>>>>> This seems to make sense to me. People can always back-port > >> features, > >>>>> and > >>>>>>>> this encourages them to use the newer ones. It also means we will > >> be > >>>>> more > >>>>>>>> rigorous about stability, which is good as it is a big plus for > >> Pig. I > >>>>>>>> think for older branches, stability trumps features in a big way. > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>> 2012/11/5 Gianmarco De Francisci Morales <[email protected]> > >>>>>>>> > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> On Mon, Nov 5, 2012 at 10:48 AM, Olga > >> Natkovich < > >>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>>> Hi Gianmarco, > >>>>>>>>>> > >>>>>>>>>> Thanks for your comments. Here is a little more information. > >>>>>>>>>> > >>>>>>>>>> At Yahoo, we consider the following issues to be P1: > >>>>>>>>>> > >>>>>>>>>> (1) Bugs that cause wrong results being produced silently > >>>>>>>>>> (2) Bugs that cause failures with no easy workaround > >>>>>>>>> > >>>>>>>>> Thanks Olga, now I get what you mean. > >>>>>>>>> I don't have a strong opinion on > >> this. > >>>>>>>>> On one hand I see why you don't want to put too many patches in > >> the > >>>>>>>>> branches in order to keep things stable. > >>>>>>>>> On the other hand when we do a 0.10.x release with x>0 the users > >>>>> would > >>>>>>>>> like to have as many bugs fixed as possible. > >>>>>>>>> > >>>>>>>>>> Regarding tests. I would suggest we have different rules for > >> trunk > >>>>> and > >>>>>>>>> branches: > >>>>>>>>>> > >>>>>>>>>> (1) For branches, I think we should run the full regression > >> suite > >>>>>>>>> (including e2e) prior to commit. This way we can ensure branch > >>>>> stability > >>>>>>>>> and, as number of patches should be small, will not be a burden > >>>>> > >>>>>>> (2) For trunk, we can go with test-commit only and fix things > >>>>> quickly > >>>>>>>>> when things break. > >>>>>>>>> > >>>>>>>>> I think this makes sense. +1 > >>>>>>>>> > >>>>>>>>>> Olga > >>>>>>> > >>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> -- > >>>>>>>>> Gianmarco > >>>> > >>>> > >>>> > >>>> -- > >>>> *Note that I'm no longer using my Yahoo! email address. Please email > me > >> at > >>>> [email protected] going forward.* > >> >
