Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Thanks Kos. Archived mailing lists come in handy. Many thanks to Apache to
have http://mail-archives.apache.org/mod_mbox/hadoop-general/.
- milind
--
Milind Bhandarkar
mbhandar...@linkedin.com
+1-650-776-3167
On 5/6/11 11:57 PM, "Konstantin Boudnik" wrote:
>Wow! Great compilation, Milind! Very nice to have the sequence of events
>handy.
>
>Thanks,
> Cos
>
>On Fri, May 6, 2011 at 23:55, Milind Bhandarkar
> wrote:
>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>> will try to answer your questions.]
>>
>> Eric,
>>
>> I think the thread
>>
>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>8C
>> 5c999-4680-4684-bc55-a430c40fd...@yahoo-inc.com%3E" will answer your
>> questions. Here is the timeline as I see it:
>>
>> 1. Arun proposes to create a release from the security patchset. Says
>>Doug
>> has proposed this earlier
>>
>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>BD
>> 1dfea.5020...@apache.org%3E April 23, 2010) ("This has been proposed
>> earlier by Doug and did not get far due to concerns about the effect
>>this
>> would have on development on trunk.") (August 24, 2010)
>>
>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>> comment is from Tom White: "I think it would be good to have a shared
>>0.20
>> Apache security branch.
>> Since security isn't in 0.21, and the 0.22 release is a some way off
>> as you mention, this would be useful for folks who want the security
>> features sooner (and want to use an Apache release)."
>>
>> 3. Arun volunteers to create a release (August 30, 2010)
>>
>> 4. Doug reminds Arun. (October 15, 2010)
>>
>> 5. Arun apologizes for not creating a branch because he was busy,
>>because
>> he had a baby. (January 11, 2011)
>>
>> 6. Lots of discussion about what to call it (the release, not the baby,
>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>call
>> your kid 20.100?" ;-).
>>
>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>about
>> something like 20.100 to show that it's a big jump? Anything else?" Jan
>> 12, 2011
>>
>> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
>> 12, 2011.
>>
>> So, as you can see, even if this release is called 0.20.x, the community
>> agreed that these are valuable patches to have, and despite backward
>> incompatibility, still have them in minor release.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandar...@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/6/11 11:14 PM, "Eric Sammer" wrote:
>>
>>>On May 6, 2011, at 4:53 AM, Steve Loughran wrote:
>>>
>>>I understand Eli's concerns that putting stuff in there that hasn't gone
>>>into trunk yet is danger. However, as the team makes no guarantees of
>>>100%
>>>compatibility between releases, I don't think it's critical. It's just
>>>something that needs to be addressed -which can be done after this
>>>release
>>>has shipped.
>>>
>>>
>>>I was under the impression that the community has been extremely strict
>>>about compatibility between minor version bumps in the past. I though
>>>there
>>>were specific guarantees and that was one of the reasons certain
>>>behaviors
>>>have persisted so long.
>>>
>>>Does this mean API changes can be made in minor releases and it can be
>>>made
>>>backward compatible in future releases? That seems very, very counter to
>>>various conversations that have happened in the past. I'm of the mind
>>>that
>>>we should continue to promise what we've always promised and if that's
>>>changing, let's make with the refactoring party!
>>>
>>>Can some PMC'ers clarify this one for me?
>>>
>>>TIA.
>>>Sammer
>>>
>>>
>>>
>>>-Steve
>>
>>
Re: [VOTE] Release candidate 0.20.203.0-rc1
On May 6, 2011, at 11:18 PM, Milind Bhandarkar wrote: > Allen, there are per job limits, and per user limits in this branch. (So, > max capacity of -1 is for the queue, but within the queue, the per user > limits come into picture.) If I remember right, the defaults were based on > a certain assumption of how many users would be on a queue simultaneously. > Of course this would need to be set in the site-specific config. Yes, I'm aware of the changes. What I'm basically saying is that even with those new limits taken into consideration, the math doesn't seem to hold up.
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
Milind:
Thanks for the pointer. I remember this thread. I guess my question
was unrelated to the specific release and more about the general mode
of development under normal release circumstances (ie. do we permit
backward incompatible changes between 0.22.0 and 0.22.1 or is this
something we've allowed just for the 203 release?).
I think it's important to be clear about what the MO is so end users
can plan upgrades appropriately.
Thanks!
Sammer
On May 6, 2011, at 11:52 PM, Milind Bhandarkar wrote:
> [I am not on PMC, but seeing that PMC may be busy with other issues, I
> will try to answer your questions.]
>
> Eric,
>
> I think the thread
> "http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C18C
> 5c999-4680-4684-bc55-a430c40fd...@yahoo-inc.com%3E" will answer your
> questions. Here is the timeline as I see it:
>
> 1. Arun proposes to create a release from the security patchset. Says Doug
> has proposed this earlier
> (http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4BD
> 1dfea.5020...@apache.org%3E April 23, 2010) ("This has been proposed
> earlier by Doug and did not get far due to concerns about the effect this
> would have on development on trunk.") (August 24, 2010)
>
> 2. Lots of +1s, between August 24 to August 30 2010. One particular
> comment is from Tom White: "I think it would be good to have a shared 0.20
> Apache security branch.
> Since security isn't in 0.21, and the 0.22 release is a some way off
> as you mention, this would be useful for folks who want the security
> features sooner (and want to use an Apache release)."
>
> 3. Arun volunteers to create a release (August 30, 2010)
>
> 4. Doug reminds Arun. (October 15, 2010)
>
> 5. Arun apologizes for not creating a branch because he was busy, because
> he had a baby. (January 11, 2011)
>
> 6. Lots of discussion about what to call it (the release, not the baby,
> although I had a good laugh at Patrick Angeles's email: "You're gonna call
> your kid 20.100?" ;-).
>
> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how about
> something like 20.100 to show that it's a big jump? Anything else?" Jan
> 12, 2011
>
> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
> 12, 2011.
>
> So, as you can see, even if this release is called 0.20.x, the community
> agreed that these are valuable patches to have, and despite backward
> incompatibility, still have them in minor release.
>
> - milind
>
> --
> Milind Bhandarkar
> mbhandar...@linkedin.com
> +1-650-776-3167
>
>
>
>
>
>
> On 5/6/11 11:14 PM, "Eric Sammer" wrote:
>
>> On May 6, 2011, at 4:53 AM, Steve Loughran wrote:
>>
>> I understand Eli's concerns that putting stuff in there that hasn't gone
>> into trunk yet is danger. However, as the team makes no guarantees of 100%
>> compatibility between releases, I don't think it's critical. It's just
>> something that needs to be addressed -which can be done after this release
>> has shipped.
>>
>>
>> I was under the impression that the community has been extremely strict
>> about compatibility between minor version bumps in the past. I though
>> there
>> were specific guarantees and that was one of the reasons certain behaviors
>> have persisted so long.
>>
>> Does this mean API changes can be made in minor releases and it can be
>> made
>> backward compatible in future releases? That seems very, very counter to
>> various conversations that have happened in the past. I'm of the mind that
>> we should continue to promise what we've always promised and if that's
>> changing, let's make with the refactoring party!
>>
>> Can some PMC'ers clarify this one for me?
>>
>> TIA.
>> Sammer
>>
>>
>>
>> -Steve
>
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
On May 8, 2011, at 9:50 AM, Eric Sammer wrote: > do we permit > backward incompatible changes between 0.22.0 and 0.22.1 or is this > something we've allowed just for the 203 release? good question. do we allow incompatible (smallish) features to be added to a 20.x release. hoping that they will eventually be put into trunk at a later stage. and if we need a process or something around it, or will just act on good faith that it will occur.
Re: [VOTE] Release candidate 0.20.203.0-rc0
-1 for rc1 I downloaded and ran the test target 3 times. First run failed because my umask is defaulted to 0002, which is a known problem HADOOP-5050 committed to 0.21 but not 0.20. Set umask to 0022 and re-ran test twice. Both resulted in failure. Here is the list of failed tests: [junit] Test org.apache.hadoop.mapred.TestJobTrackerRestart FAILED (timeout) [junit] Test org.apache.hadoop.mapred.TestJobTrackerRestartWithLostTracker FAILED [junit] Test org.apache.hadoop.mapred.TestJobTrackerSafeMode FAILED [junit] Test org.apache.hadoop.mapred.TestMiniMRMapRedDebugScript FAILED [junit] Test org.apache.hadoop.mapred.TestRecoveryManager FAILED [junit] Test org.apache.hadoop.mapred.TestTTMemoryReporting FAILED [junit] Test org.apache.hadoop.mapred.TestTaskTrackerLocalization FAILED [junit] Test org.apache.hadoop.hdfsproxy.TestHdfsProxy FAILED I am in favor of releasing hadoop-0.20.203. And we run a version of this release on a large cluster at eBay. I know it works. I understand the controversy behind it. I regret it hasn't been developed in a true community way. I think it nevertheless adds value to Apache Hadoop. Lets just make sure it passes the tests. Thanks, --Konstantin On Tue, May 3, 2011 at 1:39 AM, Konstantin Shvachko wrote: > I think its a good idea to release hadoop-0.20.203. It moves Apache Hadoop > a step forward. > > Looks like the technical difficulties are resolved now with latest Arun's > commits. > Being a superset of hadoop-0.20.2 it can be considered based on one of the > official Apache releases. > I don't think there was a lack of discussions on the lists about the issues > included in the release candidate. Todd did a thorough review of the entire > security branch. Many developers participated in discussions. > Agreeing with Stack I wish HBase was considered a primary target for Hadoop > support. But it is not realistic to have it in hadoop-0.20.203. > I have some experience running a version of this release candidate on a > large cluster. It works. I would add a couple of patches, which make it run > on Windows for me like HADOOP-7110, HADOOP-7126. But those are not blockers. > > Thanks, > --Konstantin > > > On Mon, May 2, 2011 at 5:12 PM, Ian Holsman wrote: > >> >> On May 3, 2011, at 9:58 AM, Arun C Murthy wrote: >> >> >> >> >> Owen, Suresh and I have committed everything on this list except >> >> HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/ >> >> necessary, I'll check with Cos. Other than that hadoop-0.20.203 now a >> >> superset of hadoop-0.20.2. >> >> >> > >> > Missed adding HADOOP-5759 to that list, I'll check with Amareshwari >> before committing. >> > >> > Arun >> >> Thanks for doing this so fast Arun. >> >> >
Re: Release compatibility was Re: [VOTE] Release candidate 0.20.203.0-rc1
[Mentioning again: I am not on the PMC, and this email contains
non-binding opinions based on my reading the general@hadoop.apache.org
emails.]
It is my understanding that, from the beginning, the 0.20+security was
always treated as an exception to the normal (I.e. Pre-0.20) release
process. (This has been confirmed by the mailing list threads, in which
many of those who are objecting to this release now - stating that it has
violated norms - have consented, actually argued for, breaking the norms.)
For whatever I have read on this mailing list before the vote for this
release, it looked like most of the community agreed that what Yahoo! Had
produced on their own branch, outside of Apache trunk, was important
contribution, and a release based on that would be a good idea, and that a
one-time release should proceed. (After all, whichever organization the
contributors belong to, many seem to indicate that they feel ashamed not
having an Apache release in more than a year.)
>From many emails on this thread, it has been clear to me, that it is a one
time concession given for parting ways from the normal process, and I hope
everyone understands that this is supposed to make Apache Hadoop releases
relevant once again.
So, to cut it short, the 0.20.203 backward incompatibilities etc have no
bearing on the "normal" process, in which no backward incompatibilities
should be allowed in minor releases. To answer your specific question, I
have no reason to believe that 0.22.1 could be backward incompatible with
0.22.0.
- milind
--
Milind Bhandarkar
mbhandar...@linkedin.com
+1-650-776-3167
On 5/7/11 4:50 PM, "Eric Sammer" wrote:
>Milind:
>
>Thanks for the pointer. I remember this thread. I guess my question
>was unrelated to the specific release and more about the general mode
>of development under normal release circumstances (ie. do we permit
>backward incompatible changes between 0.22.0 and 0.22.1 or is this
>something we've allowed just for the 203 release?).
>
>I think it's important to be clear about what the MO is so end users
>can plan upgrades appropriately.
>
>Thanks!
>Sammer
>
>On May 6, 2011, at 11:52 PM, Milind Bhandarkar
>wrote:
>
>> [I am not on PMC, but seeing that PMC may be busy with other issues, I
>> will try to answer your questions.]
>>
>> Eric,
>>
>> I think the thread
>>
>>"http://mail-archives.apache.org/mod_mbox/hadoop-general/201101.mbox/%3C1
>>8C
>> 5c999-4680-4684-bc55-a430c40fd...@yahoo-inc.com%3E" will answer your
>> questions. Here is the timeline as I see it:
>>
>> 1. Arun proposes to create a release from the security patchset. Says
>>Doug
>> has proposed this earlier
>>
>>(http://mail-archives.apache.org/mod_mbox/hadoop-general/201004.mbox/%3C4
>>BD
>> 1dfea.5020...@apache.org%3E April 23, 2010) ("This has been proposed
>> earlier by Doug and did not get far due to concerns about the effect
>>this
>> would have on development on trunk.") (August 24, 2010)
>>
>> 2. Lots of +1s, between August 24 to August 30 2010. One particular
>> comment is from Tom White: "I think it would be good to have a shared
>>0.20
>> Apache security branch.
>> Since security isn't in 0.21, and the 0.22 release is a some way off
>> as you mention, this would be useful for folks who want the security
>> features sooner (and want to use an Apache release)."
>>
>> 3. Arun volunteers to create a release (August 30, 2010)
>>
>> 4. Doug reminds Arun. (October 15, 2010)
>>
>> 5. Arun apologizes for not creating a branch because he was busy,
>>because
>> he had a baby. (January 11, 2011)
>>
>> 6. Lots of discussion about what to call it (the release, not the baby,
>> although I had a good laugh at Patrick Angeles's email: "You're gonna
>>call
>> your kid 20.100?" ;-).
>>
>> 7. Arun proposes to call it 0.20.100: "I'm open to suggestions - how
>>about
>> something like 20.100 to show that it's a big jump? Anything else?" Jan
>> 12, 2011
>>
>> 8. Among others, Eli says: "+1 on 0.20.x (where x is a J > 3)" on Jan
>> 12, 2011.
>>
>> So, as you can see, even if this release is called 0.20.x, the community
>> agreed that these are valuable patches to have, and despite backward
>> incompatibility, still have them in minor release.
>>
>> - milind
>>
>> --
>> Milind Bhandarkar
>> mbhandar...@linkedin.com
>> +1-650-776-3167
>>
>>
>>
>>
>>
>>
>> On 5/6/11 11:14 PM, "Eric Sammer" wrote:
>>
>>> On May 6, 2011, at 4:53 AM, Steve Loughran wrote:
>>>
>>> I understand Eli's concerns that putting stuff in there that hasn't
>>>gone
>>> into trunk yet is danger. However, as the team makes no guarantees of
>>>100%
>>> compatibility between releases, I don't think it's critical. It's just
>>> something that needs to be addressed -which can be done after this
>>>release
>>> has shipped.
>>>
>>>
>>> I was under the impression that the community has been extremely strict
>>> about compatibility between minor version bumps in the past. I though
>>> there
>>> were specific guarantees and that was one of the reasons certain
