Sorry for the last minute request. Can we add HDFS-4858 to the release, please? It solves pretty important bug related to failover. I can commit momentarily if there are no objections.
Thanks, --Konstantin On Mon, Feb 10, 2014 at 4:50 PM, Aaron T. Myers <a...@cloudera.com> wrote: > Just committed a fix for HDFS-5921 to branch-2.3. > > Fire away. > > -- > Aaron T. Myers > Software Engineer, Cloudera > > > On Mon, Feb 10, 2014 at 1:34 PM, Aaron T. Myers <a...@cloudera.com> wrote: > > > OK. I think I should be able to get it in by 6pm PT, thanks to a quick +1 > > from Andrew, but certainly don't let it hold up the train if for some > > reason it takes longer than that. > > > > -- > > Aaron T. Myers > > Software Engineer, Cloudera > > > > > > On Mon, Feb 10, 2014 at 12:04 PM, Arun C Murthy <a...@hortonworks.com > >wrote: > > > >> Looks like we are down to 0 blockers; I'll create rc0 tonight. > >> > >> ATM - Your call, you have until 6pm tonight to get this in. > >> > >> thanks, > >> Arun > >> > >> On Feb 10, 2014, at 11:44 AM, "Aaron T. Myers" <a...@cloudera.com> > wrote: > >> > >> > I just filed an issue for the fact that browsing the FS from the NN is > >> > broken if you have a directory with the sticky bit set: > >> > > >> > https://issues.apache.org/jira/browse/HDFS-5921 > >> > > >> > I didn't set this to be targeted for 2.3 because it doesn't seem like > a > >> > _blocker_ to me, but if we're not going to get 2.3 out today anyway, > I'd > >> > like to put this in. It's a small fix, and since many people have the > >> > sticky bit set on /tmp, they won't be able to browse any of the FS > >> > hierarchy from the NN without this fix. > >> > > >> > -- > >> > Aaron T. Myers > >> > Software Engineer, Cloudera > >> > > >> > > >> > On Fri, Feb 7, 2014 at 12:45 PM, Vinod Kumar Vavilapalli < > >> vino...@apache.org > >> >> wrote: > >> > > >> >> Heres what I've done: > >> >> - Reverted YARN-1493,YARN-1490,YARN-1041, > >> >> YARN-1166,YARN-1566,YARN-1689,YARN-1661 from branch-2.3. > >> >> - Updated YARN's CHANGES.txt in trunk, branch-2 and branch-2.3. > >> >> - Updated these JIRAs to have 2.4 as the fix-version. > >> >> - Compiled branch-2.3. > >> >> > >> >> Let me know if you run into any issues caused by this revert. > >> >> > >> >> Thanks, > >> >> +Vinod > >> >> > >> >> > >> >> On Fri, Feb 7, 2014 at 11:41 AM, Vinod Kumar Vavilapalli < > >> >> vino...@apache.org > >> >>> wrote: > >> >> > >> >>> Haven't heard back from Jian. Reverting the set from branch-2.3 > >> (only). > >> >> Tx > >> >>> for the offline list. > >> >>> > >> >>> +Vinod > >> >>> > >> >>> > >> >>> On Fri, Feb 7, 2014 at 9:08 AM, Alejandro Abdelnur < > t...@cloudera.com > >> >>> wrote: > >> >>> > >> >>>> Vinod, I have the patches to revert most of the JIRAs, the first > >> batch, > >> >>>> I'll send them off line to you. > >> >>>> > >> >>>> Thanks. > >> >>>> > >> >>>> > >> >>>> On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli > >> >>>> <vino...@apache.org>wrote: > >> >>>> > >> >>>>> > >> >>>>> Thanks. please post your findings, Jian wrote this part of the > code > >> >> and > >> >>>>> between him/me, we can take care of those issues. > >> >>>>> > >> >>>>> +1 for going ahead with the revert on branch-2.3. I'll go do that > >> >>>> tomorrow > >> >>>>> morning unless I hear otherwise from Jian. > >> >>>>> > >> >>>>> Thanks, > >> >>>>> +Vinod > >> >>>>> > >> >>>>> > >> >>>>> On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <t...@cloudera.com > > > >> >>>> wrote: > >> >>>>> > >> >>>>>> Hi Vinod, > >> >>>>>> > >> >>>>>> Nothing confidential, > >> >>>>>> > >> >>>>>> * With umanaged AMs I'm seeing the trace I've posted a couple of > >> >> days > >> >>>> ago > >> >>>>>> in YARN-1577 ( > >> >>>>>> > >> >>>>> > >> >>>> > >> >> > >> > https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853 > >> >>>>>> ). > >> >>>>>> > >> >>>>>> * Also, Robert has been digging in Oozie testcases > failing/getting > >> >>>> suck > >> >>>>>> with several token renewer threads, this failures happened > >> >>>> consistently > >> >>>>> at > >> >>>>>> different places around the same testcases (like some file > >> >> descriptors > >> >>>>>> leaking out), reverting YARN-1490 fixes the problem. The > potential > >> >>>> issue > >> >>>>>> with this is that a long running client (oozie) my run into this > >> >>>>> situation > >> >>>>>> thus becoming unstable. > >> >>>>>> > >> >>>>>> *Robert,* mind posting to YARN-1490 the jvm thread dump at the > time > >> >> of > >> >>>>> test > >> >>>>>> hanging? > >> >>>>>> > >> >>>>>> After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to > fix > >> >>>>> issues > >> >>>>>> introduced by them, and we still didn't get them right. > >> >>>>>> > >> >>>>>> Because this, the improvements driven by YARN-1493 & YARN-1490 > seem > >> >>>> that > >> >>>>>> require more work before being stable. > >> >>>>>> > >> >>>>>> IMO, being conservative, we should do 2.3 without them and roll > >> them > >> >>>> with > >> >>>>>> 2.4. If we want to do regular releases we will have to make this > >> >> kind > >> >>>> of > >> >>>>>> calls, else we will start dragging the releases. > >> >>>>>> > >> >>>>>> Sounds like a plan? > >> >>>>>> > >> >>>>>> Thanks. > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli > >> >>>>>> <vino...@apache.org>wrote: > >> >>>>>> > >> >>>>>>> Hey > >> >>>>>>> > >> >>>>>>> I am not against removing them from 2.3 if that is helpful for > >> >>>> progress. > >> >>>>>>> But I want to understand what the issues are before we make that > >> >>>>> decision. > >> >>>>>>> > >> >>>>>>> There is the issue with unmanaged AM that is clearly known and I > >> >> was > >> >>>>>>> thinking of coming to the past two days, but couldn't. What is > >> this > >> >>>> new > >> >>>>>>> issue that we (confidently?) pinned down to YARN-1490? > >> >>>>>>> > >> >>>>>>> Thanks > >> >>>>>>> +Vinod > >> >>>>>>> > >> >>>>>>> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur < > t...@cloudera.com > >> > > >> >>>>> wrote: > >> >>>>>>> > >> >>>>>>>> Thanks Robert, > >> >>>>>>>> > >> >>>>>>>> All, > >> >>>>>>>> > >> >>>> > >> >>>>>>>> So it seems that YARN-1493 and YARN-1490 are introducing > serious > >> >>>>>>>> regressions. > >> >>>>>>>> > >> >>>>>>>> I would propose to revert them and the follow up JIRAs from the > >> >> 2.3 > >> >>>>>>> branch > >> >>>>>>>> and keep working on them on trunk/branch-2 until the are stable > >> (I > >> >>>>> would > >> >>>>>>>> even prefer reverting them from branch-2 not to block a 2.4 if > >> >> they > >> >>>> are > >> >>>>>>> not > >> >>>>>>>> ready in time). > >> >>>>>>>> > >> >>>>>>>> As I've mentioned before, the list of JIRAs to revert were: > >> >>>>>>>> > >> >>>>>>>> YARN-1493 > >> >>>>>>>> YARN-1490 > >> >>>>>>>> YARN-1166 > >> >>>>>>>> YARN-1041 > >> >>>>>>>> YARN-1566 > >> >>>>>>>> > >> >>>>>>>> Plus 2 additional JIRAs committed since my email on this issue > 2 > >> >>>> days > >> >>>>>>> ago: > >> >>>>>>>> > >> >>>>>>>> *YARN-1661 > >> >>>>>>>> *YARN-1689 (not sure if this JIRA is related in functionality > to > >> >> the > >> >>>>>>>> previous ones but it is creating conflicts). > >> >>>>>>>> > >> >>>>>>>> I think we should hold on continuing work on top of something > >> that > >> >>>> is > >> >>>>>>>> broken until the broken stuff is fixed. > >> >>>>>>>> > >> >>>>>>>> Quoting Arun, "Committers - Henceforth, please use extreme > >> caution > >> >>>>> while > >> >>>>>>>> committing to branch-2.3. Please commit *only* blockers to > 2.3." > >> >>>>>>>> > >> >>>>>>>> YARN-1661 & YARN-1689 are not blockers. > >> >>>>>>>> > >> >>>>>>>> Unless there are objections, I'll revert all these JIRAs from > >> >>>>> branch-2.3 > >> >>>>>>>> tomorrow around noon and I'll update fixedVersion in the JIRAs. > >> >>>>>>>> > >> >>>>>>>> I'm inclined to revert them from branch-2 as well. > >> >>>>>>>> > >> >>>>>>>> Thoughts? > >> >>>>>>>> > >> >>>>>>>> Thanks. > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter < > >> >> rkan...@cloudera.com > >> >>>>> > >> >>>>>>> wrote: > >> >>>>>>>> > >> >>>>>>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I > >> >>>> think it > >> >>>>>>> was > >> >>>>>>>>> causing some strange behavior in the Oozie unit tests: > >> >>>>>>>>> > >> >>>>>>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster > >> >> across > >> >>>> all > >> >>>>>>> unit > >> >>>>>>>>> tests in a module. With YARN-1490 we saw that, regardless of > >> >> test > >> >>>>>>> order, > >> >>>>>>>>> the last few tests would timeout waiting for an MR job to > >> finish; > >> >>>> on > >> >>>>>>> slower > >> >>>>>>>>> machines, the entire test suite would timeout. Through some > >> >>>> digging, > >> >>>>> I > >> >>>>>>>>> found that we were getting a ton of "Connection refused" > >> >>>> Exceptions on > >> >>>>>>>>> LeaseRenewer talking to the NN and a few on the AM talking to > >> the > >> >>>> RM. > >> >>>>>>>>> > >> >>>>>>>>> After a bunch of investigation, I found that the problem went > >> >> away > >> >>>>> once > >> >>>>>>>>> YARN-1490 was removed. Though I couldn't figure out the exact > >> >>>>> problem. > >> >>>>>>>>> Even though this occurred in unit tests, it does make me > >> >> concerned > >> >>>>> that > >> >>>>>>> it > >> >>>>>>>>> could indicate some bigger issue in a long-running real > cluster > >> >>>> (where > >> >>>>>>>>> everything isn't running on the same machine) that we haven't > >> >> seen > >> >>>>> yet. > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla < > >> >>>> ka...@cloudera.com> > >> >>>>>>>>> wrote: > >> >>>>>>>>> > >> >>>>>>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it > >> >>>>> shortly. > >> >>>>>>>>> Will > >> >>>>>>>>>> pull it out of branch-2.3 if anyone objects. > >> >>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal < > >> >>>>>>> aagar...@hortonworks.com > >> >>>>>>>>>>> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456. > >> >>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal < > >> >>>>>>>>> aagar...@hortonworks.com > >> >>>>>>>>>>>> wrote: > >> >>>>>>>>>>> > >> >>>>>>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in > 2.3. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one > >> >> disagrees. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur < > >> >>>>>>>>> t...@cloudera.com > >> >>>>>>>>>>>> wrote: > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs > in > >> a > >> >>>> very > >> >>>>>>>>> odd > >> >>>>>>>>>>>>> ways > >> >>>>>>>>>>>>> (to the point it seems un-deterministic). > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert > >> >>>>>>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost > >> >> clean > >> >>>>>>>>>> reverts) > >> >>>>>>>>>>>>> from Hadoop 2.3 branch before doing the release. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> I've verified that after reverting those JIRAs things work > >> >> fine > >> >>>>> with > >> >>>>>>>>>>>>> unmanaged AMs. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> Thanks. > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy < > >> >>>>> a...@hortonworks.com > >> >>>>>>>>>> > >> >>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing > issue. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making > >> >> much > >> >>>>>>>>>>> progress > >> >>>>>>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. > >> >> Any > >> >>>>>>>>>>>>> objections? > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 > covered. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Overall, I'll try get this out in next couple of days if > we > >> >>>> can > >> >>>>>>>>>> clear > >> >>>>>>>>>>>>> the > >> >>>>>>>>>>>>>> list. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> thanks, > >> >>>>>>>>>>>>>> Arun > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy < > >> >>>> a...@hortonworks.com> > >> >>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> An update. Per > >> https://s.apache.org/hadoop-2.3.0-blockerswe > >> >>>>>>>>> are > >> >>>>>>>>>>> now > >> >>>>>>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is > >> >>>> helping > >> >>>>>>>>>> out > >> >>>>>>>>>>>>> with > >> >>>>>>>>>>>>>> the YARN ones. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> thanks, > >> >>>>>>>>>>>>>>> Arun > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>> Arun C. Murthy > >> >>>>>>>>>>>>>> Hortonworks Inc. > >> >>>>>>>>>>>>>> http://hortonworks.com/ > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>> CONFIDENTIALITY NOTICE > >> >>>>>>>>>>>>>> NOTICE: This message is intended for the use of the > >> >>>> individual or > >> >>>>>>>>>>>>> entity to > >> >>>>>>>>>>>>>> which it is addressed and may contain information that is > >> >>>>>>>>>>> confidential, > >> >>>>>>>>>>>>>> privileged and exempt from disclosure under applicable > law. > >> >> If > >> >>>>> the > >> >>>>>>>>>>>>> reader > >> >>>>>>>>>>>>>> of this message is not the intended recipient, you are > >> >> hereby > >> >>>>>>>>>> notified > >> >>>>>>>>>>>>> that > >> >>>>>>>>>>>>>> any printing, copying, dissemination, distribution, > >> >>>> disclosure or > >> >>>>>>>>>>>>>> forwarding of this communication is strictly prohibited. > If > >> >>>> you > >> >>>>>>>>> have > >> >>>>>>>>>>>>>> received this communication in error, please contact the > >> >>>> sender > >> >>>>>>>>>>>>> immediately > >> >>>>>>>>>>>>>> and delete it from your system. Thank You. > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> -- > >> >>>>>>>>>>>>> Alejandro > >> >>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> -- > >> >>>>>>>>>>> CONFIDENTIALITY NOTICE > >> >>>>>>>>>>> NOTICE: This message is intended for the use of the > individual > >> >> or > >> >>>>>>>>> entity > >> >>>>>>>>>> to > >> >>>>>>>>>>> which it is addressed and may contain information that is > >> >>>>>>> confidential, > >> >>>>>>>>>>> privileged and exempt from disclosure under applicable law. > If > >> >>>> the > >> >>>>>>>>> reader > >> >>>>>>>>>>> of this message is not the intended recipient, you are > hereby > >> >>>>> notified > >> >>>>>>>>>> that > >> >>>>>>>>>>> any printing, copying, dissemination, distribution, > disclosure > >> >> or > >> >>>>>>>>>>> forwarding of this communication is strictly prohibited. If > >> you > >> >>>> have > >> >>>>>>>>>>> received this communication in error, please contact the > >> sender > >> >>>>>>>>>> immediately > >> >>>>>>>>>>> and delete it from your system. Thank You. > >> >>>>>>>>>>> > >> >>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> -- > >> >>>>>>>> Alejandro > >> >>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> -- > >> >>>>>>> CONFIDENTIALITY NOTICE > >> >>>>>>> NOTICE: This message is intended for the use of the individual > or > >> >>>>> entity to > >> >>>>>>> which it is addressed and may contain information that is > >> >>>> confidential, > >> >>>>>>> privileged and exempt from disclosure under applicable law. If > the > >> >>>>> reader > >> >>>>>>> of this message is not the intended recipient, you are hereby > >> >>>> notified > >> >>>>> that > >> >>>>>>> any printing, copying, dissemination, distribution, disclosure > or > >> >>>>>>> forwarding of this communication is strictly prohibited. If you > >> >> have > >> >>>>>>> received this communication in error, please contact the sender > >> >>>>> immediately > >> >>>>>>> and delete it from your system. Thank You. > >> >>>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> -- > >> >>>>>> Alejandro > >> >>>> > >> >>>>> > >> >>>>> > >> >>>>> -- > >> >>>>> CONFIDENTIALITY NOTICE > >> >>>>> NOTICE: This message is intended for the use of the individual or > >> >>>> entity to > >> >>>>> which it is addressed and may contain information that is > >> >> confidential, > >> >>>>> privileged and exempt from disclosure under applicable law. If the > >> >>>> reader > >> >>>>> of this message is not the intended recipient, you are hereby > >> notified > >> >>>> that > >> >>>>> any printing, copying, dissemination, distribution, disclosure or > >> >>>>> forwarding of this communication is strictly prohibited. If you > have > >> >>>>> received this communication in error, please contact the sender > >> >>>> immediately > >> >>>>> and delete it from your system. Thank You. > >> >>>>> > >> >>>> > >> >>>> > >> >>>> > >> >>>> -- > >> >>>> Alejandro > >> >>>> > >> >>> > >> >>> > >> >> > >> >> -- > >> >> CONFIDENTIALITY NOTICE > >> >> NOTICE: This message is intended for the use of the individual or > >> entity to > >> >> which it is addressed and may contain information that is > confidential, > >> >> privileged and exempt from disclosure under applicable law. If the > >> reader > >> >> of this message is not the intended recipient, you are hereby > notified > >> that > >> >> any printing, copying, dissemination, distribution, disclosure or > >> >> forwarding of this communication is strictly prohibited. If you have > >> >> received this communication in error, please contact the sender > >> immediately > >> >> and delete it from your system. Thank You. > >> >> > >> > >> -- > >> Arun C. Murthy > >> Hortonworks Inc. > >> http://hortonworks.com/ > >> > >> > >> > >> -- > >> CONFIDENTIALITY NOTICE > >> NOTICE: This message is intended for the use of the individual or entity > >> to > >> which it is addressed and may contain information that is confidential, > >> privileged and exempt from disclosure under applicable law. If the > reader > >> of this message is not the intended recipient, you are hereby notified > >> that > >> any printing, copying, dissemination, distribution, disclosure or > >> forwarding of this communication is strictly prohibited. If you have > >> received this communication in error, please contact the sender > >> immediately > >> and delete it from your system. Thank You. > >> > > > > >
