Re: hadoop-2.5 - June end?
Hi Karthik, Regarding YARN-1492 (shared cache), it should not hold up 2.5 if it is not there. If it's not there, it's not there. Also, FYI, since it turns out the changes are not too large and should not have a lot of conflicts, we're thinking of not using a separate branch. Regards, Sangjin On Fri, Jun 27, 2014 at 4:23 PM, Karthik Kambatla wrote: > Spoke to Vinod offline. The tentative plan is cut branch-2.5 middle of next > week and work towards getting the remaining blockers in as soon as > possible. > > > On Fri, Jun 27, 2014 at 12:56 PM, Karthik Kambatla > wrote: > > > Including YARN-1695 makes sense, just made it a blocker to ensure we get > > it in. YARN-1492 - would be great if we get it in, but I am not sure how > > feasible it is. Given it is being developed on a branch, there is also > the > > merge to trunk and branch-2 overhead. > > > > I would prefer branching soon so we can work towards stabilizing the > > branch. People can continue to commit blockers to branch-2.5. > > > > > > > > > > On Fri, Jun 27, 2014 at 12:17 PM, Vinod Kumar Vavilapalli < > > vino...@apache.org> wrote: > > > >> Like I mentioned before, I'd like to get RM-webservices : YARN-1695 > >> included. May be we can wait for the branch till that point of time. > >> > >> I am also pushing for Shared Cache: YARN-1492 reviews. Again, like I > said > >> before, we can take it as we go. > >> > >> +Vinod > >> Hortonworks Inc. > >> http://hortonworks.com/ > >> > >> > >> On Thu, Jun 26, 2014 at 12:08 AM, Karthik Kambatla > >> wrote: > >> > >> > Hi folks, > >> > > >> > For 2.5, looks like there are only 6 JIRAs marked Blocker and another > 19 > >> > Critical ones [1]. I guess it makes sense to cut a branch for 2.5 and > >> work > >> > on getting the remaining items in. > >> > > >> > I ll go ahead and create the 2.5 branch in the next couple of days. I > >> hope > >> > http://wiki.apache.org/hadoop/HowToReleasePostMavenization is the > right > >> > steps to follow. > >> > > >> > [1] JIRA Query: project in (Hadoop, HDFS, YARN, MapReduce) AND > >> resolution = > >> > Unresolved AND "Target Version/s" = 2.5.0 AND priority in (Blocker) > >> > > >> > > >> > On Mon, Jun 23, 2014 at 8:48 PM, Vinod Kumar Vavilapalli < > >> > vino...@apache.org > >> > > wrote: > >> > > >> > > I started reviewing YARN-1492 and synced up offline with Sangjin and > >> > Chris > >> > > who are leading this. I proposed that we should get the reviews and > >> > commits > >> > > going as they go and then mark the feature alpha, beta or stable > >> > depending > >> > > on how far we are ahead with the feature and how it aligns with the > >> > release > >> > > in progress. > >> > > > >> > > +Vinod > >> > > > >> > > On Jun 23, 2014, at 2:30 PM, Sangjin Lee wrote: > >> > > > >> > > > It would be great if we can consider the shared cache (YARN-1492) > as > >> > part > >> > > > of 2.5. It obviously will depend on the reviews from folks, but > the > >> > > feature > >> > > > itself should be reasonably solid. > >> > > > > >> > > > Thanks, > >> > > > Sangjin > >> > > > > >> > > > > >> > > > On Mon, Jun 23, 2014 at 2:24 PM, Arun C Murthy < > a...@hortonworks.com > >> > > >> > > wrote: > >> > > > > >> > > >> Thanks Karthik! > >> > > >> > >> > > >> I've updated https://wiki.apache.org/hadoop/Roadmap with > features > >> > which > >> > > >> are very close to completion. Let's see if that makes sense and > if > >> we > >> > > get > >> > > >> any further feedback. > >> > > >> > >> > > >> Arun > >> > > >> > >> > > >> On Jun 23, 2014, at 2:09 PM, Karthik Kambatla < > ka...@cloudera.com> > >> > > wrote: > >> > > >> > >> > > >>> I can pick up the RM duties for 2.5. If I run into any HDFS > >> doubts, I > >> > > >> might > >> > > >>> need some help from someone more familiar with HDFS. > >> > > >>> > >> > > >>> > >> > > >>> On Mon, Jun 23, 2014 at 12:07 PM, Arun C Murthy < > >> a...@hortonworks.com > >> > > > >> > > >> wrote: > >> > > >>> > >> > > Folks, > >> > > > >> > > I'd appreciate some help here. Due to family reasons (all good > >> > ones), > >> > > I'll be away for a couple of weeks. Can someone else pick up > the > >> RM > >> > > >> duties > >> > > for hadoop-2.5? Maybe Andrew since he's expressed interest in > the > >> > > past? > >> > > >> I > >> > > will pick up the thread again for hadoop-2.6, but I don't want > to > >> > > block > >> > > hadoop-2.5 due to my non-availability. > >> > > > >> > > thanks, > >> > > Arun > >> > > > >> > > On Jun 9, 2014, at 9:39 AM, Arun C Murthy > > >> > > wrote: > >> > > > >> > > > Folks, > >> > > > > >> > > > As you can see from the Roadmap wiki, it looks like several > >> items > >> > are > >> > > still a bit away from being ready. > >> > > > > >> > > > I think rather than wait for them, it will be useful to create > >> an > >> > > intermediate release (2.5) this month - I think ATS security is > >> > pretty > >> > > close, so we can
Re: hadoop-2.5 - June end?
Spoke to Vinod offline. The tentative plan is cut branch-2.5 middle of next week and work towards getting the remaining blockers in as soon as possible. On Fri, Jun 27, 2014 at 12:56 PM, Karthik Kambatla wrote: > Including YARN-1695 makes sense, just made it a blocker to ensure we get > it in. YARN-1492 - would be great if we get it in, but I am not sure how > feasible it is. Given it is being developed on a branch, there is also the > merge to trunk and branch-2 overhead. > > I would prefer branching soon so we can work towards stabilizing the > branch. People can continue to commit blockers to branch-2.5. > > > > > On Fri, Jun 27, 2014 at 12:17 PM, Vinod Kumar Vavilapalli < > vino...@apache.org> wrote: > >> Like I mentioned before, I'd like to get RM-webservices : YARN-1695 >> included. May be we can wait for the branch till that point of time. >> >> I am also pushing for Shared Cache: YARN-1492 reviews. Again, like I said >> before, we can take it as we go. >> >> +Vinod >> Hortonworks Inc. >> http://hortonworks.com/ >> >> >> On Thu, Jun 26, 2014 at 12:08 AM, Karthik Kambatla >> wrote: >> >> > Hi folks, >> > >> > For 2.5, looks like there are only 6 JIRAs marked Blocker and another 19 >> > Critical ones [1]. I guess it makes sense to cut a branch for 2.5 and >> work >> > on getting the remaining items in. >> > >> > I ll go ahead and create the 2.5 branch in the next couple of days. I >> hope >> > http://wiki.apache.org/hadoop/HowToReleasePostMavenization is the right >> > steps to follow. >> > >> > [1] JIRA Query: project in (Hadoop, HDFS, YARN, MapReduce) AND >> resolution = >> > Unresolved AND "Target Version/s" = 2.5.0 AND priority in (Blocker) >> > >> > >> > On Mon, Jun 23, 2014 at 8:48 PM, Vinod Kumar Vavilapalli < >> > vino...@apache.org >> > > wrote: >> > >> > > I started reviewing YARN-1492 and synced up offline with Sangjin and >> > Chris >> > > who are leading this. I proposed that we should get the reviews and >> > commits >> > > going as they go and then mark the feature alpha, beta or stable >> > depending >> > > on how far we are ahead with the feature and how it aligns with the >> > release >> > > in progress. >> > > >> > > +Vinod >> > > >> > > On Jun 23, 2014, at 2:30 PM, Sangjin Lee wrote: >> > > >> > > > It would be great if we can consider the shared cache (YARN-1492) as >> > part >> > > > of 2.5. It obviously will depend on the reviews from folks, but the >> > > feature >> > > > itself should be reasonably solid. >> > > > >> > > > Thanks, >> > > > Sangjin >> > > > >> > > > >> > > > On Mon, Jun 23, 2014 at 2:24 PM, Arun C Murthy > > >> > > wrote: >> > > > >> > > >> Thanks Karthik! >> > > >> >> > > >> I've updated https://wiki.apache.org/hadoop/Roadmap with features >> > which >> > > >> are very close to completion. Let's see if that makes sense and if >> we >> > > get >> > > >> any further feedback. >> > > >> >> > > >> Arun >> > > >> >> > > >> On Jun 23, 2014, at 2:09 PM, Karthik Kambatla >> > > wrote: >> > > >> >> > > >>> I can pick up the RM duties for 2.5. If I run into any HDFS >> doubts, I >> > > >> might >> > > >>> need some help from someone more familiar with HDFS. >> > > >>> >> > > >>> >> > > >>> On Mon, Jun 23, 2014 at 12:07 PM, Arun C Murthy < >> a...@hortonworks.com >> > > >> > > >> wrote: >> > > >>> >> > > Folks, >> > > >> > > I'd appreciate some help here. Due to family reasons (all good >> > ones), >> > > I'll be away for a couple of weeks. Can someone else pick up the >> RM >> > > >> duties >> > > for hadoop-2.5? Maybe Andrew since he's expressed interest in the >> > > past? >> > > >> I >> > > will pick up the thread again for hadoop-2.6, but I don't want to >> > > block >> > > hadoop-2.5 due to my non-availability. >> > > >> > > thanks, >> > > Arun >> > > >> > > On Jun 9, 2014, at 9:39 AM, Arun C Murthy >> > > wrote: >> > > >> > > > Folks, >> > > > >> > > > As you can see from the Roadmap wiki, it looks like several >> items >> > are >> > > still a bit away from being ready. >> > > > >> > > > I think rather than wait for them, it will be useful to create >> an >> > > intermediate release (2.5) this month - I think ATS security is >> > pretty >> > > close, so we can ship that. I'm thinking of creating hadoop-2.5 >> by >> > end >> > > >> of >> > > the month, with a branch a couple of weeks prior. >> > > > >> > > > Thoughts? >> > > > >> > > > thanks, >> > > > Arun >> > > > >> > > >> > > >> > > >> > > -- >> > > CONFIDENTIALITY NOTICE >> > > NOTICE: This message is intended for the use of the individual or >> > > >> entity to >> > > which it is addressed and may contain information that is >> > > confidential, >> > > privileged and exempt from disclosure under applicable law. If >> the >> > > >> reader >> > > of this message is not the intended recipient, you are hereby >> > notified >> > > >> t
Re: [VOTE] Release Apache Hadoop 2.4.1
+1 - Verified signatures and digests - Built from source, installed on single-node cluster and ran some sample jobs Jason On 06/21/2014 01:51 AM, Arun C Murthy wrote: Folks, I've created another release candidate (rc1) for hadoop-2.4.1 based on the feedback that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/
Re: [VOTE] Release Apache Hadoop 2.4.1
> What's in branch-2.4.1 doesn't currently match what's in this RC, but there is a tag that matches, right? Else we need to fix that. On Fri, Jun 27, 2014 at 3:26 PM, Aaron T. Myers wrote: > That's fine by me. Like I said, assuming that rc1 does indeed include the > fix in HDFS-6527, and not the revert, then rc1 should be functionally > correct. What's in branch-2.4.1 doesn't currently match what's in this RC, > but if that doesn't bother anyone else then I won't lose any sleep over it. > > -- > Aaron T. Myers > Software Engineer, Cloudera > > > On Jun 27, 2014, at 3:04 PM, "Arun C. Murthy" > wrote: > > > > Aaron, > > > > Since the amend was just to the test, I'll keep this RC as-is. > > > > I'll also comment on jira. > > > > thanks, > > Arun > > > > > > > >> On Jun 27, 2014, at 2:40 PM, "Aaron T. Myers" wrote: > >> > >> I'm -0 on rc1. > >> > >> Note the latest discussion on HDFS-6527 which first resulted in that > patch > >> being reverted from branch-2.4.1 because it was believed it wasn't > >> necessary, and then some more discussion which indicates that in fact > the > >> patch for HDFS-6527 should be included in 2.4.1, but with a slightly > >> different test case. > >> > >> I believe that rc1 was actually created after the first backport of > >> HDFS-6527, but before the revert, so rc1 should be functionally correct, > >> but the test case is not quite correct in rc1, and I believe that rc1 > does > >> not currently reflect the actual tip of branch-2.4.1. I'm not going to > >> consider this a deal-breaker, but seems like we should probably clean > it up. > >> > >> To get this all sorted out properly, if we wanted to, I believe we > should > >> do another backport of HDFS-6527 to branch-2.4.1 including only the > amended > >> test case, and create a new RC from that point. > >> > >> Best, > >> Aaron > >> > >> -- > >> Aaron T. Myers > >> Software Engineer, Cloudera > >> > >> > >>> On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy > wrote: > >>> > >>> Folks, > >>> > >>> I've created another release candidate (rc1) for hadoop-2.4.1 based on > the > >>> feedback that I would like to push out. > >>> > >>> The RC is available at: > >>> http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 > >>> The RC tag in svn is here: > >>> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 > >>> > >>> The maven artifacts are available via repository.apache.org. > >>> > >>> Please try the release and vote; the vote will run for the usual 7 > days. > >>> > >>> thanks, > >>> Arun > >>> > >>> > >>> > >>> -- > >>> Arun C. Murthy > >>> Hortonworks Inc. > >>> http://hortonworks.com/hdp/ > >>> > >>> > >>> > >>> -- > >>> CONFIDENTIALITY NOTICE > >>> NOTICE: This message is intended for the use of the individual or > entity to > >>> which it is addressed and may contain information that is confidential, > >>> privileged and exempt from disclosure under applicable law. If the > reader > >>> of this message is not the intended recipient, you are hereby notified > that > >>> any printing, copying, dissemination, distribution, disclosure or > >>> forwarding of this communication is strictly prohibited. If you have > >>> received this communication in error, please contact the sender > immediately > >>> and delete it from your system. Thank You. > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > -- Alejandro
Re: [VOTE] Release Apache Hadoop 2.4.1
That's fine by me. Like I said, assuming that rc1 does indeed include the fix in HDFS-6527, and not the revert, then rc1 should be functionally correct. What's in branch-2.4.1 doesn't currently match what's in this RC, but if that doesn't bother anyone else then I won't lose any sleep over it. -- Aaron T. Myers Software Engineer, Cloudera > On Jun 27, 2014, at 3:04 PM, "Arun C. Murthy" wrote: > > Aaron, > > Since the amend was just to the test, I'll keep this RC as-is. > > I'll also comment on jira. > > thanks, > Arun > > > >> On Jun 27, 2014, at 2:40 PM, "Aaron T. Myers" wrote: >> >> I'm -0 on rc1. >> >> Note the latest discussion on HDFS-6527 which first resulted in that patch >> being reverted from branch-2.4.1 because it was believed it wasn't >> necessary, and then some more discussion which indicates that in fact the >> patch for HDFS-6527 should be included in 2.4.1, but with a slightly >> different test case. >> >> I believe that rc1 was actually created after the first backport of >> HDFS-6527, but before the revert, so rc1 should be functionally correct, >> but the test case is not quite correct in rc1, and I believe that rc1 does >> not currently reflect the actual tip of branch-2.4.1. I'm not going to >> consider this a deal-breaker, but seems like we should probably clean it up. >> >> To get this all sorted out properly, if we wanted to, I believe we should >> do another backport of HDFS-6527 to branch-2.4.1 including only the amended >> test case, and create a new RC from that point. >> >> Best, >> Aaron >> >> -- >> Aaron T. Myers >> Software Engineer, Cloudera >> >> >>> On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy >>> wrote: >>> >>> Folks, >>> >>> I've created another release candidate (rc1) for hadoop-2.4.1 based on the >>> feedback that I would like to push out. >>> >>> The RC is available at: >>> http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 >>> The RC tag in svn is here: >>> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 >>> >>> The maven artifacts are available via repository.apache.org. >>> >>> Please try the release and vote; the vote will run for the usual 7 days. >>> >>> thanks, >>> Arun >>> >>> >>> >>> -- >>> Arun C. Murthy >>> Hortonworks Inc. >>> http://hortonworks.com/hdp/ >>> >>> >>> >>> -- >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity to >>> which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
Re: Moving to JDK7, JDK8 and new major releases
+1 to making 2.6 the last JDK6 release. If we want, 2.7 could be a parallel release or one soon after 2.6. We could upgrade other dependencies that require JDK7 as well. On Fri, Jun 27, 2014 at 3:01 PM, Arun C. Murthy wrote: > Thanks everyone for the discussion. Looks like we have come to a pragmatic > and progressive conclusion. > > In terms of execution of the consensus plan, I think a little bit of > caution is in order. > > Let's give downstream projects more of a runway. > > I propose we inform HBase, Pig, Hive etc. that we are considering making > 2.6 (not 2.5) the last JDK6 release and solicit their feedback. Once they > are comfortable we can pull the trigger in 2.7. > > thanks, > Arun > > > > On Jun 27, 2014, at 11:34 AM, Karthik Kambatla > wrote: > > > > As someone else already mentioned, we should announce one future release > > (may be, 2.5) as the last JDK6-based release before making the move to > JDK7. > > > > I am comfortable calling 2.5 the last JDK6 release. > > > > > > On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang > > wrote: > > > >> Hi all, responding to multiple messages here, > >> > >> Arun, thanks for the clarification regarding MR classpaths. It sounds > like > >> the story there is improved and still improving. > >> > >> However, I think we still suffer from this at least on the HDFS side. We > >> have a single JAR for all of HDFS, and our clients need to have all the > fun > >> deps like Guava on the classpath. I'm told Spark sticks a newer Guava at > >> the front of the classpath and the HDFS client still works okay, but > this > >> is more happy coincidence than anything else. While we're leaking deps, > >> we're in a scary situation. > >> > >> API compat to me means that an app should be able to run on a new minor > >> version of Hadoop and not have anything break. MAPREDUCE-4421 sounds > like > >> it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what > >> should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs > and > >> have nothing break. If we muck with the classpath, my understanding is > that > >> this could break. > >> > >> Owen, bumping the minimum JDK version in a minor release like this > should > >> be a one-time exception as Tucu stated. A number of people have pointed > out > >> how painful a forced JDK upgrade is for end users, and it's not > something > >> we should be springing on them in a minor release unless we're *very* > >> confident like in this case. > >> > >> Chris, thanks for bringing up the ecosystem. For CDH5, we standardized > on > >> JDK7 across the CDH stack, so I think that's an indication that most > >> ecosystem projects are ready to make the jump. Is that sufficient in > your > >> mind? > >> > >> For the record, I'm also +1 on the Tucu plan. Is it too late to do this > for > >> 2.5? I'll offer to help out with some of the mechanics. > >> > >> Thanks, > >> Andrew > >> > >> On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth < > cnaur...@hortonworks.com> > >> wrote: > >> > >>> I understood the plan for avoiding JDK7-specific features in our code, > >> and > >>> your suggestion to add an extra Jenkins job is a great way to guard > >> against > >>> that. The thing I haven't seen discussed yet is how downstream > projects > >>> will continue to consume our built artifacts. If a downstream project > >>> upgrades to pick up a bug fix, and the jar switches to 1.7 class files, > >> but > >>> their project is still building with 1.6, then it would be a nasty > >>> surprise. > >>> > >>> These are the options I see: > >>> > >>> 1. Make sure all other projects upgrade first. This doesn't sound > >>> feasible, unless all other ecosystem projects have moved to JDK7 > already. > >>> If not, then waiting on a single long pole project would hold up our > >>> migration indefinitely. > >>> > >>> 2. We switch to JDK7, but run javac with -target 1.6 until the whole > >>> ecosystem upgrades. I find this undesirable, because in a certain > sense, > >>> it still leaves a bit of 1.6 lingering in the project. (I'll assume > that > >>> end-of-life for JDK6 also means end-of-life for the 1.6 bytecode > format.) > >>> > >>> 3. Just declare a clean break on some version (your earlier email said > >> 2.5) > >>> and start publishing artifacts built with JDK7 and no -target option. > >>> Overall, this is my preferred option. However, as a side effect, this > >>> sets us up for longer-term maintenance and patch releases off of the > 2.4 > >>> branch if a downstream project that's still on 1.6 needs to pick up a > >>> critical bug fix. > >>> > >>> Of course, this is all a moot point if all the downstream ecosystem > >>> projects have already made the switch to JDK7. I don't know the status > >> of > >>> that off the top of my head. Maybe someone else out there knows? If > >> not, > >>> then I expect I can free up enough in a few weeks to volunteer for > >> tracking > >>> down that information. > >>> > >>> Chris Nauroth > >>> Hortonworks >
Re: [VOTE] Release Apache Hadoop 2.4.1
Aaron, Since the amend was just to the test, I'll keep this RC as-is. I'll also comment on jira. thanks, Arun > On Jun 27, 2014, at 2:40 PM, "Aaron T. Myers" wrote: > > I'm -0 on rc1. > > Note the latest discussion on HDFS-6527 which first resulted in that patch > being reverted from branch-2.4.1 because it was believed it wasn't > necessary, and then some more discussion which indicates that in fact the > patch for HDFS-6527 should be included in 2.4.1, but with a slightly > different test case. > > I believe that rc1 was actually created after the first backport of > HDFS-6527, but before the revert, so rc1 should be functionally correct, > but the test case is not quite correct in rc1, and I believe that rc1 does > not currently reflect the actual tip of branch-2.4.1. I'm not going to > consider this a deal-breaker, but seems like we should probably clean it up. > > To get this all sorted out properly, if we wanted to, I believe we should > do another backport of HDFS-6527 to branch-2.4.1 including only the amended > test case, and create a new RC from that point. > > Best, > Aaron > > -- > Aaron T. Myers > Software Engineer, Cloudera > > >> On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy wrote: >> >> Folks, >> >> I've created another release candidate (rc1) for hadoop-2.4.1 based on the >> feedback that I would like to push out. >> >> The RC is available at: >> http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 >> The RC tag in svn is here: >> https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 >> >> The maven artifacts are available via repository.apache.org. >> >> Please try the release and vote; the vote will run for the usual 7 days. >> >> thanks, >> Arun >> >> >> >> -- >> Arun C. Murthy >> Hortonworks Inc. >> http://hortonworks.com/hdp/ >> >> >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity to >> which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. >> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Moving to JDK7, JDK8 and new major releases
Thanks everyone for the discussion. Looks like we have come to a pragmatic and progressive conclusion. In terms of execution of the consensus plan, I think a little bit of caution is in order. Let's give downstream projects more of a runway. I propose we inform HBase, Pig, Hive etc. that we are considering making 2.6 (not 2.5) the last JDK6 release and solicit their feedback. Once they are comfortable we can pull the trigger in 2.7. thanks, Arun > On Jun 27, 2014, at 11:34 AM, Karthik Kambatla wrote: > > As someone else already mentioned, we should announce one future release > (may be, 2.5) as the last JDK6-based release before making the move to JDK7. > > I am comfortable calling 2.5 the last JDK6 release. > > > On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang > wrote: > >> Hi all, responding to multiple messages here, >> >> Arun, thanks for the clarification regarding MR classpaths. It sounds like >> the story there is improved and still improving. >> >> However, I think we still suffer from this at least on the HDFS side. We >> have a single JAR for all of HDFS, and our clients need to have all the fun >> deps like Guava on the classpath. I'm told Spark sticks a newer Guava at >> the front of the classpath and the HDFS client still works okay, but this >> is more happy coincidence than anything else. While we're leaking deps, >> we're in a scary situation. >> >> API compat to me means that an app should be able to run on a new minor >> version of Hadoop and not have anything break. MAPREDUCE-4421 sounds like >> it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what >> should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs and >> have nothing break. If we muck with the classpath, my understanding is that >> this could break. >> >> Owen, bumping the minimum JDK version in a minor release like this should >> be a one-time exception as Tucu stated. A number of people have pointed out >> how painful a forced JDK upgrade is for end users, and it's not something >> we should be springing on them in a minor release unless we're *very* >> confident like in this case. >> >> Chris, thanks for bringing up the ecosystem. For CDH5, we standardized on >> JDK7 across the CDH stack, so I think that's an indication that most >> ecosystem projects are ready to make the jump. Is that sufficient in your >> mind? >> >> For the record, I'm also +1 on the Tucu plan. Is it too late to do this for >> 2.5? I'll offer to help out with some of the mechanics. >> >> Thanks, >> Andrew >> >> On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth >> wrote: >> >>> I understood the plan for avoiding JDK7-specific features in our code, >> and >>> your suggestion to add an extra Jenkins job is a great way to guard >> against >>> that. The thing I haven't seen discussed yet is how downstream projects >>> will continue to consume our built artifacts. If a downstream project >>> upgrades to pick up a bug fix, and the jar switches to 1.7 class files, >> but >>> their project is still building with 1.6, then it would be a nasty >>> surprise. >>> >>> These are the options I see: >>> >>> 1. Make sure all other projects upgrade first. This doesn't sound >>> feasible, unless all other ecosystem projects have moved to JDK7 already. >>> If not, then waiting on a single long pole project would hold up our >>> migration indefinitely. >>> >>> 2. We switch to JDK7, but run javac with -target 1.6 until the whole >>> ecosystem upgrades. I find this undesirable, because in a certain sense, >>> it still leaves a bit of 1.6 lingering in the project. (I'll assume that >>> end-of-life for JDK6 also means end-of-life for the 1.6 bytecode format.) >>> >>> 3. Just declare a clean break on some version (your earlier email said >> 2.5) >>> and start publishing artifacts built with JDK7 and no -target option. >>> Overall, this is my preferred option. However, as a side effect, this >>> sets us up for longer-term maintenance and patch releases off of the 2.4 >>> branch if a downstream project that's still on 1.6 needs to pick up a >>> critical bug fix. >>> >>> Of course, this is all a moot point if all the downstream ecosystem >>> projects have already made the switch to JDK7. I don't know the status >> of >>> that off the top of my head. Maybe someone else out there knows? If >> not, >>> then I expect I can free up enough in a few weeks to volunteer for >> tracking >>> down that information. >>> >>> Chris Nauroth >>> Hortonworks >>> http://hortonworks.com/ >>> >>> >>> >>> On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur >>> wrote: >>> Chris, Compiling with jdk7 and doing javac -target 1.6 is not sufficient, you >>> are still using jdk7 libraries and you could use new APIs, thus breaking >> jdk6 both at compile and runtime. you need to compile with jdk6 to ensure you are not running into that scenario. that is why i was suggesting the nightly jdk6 build/test
Re: [VOTE] Release Apache Hadoop 2.4.1
I'm -0 on rc1. Note the latest discussion on HDFS-6527 which first resulted in that patch being reverted from branch-2.4.1 because it was believed it wasn't necessary, and then some more discussion which indicates that in fact the patch for HDFS-6527 should be included in 2.4.1, but with a slightly different test case. I believe that rc1 was actually created after the first backport of HDFS-6527, but before the revert, so rc1 should be functionally correct, but the test case is not quite correct in rc1, and I believe that rc1 does not currently reflect the actual tip of branch-2.4.1. I'm not going to consider this a deal-breaker, but seems like we should probably clean it up. To get this all sorted out properly, if we wanted to, I believe we should do another backport of HDFS-6527 to branch-2.4.1 including only the amended test case, and create a new RC from that point. Best, Aaron -- Aaron T. Myers Software Engineer, Cloudera On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy wrote: > Folks, > > I've created another release candidate (rc1) for hadoop-2.4.1 based on the > feedback that I would like to push out. > > The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 > The RC tag in svn is here: > https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 > > The maven artifacts are available via repository.apache.org. > > Please try the release and vote; the vote will run for the usual 7 days. > > thanks, > Arun > > > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/hdp/ > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: Anyone know how to mock a secured hdfs for unit test?
Hi David and Kai, There are a couple of challenges with this, but I just figured out a pretty decent setup while working on HDFS-2856. That code isn't committed yet, but if you open patch version 5 attached to that issue and look for the TestSaslDataTransfer class, then you'll see how it works. Most of the logic for bootstrapping a MiniKDC and setting up the right HDFS configuration properties is in an abstract base class named SaslDataTransferTestCase. I hope this helps. There are a few other open issues out there related to tests in secure mode. I know of HDFS-4312 and HDFS-5410. It would be great to get more regular test coverage with something that more closely approximates a secured deployment. Chris Nauroth Hortonworks http://hortonworks.com/ On Thu, Jun 26, 2014 at 7:27 AM, Zheng, Kai wrote: > Hi David, > > Quite some time ago I opened HADOOP-9952 and planned to create secured > MiniClusters by making use of MiniKDC. Unfortunately since then I didn't > get the chance to work on it yet. If you need something like that and would > contribute, please let me know and see if anything I can help with. Thanks. > > Regards, > Kai > > -Original Message- > From: Liu, David [mailto:liujion...@gmail.com] > Sent: Thursday, June 26, 2014 10:12 PM > To: hdfs-...@hadoop.apache.org; hdfs-iss...@hadoop.apache.org; > yarn-...@hadoop.apache.org; yarn-iss...@hadoop.apache.org; > mapreduce-dev@hadoop.apache.org; secur...@hadoop.apache.org > Subject: Anyone know how to mock a secured hdfs for unit test? > > Hi all, > > I need to test my code which read data from secured hdfs, is there any > library to mock secured hdfs, can minihdfscluster do the work? > Any suggestion is appreciated. > > > Thanks > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: hadoop-2.5 - June end?
Including YARN-1695 makes sense, just made it a blocker to ensure we get it in. YARN-1492 - would be great if we get it in, but I am not sure how feasible it is. Given it is being developed on a branch, there is also the merge to trunk and branch-2 overhead. I would prefer branching soon so we can work towards stabilizing the branch. People can continue to commit blockers to branch-2.5. On Fri, Jun 27, 2014 at 12:17 PM, Vinod Kumar Vavilapalli < vino...@apache.org> wrote: > Like I mentioned before, I'd like to get RM-webservices : YARN-1695 > included. May be we can wait for the branch till that point of time. > > I am also pushing for Shared Cache: YARN-1492 reviews. Again, like I said > before, we can take it as we go. > > +Vinod > Hortonworks Inc. > http://hortonworks.com/ > > > On Thu, Jun 26, 2014 at 12:08 AM, Karthik Kambatla > wrote: > > > Hi folks, > > > > For 2.5, looks like there are only 6 JIRAs marked Blocker and another 19 > > Critical ones [1]. I guess it makes sense to cut a branch for 2.5 and > work > > on getting the remaining items in. > > > > I ll go ahead and create the 2.5 branch in the next couple of days. I > hope > > http://wiki.apache.org/hadoop/HowToReleasePostMavenization is the right > > steps to follow. > > > > [1] JIRA Query: project in (Hadoop, HDFS, YARN, MapReduce) AND > resolution = > > Unresolved AND "Target Version/s" = 2.5.0 AND priority in (Blocker) > > > > > > On Mon, Jun 23, 2014 at 8:48 PM, Vinod Kumar Vavilapalli < > > vino...@apache.org > > > wrote: > > > > > I started reviewing YARN-1492 and synced up offline with Sangjin and > > Chris > > > who are leading this. I proposed that we should get the reviews and > > commits > > > going as they go and then mark the feature alpha, beta or stable > > depending > > > on how far we are ahead with the feature and how it aligns with the > > release > > > in progress. > > > > > > +Vinod > > > > > > On Jun 23, 2014, at 2:30 PM, Sangjin Lee wrote: > > > > > > > It would be great if we can consider the shared cache (YARN-1492) as > > part > > > > of 2.5. It obviously will depend on the reviews from folks, but the > > > feature > > > > itself should be reasonably solid. > > > > > > > > Thanks, > > > > Sangjin > > > > > > > > > > > > On Mon, Jun 23, 2014 at 2:24 PM, Arun C Murthy > > > wrote: > > > > > > > >> Thanks Karthik! > > > >> > > > >> I've updated https://wiki.apache.org/hadoop/Roadmap with features > > which > > > >> are very close to completion. Let's see if that makes sense and if > we > > > get > > > >> any further feedback. > > > >> > > > >> Arun > > > >> > > > >> On Jun 23, 2014, at 2:09 PM, Karthik Kambatla > > > wrote: > > > >> > > > >>> I can pick up the RM duties for 2.5. If I run into any HDFS > doubts, I > > > >> might > > > >>> need some help from someone more familiar with HDFS. > > > >>> > > > >>> > > > >>> On Mon, Jun 23, 2014 at 12:07 PM, Arun C Murthy < > a...@hortonworks.com > > > > > > >> wrote: > > > >>> > > > Folks, > > > > > > I'd appreciate some help here. Due to family reasons (all good > > ones), > > > I'll be away for a couple of weeks. Can someone else pick up the > RM > > > >> duties > > > for hadoop-2.5? Maybe Andrew since he's expressed interest in the > > > past? > > > >> I > > > will pick up the thread again for hadoop-2.6, but I don't want to > > > block > > > hadoop-2.5 due to my non-availability. > > > > > > thanks, > > > Arun > > > > > > On Jun 9, 2014, at 9:39 AM, Arun C Murthy > > > wrote: > > > > > > > Folks, > > > > > > > > As you can see from the Roadmap wiki, it looks like several items > > are > > > still a bit away from being ready. > > > > > > > > I think rather than wait for them, it will be useful to create an > > > intermediate release (2.5) this month - I think ATS security is > > pretty > > > close, so we can ship that. I'm thinking of creating hadoop-2.5 by > > end > > > >> of > > > the month, with a branch a couple of weeks prior. > > > > > > > > Thoughts? > > > > > > > > thanks, > > > > Arun > > > > > > > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > > > >> entity to > > > which it is addressed and may contain information that is > > > confidential, > > > privileged and exempt from disclosure under applicable law. If the > > > >> reader > > > of this message is not the intended recipient, you are hereby > > notified > > > >> that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you > have > > > received this communication in error, please contact the sender > > > >> immediately > > > and delete it from your system. Thank You. > > > > > > >> > > > >> -- > > > >> Arun C. Murthy > > > >> Hortonworks In
Re: hadoop-2.5 - June end?
Like I mentioned before, I'd like to get RM-webservices : YARN-1695 included. May be we can wait for the branch till that point of time. I am also pushing for Shared Cache: YARN-1492 reviews. Again, like I said before, we can take it as we go. +Vinod Hortonworks Inc. http://hortonworks.com/ On Thu, Jun 26, 2014 at 12:08 AM, Karthik Kambatla wrote: > Hi folks, > > For 2.5, looks like there are only 6 JIRAs marked Blocker and another 19 > Critical ones [1]. I guess it makes sense to cut a branch for 2.5 and work > on getting the remaining items in. > > I ll go ahead and create the 2.5 branch in the next couple of days. I hope > http://wiki.apache.org/hadoop/HowToReleasePostMavenization is the right > steps to follow. > > [1] JIRA Query: project in (Hadoop, HDFS, YARN, MapReduce) AND resolution = > Unresolved AND "Target Version/s" = 2.5.0 AND priority in (Blocker) > > > On Mon, Jun 23, 2014 at 8:48 PM, Vinod Kumar Vavilapalli < > vino...@apache.org > > wrote: > > > I started reviewing YARN-1492 and synced up offline with Sangjin and > Chris > > who are leading this. I proposed that we should get the reviews and > commits > > going as they go and then mark the feature alpha, beta or stable > depending > > on how far we are ahead with the feature and how it aligns with the > release > > in progress. > > > > +Vinod > > > > On Jun 23, 2014, at 2:30 PM, Sangjin Lee wrote: > > > > > It would be great if we can consider the shared cache (YARN-1492) as > part > > > of 2.5. It obviously will depend on the reviews from folks, but the > > feature > > > itself should be reasonably solid. > > > > > > Thanks, > > > Sangjin > > > > > > > > > On Mon, Jun 23, 2014 at 2:24 PM, Arun C Murthy > > wrote: > > > > > >> Thanks Karthik! > > >> > > >> I've updated https://wiki.apache.org/hadoop/Roadmap with features > which > > >> are very close to completion. Let's see if that makes sense and if we > > get > > >> any further feedback. > > >> > > >> Arun > > >> > > >> On Jun 23, 2014, at 2:09 PM, Karthik Kambatla > > wrote: > > >> > > >>> I can pick up the RM duties for 2.5. If I run into any HDFS doubts, I > > >> might > > >>> need some help from someone more familiar with HDFS. > > >>> > > >>> > > >>> On Mon, Jun 23, 2014 at 12:07 PM, Arun C Murthy > > > >> wrote: > > >>> > > Folks, > > > > I'd appreciate some help here. Due to family reasons (all good > ones), > > I'll be away for a couple of weeks. Can someone else pick up the RM > > >> duties > > for hadoop-2.5? Maybe Andrew since he's expressed interest in the > > past? > > >> I > > will pick up the thread again for hadoop-2.6, but I don't want to > > block > > hadoop-2.5 due to my non-availability. > > > > thanks, > > Arun > > > > On Jun 9, 2014, at 9:39 AM, Arun C Murthy > > wrote: > > > > > Folks, > > > > > > As you can see from the Roadmap wiki, it looks like several items > are > > still a bit away from being ready. > > > > > > I think rather than wait for them, it will be useful to create an > > intermediate release (2.5) this month - I think ATS security is > pretty > > close, so we can ship that. I'm thinking of creating hadoop-2.5 by > end > > >> of > > the month, with a branch a couple of weeks prior. > > > > > > Thoughts? > > > > > > thanks, > > > Arun > > > > > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or > > >> entity to > > which it is addressed and may contain information that is > > confidential, > > privileged and exempt from disclosure under applicable law. If the > > >> reader > > of this message is not the intended recipient, you are hereby > notified > > >> that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > > >> immediately > > and delete it from your system. Thank You. > > > > >> > > >> -- > > >> Arun C. Murthy > > >> Hortonworks Inc. > > >> http://hortonworks.com/hdp/ > > >> > > >> > > >> > > >> -- > > >> CONFIDENTIALITY NOTICE > > >> NOTICE: This message is intended for the use of the individual or > > entity to > > >> which it is addressed and may contain information that is > confidential, > > >> privileged and exempt from disclosure under applicable law. If the > > reader > > >> of this message is not the intended recipient, you are hereby notified > > that > > >> any printing, copying, dissemination, distribution, disclosure or > > >> forwarding of this communication is strictly prohibited. If you have > > >> received this communication in error, please contact the sender > > immediately > > >> and delete it from your system. Thank You. > > >> > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is in
Re: Moving to JDK7, JDK8 and new major releases
FYI I also just updated the wiki page with a Proposal D, aka Tucu plan, which I think is essentially Proposal C but tabling JDK8 plans for now. https://wiki.apache.org/hadoop/MovingToJdk7and8 Karthik, thanks for ringing in re: 2.5. I guess there's nothing urgently required, the Jenkins stuff just needs to happen before 2.6. Still, I'm happy to help with anything. Thanks, Andrew On Fri, Jun 27, 2014 at 11:34 AM, Karthik Kambatla wrote: > As someone else already mentioned, we should announce one future release > (may be, 2.5) as the last JDK6-based release before making the move to > JDK7. > > I am comfortable calling 2.5 the last JDK6 release. > > > On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang > wrote: > > > Hi all, responding to multiple messages here, > > > > Arun, thanks for the clarification regarding MR classpaths. It sounds > like > > the story there is improved and still improving. > > > > However, I think we still suffer from this at least on the HDFS side. We > > have a single JAR for all of HDFS, and our clients need to have all the > fun > > deps like Guava on the classpath. I'm told Spark sticks a newer Guava at > > the front of the classpath and the HDFS client still works okay, but this > > is more happy coincidence than anything else. While we're leaking deps, > > we're in a scary situation. > > > > API compat to me means that an app should be able to run on a new minor > > version of Hadoop and not have anything break. MAPREDUCE-4421 sounds like > > it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what > > should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs and > > have nothing break. If we muck with the classpath, my understanding is > that > > this could break. > > > > Owen, bumping the minimum JDK version in a minor release like this should > > be a one-time exception as Tucu stated. A number of people have pointed > out > > how painful a forced JDK upgrade is for end users, and it's not something > > we should be springing on them in a minor release unless we're *very* > > confident like in this case. > > > > Chris, thanks for bringing up the ecosystem. For CDH5, we standardized on > > JDK7 across the CDH stack, so I think that's an indication that most > > ecosystem projects are ready to make the jump. Is that sufficient in your > > mind? > > > > For the record, I'm also +1 on the Tucu plan. Is it too late to do this > for > > 2.5? I'll offer to help out with some of the mechanics. > > > > Thanks, > > Andrew > > > > On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth > > > wrote: > > > > > I understood the plan for avoiding JDK7-specific features in our code, > > and > > > your suggestion to add an extra Jenkins job is a great way to guard > > against > > > that. The thing I haven't seen discussed yet is how downstream > projects > > > will continue to consume our built artifacts. If a downstream project > > > upgrades to pick up a bug fix, and the jar switches to 1.7 class files, > > but > > > their project is still building with 1.6, then it would be a nasty > > > surprise. > > > > > > These are the options I see: > > > > > > 1. Make sure all other projects upgrade first. This doesn't sound > > > feasible, unless all other ecosystem projects have moved to JDK7 > already. > > > If not, then waiting on a single long pole project would hold up our > > > migration indefinitely. > > > > > > 2. We switch to JDK7, but run javac with -target 1.6 until the whole > > > ecosystem upgrades. I find this undesirable, because in a certain > sense, > > > it still leaves a bit of 1.6 lingering in the project. (I'll assume > that > > > end-of-life for JDK6 also means end-of-life for the 1.6 bytecode > format.) > > > > > > 3. Just declare a clean break on some version (your earlier email said > > 2.5) > > > and start publishing artifacts built with JDK7 and no -target option. > > > Overall, this is my preferred option. However, as a side effect, this > > > sets us up for longer-term maintenance and patch releases off of the > 2.4 > > > branch if a downstream project that's still on 1.6 needs to pick up a > > > critical bug fix. > > > > > > Of course, this is all a moot point if all the downstream ecosystem > > > projects have already made the switch to JDK7. I don't know the status > > of > > > that off the top of my head. Maybe someone else out there knows? If > > not, > > > then I expect I can free up enough in a few weeks to volunteer for > > tracking > > > down that information. > > > > > > Chris Nauroth > > > Hortonworks > > > http://hortonworks.com/ > > > > > > > > > > > > On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur > > > > wrote: > > > > > > > Chris, > > > > > > > > Compiling with jdk7 and doing javac -target 1.6 is not sufficient, > you > > > are > > > > still using jdk7 libraries and you could use new APIs, thus breaking > > jdk6 > > > > both at compile and runtime. > > > > > > > > you need to compile with jdk6 to ensure you are not run
Re: Moving to JDK7, JDK8 and new major releases
As someone else already mentioned, we should announce one future release (may be, 2.5) as the last JDK6-based release before making the move to JDK7. I am comfortable calling 2.5 the last JDK6 release. On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang wrote: > Hi all, responding to multiple messages here, > > Arun, thanks for the clarification regarding MR classpaths. It sounds like > the story there is improved and still improving. > > However, I think we still suffer from this at least on the HDFS side. We > have a single JAR for all of HDFS, and our clients need to have all the fun > deps like Guava on the classpath. I'm told Spark sticks a newer Guava at > the front of the classpath and the HDFS client still works okay, but this > is more happy coincidence than anything else. While we're leaking deps, > we're in a scary situation. > > API compat to me means that an app should be able to run on a new minor > version of Hadoop and not have anything break. MAPREDUCE-4421 sounds like > it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what > should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs and > have nothing break. If we muck with the classpath, my understanding is that > this could break. > > Owen, bumping the minimum JDK version in a minor release like this should > be a one-time exception as Tucu stated. A number of people have pointed out > how painful a forced JDK upgrade is for end users, and it's not something > we should be springing on them in a minor release unless we're *very* > confident like in this case. > > Chris, thanks for bringing up the ecosystem. For CDH5, we standardized on > JDK7 across the CDH stack, so I think that's an indication that most > ecosystem projects are ready to make the jump. Is that sufficient in your > mind? > > For the record, I'm also +1 on the Tucu plan. Is it too late to do this for > 2.5? I'll offer to help out with some of the mechanics. > > Thanks, > Andrew > > On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth > wrote: > > > I understood the plan for avoiding JDK7-specific features in our code, > and > > your suggestion to add an extra Jenkins job is a great way to guard > against > > that. The thing I haven't seen discussed yet is how downstream projects > > will continue to consume our built artifacts. If a downstream project > > upgrades to pick up a bug fix, and the jar switches to 1.7 class files, > but > > their project is still building with 1.6, then it would be a nasty > > surprise. > > > > These are the options I see: > > > > 1. Make sure all other projects upgrade first. This doesn't sound > > feasible, unless all other ecosystem projects have moved to JDK7 already. > > If not, then waiting on a single long pole project would hold up our > > migration indefinitely. > > > > 2. We switch to JDK7, but run javac with -target 1.6 until the whole > > ecosystem upgrades. I find this undesirable, because in a certain sense, > > it still leaves a bit of 1.6 lingering in the project. (I'll assume that > > end-of-life for JDK6 also means end-of-life for the 1.6 bytecode format.) > > > > 3. Just declare a clean break on some version (your earlier email said > 2.5) > > and start publishing artifacts built with JDK7 and no -target option. > > Overall, this is my preferred option. However, as a side effect, this > > sets us up for longer-term maintenance and patch releases off of the 2.4 > > branch if a downstream project that's still on 1.6 needs to pick up a > > critical bug fix. > > > > Of course, this is all a moot point if all the downstream ecosystem > > projects have already made the switch to JDK7. I don't know the status > of > > that off the top of my head. Maybe someone else out there knows? If > not, > > then I expect I can free up enough in a few weeks to volunteer for > tracking > > down that information. > > > > Chris Nauroth > > Hortonworks > > http://hortonworks.com/ > > > > > > > > On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur > > wrote: > > > > > Chris, > > > > > > Compiling with jdk7 and doing javac -target 1.6 is not sufficient, you > > are > > > still using jdk7 libraries and you could use new APIs, thus breaking > jdk6 > > > both at compile and runtime. > > > > > > you need to compile with jdk6 to ensure you are not running into that > > > scenario. that is why i was suggesting the nightly jdk6 build/test > > jenkins > > > job. > > > > > > > > > On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth < > cnaur...@hortonworks.com > > > > > > wrote: > > > > > > > I'm also +1 for getting us to JDK7 within the 2.x line after reading > > the > > > > proposals and catching up on the discussion in this thread. > > > > > > > > Has anyone yet considered how to coordinate this change with > downstream > > > > projects? Would we request downstream projects to upgrade to JDK7 > > first > > > > before we make the move? Would we switch to JDK7, but run javac > > -target > > > > 1.6 to maintain compatibility for
Re: Moving to JDK7, JDK8 and new major releases
Hi all, responding to multiple messages here, Arun, thanks for the clarification regarding MR classpaths. It sounds like the story there is improved and still improving. However, I think we still suffer from this at least on the HDFS side. We have a single JAR for all of HDFS, and our clients need to have all the fun deps like Guava on the classpath. I'm told Spark sticks a newer Guava at the front of the classpath and the HDFS client still works okay, but this is more happy coincidence than anything else. While we're leaking deps, we're in a scary situation. API compat to me means that an app should be able to run on a new minor version of Hadoop and not have anything break. MAPREDUCE-4421 sounds like it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs and have nothing break. If we muck with the classpath, my understanding is that this could break. Owen, bumping the minimum JDK version in a minor release like this should be a one-time exception as Tucu stated. A number of people have pointed out how painful a forced JDK upgrade is for end users, and it's not something we should be springing on them in a minor release unless we're *very* confident like in this case. Chris, thanks for bringing up the ecosystem. For CDH5, we standardized on JDK7 across the CDH stack, so I think that's an indication that most ecosystem projects are ready to make the jump. Is that sufficient in your mind? For the record, I'm also +1 on the Tucu plan. Is it too late to do this for 2.5? I'll offer to help out with some of the mechanics. Thanks, Andrew On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth wrote: > I understood the plan for avoiding JDK7-specific features in our code, and > your suggestion to add an extra Jenkins job is a great way to guard against > that. The thing I haven't seen discussed yet is how downstream projects > will continue to consume our built artifacts. If a downstream project > upgrades to pick up a bug fix, and the jar switches to 1.7 class files, but > their project is still building with 1.6, then it would be a nasty > surprise. > > These are the options I see: > > 1. Make sure all other projects upgrade first. This doesn't sound > feasible, unless all other ecosystem projects have moved to JDK7 already. > If not, then waiting on a single long pole project would hold up our > migration indefinitely. > > 2. We switch to JDK7, but run javac with -target 1.6 until the whole > ecosystem upgrades. I find this undesirable, because in a certain sense, > it still leaves a bit of 1.6 lingering in the project. (I'll assume that > end-of-life for JDK6 also means end-of-life for the 1.6 bytecode format.) > > 3. Just declare a clean break on some version (your earlier email said 2.5) > and start publishing artifacts built with JDK7 and no -target option. > Overall, this is my preferred option. However, as a side effect, this > sets us up for longer-term maintenance and patch releases off of the 2.4 > branch if a downstream project that's still on 1.6 needs to pick up a > critical bug fix. > > Of course, this is all a moot point if all the downstream ecosystem > projects have already made the switch to JDK7. I don't know the status of > that off the top of my head. Maybe someone else out there knows? If not, > then I expect I can free up enough in a few weeks to volunteer for tracking > down that information. > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > > On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur > wrote: > > > Chris, > > > > Compiling with jdk7 and doing javac -target 1.6 is not sufficient, you > are > > still using jdk7 libraries and you could use new APIs, thus breaking jdk6 > > both at compile and runtime. > > > > you need to compile with jdk6 to ensure you are not running into that > > scenario. that is why i was suggesting the nightly jdk6 build/test > jenkins > > job. > > > > > > On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth > > > wrote: > > > > > I'm also +1 for getting us to JDK7 within the 2.x line after reading > the > > > proposals and catching up on the discussion in this thread. > > > > > > Has anyone yet considered how to coordinate this change with downstream > > > projects? Would we request downstream projects to upgrade to JDK7 > first > > > before we make the move? Would we switch to JDK7, but run javac > -target > > > 1.6 to maintain compatibility for downstream projects during an interim > > > period? > > > > > > Chris Nauroth > > > Hortonworks > > > http://hortonworks.com/ > > > > > > > > > > > > On Wed, Jun 25, 2014 at 9:48 AM, Owen O'Malley > > wrote: > > > > > > > On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur < > t...@cloudera.com > > > > > > > wrote: > > > > > > > > > After reading this thread and thinking a bit about it, I think it > > > should > > > > be > > > > > OK such move up to JDK7 in Hadoop > > > > > > > > > > > > I agree with Alejand
Re: hadoop-2.5 - June end?
I was trying to do a JIRA bulk update to target all non-blockers to 2.6, but doesn't look like the bulk-update tool doesn't have Target Version. Does anyone know if that is true or if it is me not having permissions? On Thu, Jun 26, 2014 at 12:08 AM, Karthik Kambatla wrote: > Hi folks, > > For 2.5, looks like there are only 6 JIRAs marked Blocker and another 19 > Critical ones [1]. I guess it makes sense to cut a branch for 2.5 and work > on getting the remaining items in. > > I ll go ahead and create the 2.5 branch in the next couple of days. I hope > http://wiki.apache.org/hadoop/HowToReleasePostMavenization is the right > steps to follow. > > [1] JIRA Query: project in (Hadoop, HDFS, YARN, MapReduce) AND resolution > = Unresolved AND "Target Version/s" = 2.5.0 AND priority in (Blocker) > > > On Mon, Jun 23, 2014 at 8:48 PM, Vinod Kumar Vavilapalli < > vino...@apache.org> wrote: > >> I started reviewing YARN-1492 and synced up offline with Sangjin and >> Chris who are leading this. I proposed that we should get the reviews and >> commits going as they go and then mark the feature alpha, beta or stable >> depending on how far we are ahead with the feature and how it aligns with >> the release in progress. >> >> +Vinod >> >> On Jun 23, 2014, at 2:30 PM, Sangjin Lee wrote: >> >> > It would be great if we can consider the shared cache (YARN-1492) as >> part >> > of 2.5. It obviously will depend on the reviews from folks, but the >> feature >> > itself should be reasonably solid. >> > >> > Thanks, >> > Sangjin >> > >> > >> > On Mon, Jun 23, 2014 at 2:24 PM, Arun C Murthy >> wrote: >> > >> >> Thanks Karthik! >> >> >> >> I've updated https://wiki.apache.org/hadoop/Roadmap with features >> which >> >> are very close to completion. Let's see if that makes sense and if we >> get >> >> any further feedback. >> >> >> >> Arun >> >> >> >> On Jun 23, 2014, at 2:09 PM, Karthik Kambatla >> wrote: >> >> >> >>> I can pick up the RM duties for 2.5. If I run into any HDFS doubts, I >> >> might >> >>> need some help from someone more familiar with HDFS. >> >>> >> >>> >> >>> On Mon, Jun 23, 2014 at 12:07 PM, Arun C Murthy >> >> wrote: >> >>> >> Folks, >> >> I'd appreciate some help here. Due to family reasons (all good ones), >> I'll be away for a couple of weeks. Can someone else pick up the RM >> >> duties >> for hadoop-2.5? Maybe Andrew since he's expressed interest in the >> past? >> >> I >> will pick up the thread again for hadoop-2.6, but I don't want to >> block >> hadoop-2.5 due to my non-availability. >> >> thanks, >> Arun >> >> On Jun 9, 2014, at 9:39 AM, Arun C Murthy >> wrote: >> >> > Folks, >> > >> > As you can see from the Roadmap wiki, it looks like several items >> are >> still a bit away from being ready. >> > >> > I think rather than wait for them, it will be useful to create an >> intermediate release (2.5) this month - I think ATS security is >> pretty >> close, so we can ship that. I'm thinking of creating hadoop-2.5 by >> end >> >> of >> the month, with a branch a couple of weeks prior. >> > >> > Thoughts? >> > >> > thanks, >> > Arun >> > >> >> >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or >> >> entity to >> which it is addressed and may contain information that is >> confidential, >> privileged and exempt from disclosure under applicable law. If the >> >> reader >> of this message is not the intended recipient, you are hereby >> notified >> >> that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender >> >> immediately >> and delete it from your system. Thank You. >> >> >> >> >> -- >> >> Arun C. Murthy >> >> Hortonworks Inc. >> >> http://hortonworks.com/hdp/ >> >> >> >> >> >> >> >> -- >> >> CONFIDENTIALITY NOTICE >> >> NOTICE: This message is intended for the use of the individual or >> entity to >> >> which it is addressed and may contain information that is confidential, >> >> privileged and exempt from disclosure under applicable law. If the >> reader >> >> of this message is not the intended recipient, you are hereby notified >> that >> >> any printing, copying, dissemination, distribution, disclosure or >> >> forwarding of this communication is strictly prohibited. If you have >> >> received this communication in error, please contact the sender >> immediately >> >> and delete it from your system. Thank You. >> >> >> >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to >> which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of
Hadoop-Mapreduce-trunk - Build # 1814 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1814/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 35145 lines...] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.782 sec - in org.apache.hadoop.mapreduce.TestChild Running org.apache.hadoop.mapreduce.filecache.TestURIFragments Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.064 sec - in org.apache.hadoop.mapreduce.filecache.TestURIFragments Running org.apache.hadoop.mapreduce.TestMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.016 sec - in org.apache.hadoop.mapreduce.TestMapReduce Running org.apache.hadoop.mapreduce.TestNewCombinerGrouping Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.613 sec - in org.apache.hadoop.mapreduce.TestNewCombinerGrouping Results : Tests run: 494, Failures: 0, Errors: 0, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [2.488s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [54.104s] [INFO] hadoop-mapreduce-client-common SUCCESS [26.573s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [4.280s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [7:05.509s] [INFO] hadoop-mapreduce-client-hs SUCCESS [4:44.196s] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [1:37:18.317s] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1:50:36.512s [INFO] Finished at: Fri Jun 27 15:16:28 UTC 2014 [INFO] Final Memory: 28M/99M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #1765 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 16955295 bytes Compression is 0.0% Took 12 sec Updating MAPREDUCE-5939 Updating HDFS-6572 Updating HADOOP-8943 Updating HADOOP-10565 Updating HADOOP-10701 Updating HADOOP-10754 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: [VOTE] Release Apache Hadoop 0.23.11
Thanks everyone for voting. The Vote passes with 11 +1¹s (5 binding, 6 non-binding) and no -1¹s. I¹ll push the release out. Tom On 6/19/14, 10:14 AM, "Thomas Graves" wrote: >Hey Everyone, > >There have been various bug fixes that have went into >branch-0.23 since the 0.23.10 release. We think its time to do a 0.23.11. > >This is also the last planned release off of branch-0.23 we plan on doing. > >The RC is available at: >http://people.apache.org/~tgraves/hadoop-0.23.11-candidate-0/ > > >The RC Tag in svn is here: >http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.11-rc0/ > >The maven artifacts are available via repository.apache.org. > >Please try the release and vote; the vote will run for the usual 7 days >til June 26th. > >I am +1 (binding). > >thanks, >Tom Graves > > > >
ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint
I use hadoop2.2.0, first I set hdfs-site.xml,like follows: [yarn@localhost hadoop]$ vi hdfs-site.xml dfs.replication 1 dfs.namenode.name.dir file:/var/data/hadoop/hdfs/nn fs.checkpoint.dir file:/var/data/hadoop/hdfs/snn fs.checkpoint.edits.dir file:/var/data/hadoop/hdfs/snn dfs.datanode.data.dir file:/var/data/hadoop/hdfs/dn dfs.namenode.http-address http://localhost:50070 dfs.namenode.secondary.http-address http://localhost:50090 Then I try hadoop-mapreduce-examples like follows: [yarn@localhost hadoop]$ hadoop jar /opt/yarn/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -libjars /opt/yarn/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar 16 1 But it raise following error,why raise SecondaryNameNode: Exception in doCheckpoint? How to correct hdfs-site.xml file? Thanks. Safe mode will be turned off automatically at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:5047) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:832) at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139) at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:11214) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) at org.apache.hadoop.ipc.Client.call(Client.java:1347) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy8.rollEditLog(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy8.rollEditLog(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:502) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:380) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$2.run(SecondaryNameNode.java:346) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:342) at java.lang.Thread.run(Thread.java:745) 2014-06-26 23:27:56,072 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.SafeModeException): Log not rolled. Name node is in safe mode. The reported blocks 0 needs additional 25 blocks to reach the threshold 0.9990 of total blocks 25. Safe mode will be turned off automatically at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:5047) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:832) at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:139) at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:11214) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042
