Re: Apache Hadoop 3.0.1 Release plan
Hey Anu, My feeling on HDFS-12990 is that we've discussed it quite a bit already and it doesn't seem at this point like either side is going to budge. I'm certainly happy to have a phone call about it, but I don't expect that we'd make much progress. My suggestion is that we simply include the patch posted to HDFS-12990 in the 3.0.1 RC and call this issue out clearly in the subsequent VOTE thread for the 3.0.1 release. Eddy, are you up for that? Best, Aaron On Thu, Feb 1, 2018 at 1:13 PM, Lei Xuwrote: > +Xiao > > My understanding is that we will have this for 3.0.1. Xiao, could > you give your inputs here? > > On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer > wrote: > > Hi Eddy, > > > > Thanks for driving this release. Just a quick question, do we have time > to close this issue? > > https://issues.apache.org/jira/browse/HDFS-12990 > > > > or are we abandoning it? I believe that this is the last window for us > to fix this issue. > > > > Should we have a call and get this resolved one way or another? > > > > Thanks > > Anu > > > > On 2/1/18, 10:51 AM, "Lei Xu" wrote: > > > > Hi, All > > > > I just cut branch-3.0.1 from branch-3.0. Please make sure all > patches > > targeted to 3.0.1 being checked in both branch-3.0 and branch-3.0.1. > > > > Thanks! > > Eddy > > > > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu wrote: > > > Hi, All > > > > > > We have released Apache Hadoop 3.0.0 in December [1]. To further > > > improve the quality of release, we plan to cut branch-3.0.1 branch > > > tomorrow for the preparation of Apache Hadoop 3.0.1 release. The > focus > > > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug > fixes > > > [2]. No new features and improvement should be included. > > > > > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on > Feb > > > 1st, targeting for Feb 9th release. > > > > > > Please feel free to share your insights. > > > > > > [1] https://www.mail-archive.com/general@hadoop.apache.org/ > msg07757.html > > > [2] https://issues.apache.org/jira/issues/?filter=12342842 > > > > > > Best, > > > -- > > > Lei (Eddy) Xu > > > Software Engineer, Cloudera > > > > > > > > -- > > Lei (Eddy) Xu > > Software Engineer, Cloudera > > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > > > > > > > -- > Lei (Eddy) Xu > Software Engineer, Cloudera > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 3.0.0 RC1
+1 (binding) - downloaded the src tarball and built the source (-Pdist -Pnative) - verified the checksum - brought up a secure pseudo distributed cluster - did some basic file system operations (mkdir, list, put, cat) and confirmed that everything was working - confirmed that the web UI worked Best, Aaron On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wangwrote: > Hi all, > > Let me start, as always, by thanking the efforts of all the contributors > who contributed to this release, especially those who jumped on the issues > found in RC0. > > I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302 > fixed JIRAs since the previous 3.0.0-beta1 release. > > You can find the artifacts here: > > http://home.apache.org/~wang/3.0.0-RC1/ > > I've done the traditional testing of building from the source tarball and > running a Pi job on a single node cluster. I also verified that the shaded > jars are not empty. > > Found one issue that create-release (probably due to the mvn deploy change) > didn't sign the artifacts, but I fixed that by calling mvn one more time. > Available here: > > https://repository.apache.org/content/repositories/orgapachehadoop-1075/ > > This release will run the standard 5 days, closing on Dec 13th at 12:31pm > Pacific. My +1 to start. > > Best, > Andrew >
Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0
+1 (binding) from me. Downloaded the source, built from source, set up a pseudo cluster, and ran a few of the sample jobs. Thanks a lot for doing all this release work, Andrew. -- Aaron T. Myers Software Engineer, Cloudera On Tue, Aug 30, 2016 at 8:51 AM, Andrew Wang <andrew.w...@cloudera.com> wrote: > Hi all, > > Thanks to the combined work of many, many contributors, here's an RC0 for > 3.0.0-alpha1: > > http://home.apache.org/~wang/3.0.0-alpha1-RC0/ > > alpha1 is the first in a series of planned alpha releases leading up to GA. > The objective is to get an artifact out to downstreams for testing and to > iterate quickly based on their feedback. So, please keep that in mind when > voting; hopefully most issues can be addressed by future alphas rather than > future RCs. > > Sorry for getting this out on a Tuesday, but I'd still like this vote to > run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend > if we lack the votes. > > Please try it out and let me know what you think. > > Best, > Andrew >
Re: Why there are so many revert operations on trunk?
Junping, All of this is being discussed on HDFS-9924. Suggest you follow the conversation there. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Jun 6, 2016 at 7:20 AM, Junping Du <j...@hortonworks.com> wrote: > Hi Andrew, > > I just noticed you revert 8 commits on trunk last Friday: > > HADOOP-13226 > > HDFS-10430 > > HDFS-10431 > > HDFS-10390 > > HADOOP-13168 > > HDFS-10390 > > HADOOP-13168 > > HDFS-10346 > > HADOOP-12957 > > HDFS-10224 > >And I didn't see you have any comments on JIRA or email discussion > before you did this. I don't think we are legally allowed to do this even > as committer/PMC member. Can you explain what's your intention to do this? > >BTW, thanks Nicolas to revert all these "illegal" revert operations. > > > > Thanks, > > > Junping >
Re: Looking to a Hadoop 3 release
+1, this sounds like a good plan to me. Thanks a lot for volunteering to take this on, Andrew. Best, Aaron On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hi devs, It's been a year and a half since 2.x went GA, and I think we're about due for a 3.x release. Notably, there are two incompatible changes I'd like to call out, that will have a tremendous positive impact for our users. First, classpath isolation being done at HADOOP-11656, which has been a long-standing request from many downstreams and Hadoop users. Second, bumping the source and target JDK version to JDK8 (related to HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two months from now). In the past, we've had issues with our dependencies discontinuing support for old JDKs, so this will future-proof us. Between the two, we'll also have quite an opportunity to clean up and upgrade our dependencies, another common user and developer request. I'd like to propose that we start rolling a series of monthly-ish series of 3.0 alpha releases ASAP, with myself volunteering to take on the RM and other cat herding responsibilities. There are already quite a few changes slated for 3.0 besides the above (for instance the shell script rewrite) so there's already value in a 3.0 alpha, and the more time we give downstreams to integrate, the better. This opens up discussion about inclusion of other changes, but I'm hoping to freeze incompatible changes after maybe two alphas, do a beta (with no further incompat changes allowed), and then finally a 3.x GA. For those keeping track, that means a 3.x GA in about four months. I would also like to stress though that this is not intended to be a big bang release. For instance, it would be great if we could maintain wire compatibility between 2.x and 3.x, so rolling upgrades work. Keeping branch-2 and branch-3 similar also makes backports easier, since we're likely maintaining 2.x for a while yet. Please let me know any comments / concerns related to the above. If people are friendly to the idea, I'd like to cut a branch-3 and start working on the first alpha. Best, Andrew
Re: [VOTE] Migration from subversion to git for version control
+1 (binding) Thanks for driving this, Karthik. -- Aaron T. Myers Software Engineer, Cloudera On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote: I have put together this proposal based on recent discussion on this topic. Please vote on the proposal. The vote runs for 7 days. 1. Migrate from subversion to git for version control. 2. Force-push to be disabled on trunk and branch-* branches. Applying changes from any of trunk/branch-* to any of branch-* should be through git cherry-pick -x. 3. Force-push on feature-branches is allowed. Before pulling in a feature, the feature-branch should be rebased on latest trunk and the changes applied to trunk through git rebase --onto or git cherry-pick commit-range. 4. Every time a feature branch is rebased on trunk, a tag that identifies the state before the rebase needs to be created (e.g. tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once the feature is pulled into trunk and the tags are no longer useful. 5. The relevance/use of tags stay the same after the migration. Thanks Karthik PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of vote and will be Lazy 2/3 majority of PMC members.
Re: [VOTE] Release Apache Hadoop 2.4.1
I'm -0 on rc1. Note the latest discussion on HDFS-6527 which first resulted in that patch being reverted from branch-2.4.1 because it was believed it wasn't necessary, and then some more discussion which indicates that in fact the patch for HDFS-6527 should be included in 2.4.1, but with a slightly different test case. I believe that rc1 was actually created after the first backport of HDFS-6527, but before the revert, so rc1 should be functionally correct, but the test case is not quite correct in rc1, and I believe that rc1 does not currently reflect the actual tip of branch-2.4.1. I'm not going to consider this a deal-breaker, but seems like we should probably clean it up. To get this all sorted out properly, if we wanted to, I believe we should do another backport of HDFS-6527 to branch-2.4.1 including only the amended test case, and create a new RC from that point. Best, Aaron -- Aaron T. Myers Software Engineer, Cloudera On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.4.1 based on the feedback that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.4.1
That's fine by me. Like I said, assuming that rc1 does indeed include the fix in HDFS-6527, and not the revert, then rc1 should be functionally correct. What's in branch-2.4.1 doesn't currently match what's in this RC, but if that doesn't bother anyone else then I won't lose any sleep over it. -- Aaron T. Myers Software Engineer, Cloudera On Jun 27, 2014, at 3:04 PM, Arun C. Murthy a...@hortonworks.com wrote: Aaron, Since the amend was just to the test, I'll keep this RC as-is. I'll also comment on jira. thanks, Arun On Jun 27, 2014, at 2:40 PM, Aaron T. Myers a...@cloudera.com wrote: I'm -0 on rc1. Note the latest discussion on HDFS-6527 which first resulted in that patch being reverted from branch-2.4.1 because it was believed it wasn't necessary, and then some more discussion which indicates that in fact the patch for HDFS-6527 should be included in 2.4.1, but with a slightly different test case. I believe that rc1 was actually created after the first backport of HDFS-6527, but before the revert, so rc1 should be functionally correct, but the test case is not quite correct in rc1, and I believe that rc1 does not currently reflect the actual tip of branch-2.4.1. I'm not going to consider this a deal-breaker, but seems like we should probably clean it up. To get this all sorted out properly, if we wanted to, I believe we should do another backport of HDFS-6527 to branch-2.4.1 including only the amended test case, and create a new RC from that point. Best, Aaron -- Aaron T. Myers Software Engineer, Cloudera On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created another release candidate (rc1) for hadoop-2.4.1 based on the feedback that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Change by-laws on release votes: 5 days instead of 7
+1 (binding) -- Aaron T. Myers Software Engineer, Cloudera On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, As discussed, I'd like to call a vote on changing our by-laws to change release votes from 7 days to 5. I've attached the change to by-laws I'm proposing. Please vote, the vote will the usual period of 7 days. thanks, Arun [main]$ svn diff Index: author/src/documentation/content/xdocs/bylaws.xml === --- author/src/documentation/content/xdocs/bylaws.xml (revision 1605015) +++ author/src/documentation/content/xdocs/bylaws.xml (working copy) @@ -344,7 +344,16 @@ pVotes are open for a period of 7 days to allow all active voters time to consider the vote. Votes relating to code changes are not subject to a strict timetable but should be -made as timely as possible./p/li +made as timely as possible./p + + ul + li strongProduct Release - Vote Timeframe/strong + pRelease votes, alone, run for a period of 5 days. All other + votes are subject to the above timeframe of 7 days./p + /li + /ul + /li + /ul /section /body -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.3.0
+1 (binding) I downloaded the source tar ball, checked signatures, built from the source, ran a few of the sample jobs on a pseudo cluster. Everything was as expected. -- Aaron T. Myers Software Engineer, Cloudera On Tue, Feb 11, 2014 at 6:49 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to get released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun PS: Thanks to Andrew, Vinod Alejandro for all their help in various release activities. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.3.0
I don't think any of what you describe below is a regression in behavior from earlier releases. The fs.defaultFS has been set to file:/// for a long time, and likewise you've similarly had to set up your YARN configs. Given that, I don't think this warrants a new RC. -- Aaron T. Myers Software Engineer, Cloudera On Wed, Feb 12, 2014 at 5:37 PM, Alejandro Abdelnur t...@cloudera.comwrote: Running pseudo cluster out of the box (expanding the binary tar, or building from source) does not work, you have to go an set the MR framework to yarn, the default FS URI to hdfs://localhost:8020, and so on. While I don't see this as showstopper (for the knowledgeable user), it will make may users to fail miserably. Plus, running a an example MR job out of the box uses the local runner. If the user does not pay attention to the output will think the job run in the cluster. Should we do a new RC fixing this? Thanks. On Wed, Feb 12, 2014 at 5:10 PM, Zhijie Shen zs...@hortonworks.com wrote: +1 (non-binding) I download the source tar ball, built from it, ran a number of MR jobs on a single-node cluster, checked the job history from job history server. On Wed, Feb 12, 2014 at 2:47 PM, Jian He j...@hortonworks.com wrote: +1 (non-binding) Built from source. Ran a few MR sample jobs on a pseudo cluster. Everything works fine. Jian On Wed, Feb 12, 2014 at 2:32 PM, Aaron T. Myers a...@cloudera.com wrote: +1 (binding) I downloaded the source tar ball, checked signatures, built from the source, ran a few of the sample jobs on a pseudo cluster. Everything was as expected. -- Aaron T. Myers Software Engineer, Cloudera On Tue, Feb 11, 2014 at 6:49 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to get released. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun PS: Thanks to Andrew, Vinod Alejandro for all their help in various release activities. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Zhijie Shen Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro
Re: Re-swizzle 2.3
I just filed an issue for the fact that browsing the FS from the NN is broken if you have a directory with the sticky bit set: https://issues.apache.org/jira/browse/HDFS-5921 I didn't set this to be targeted for 2.3 because it doesn't seem like a _blocker_ to me, but if we're not going to get 2.3 out today anyway, I'd like to put this in. It's a small fix, and since many people have the sticky bit set on /tmp, they won't be able to browse any of the FS hierarchy from the NN without this fix. -- Aaron T. Myers Software Engineer, Cloudera On Fri, Feb 7, 2014 at 12:45 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Heres what I've done: - Reverted YARN-1493,YARN-1490,YARN-1041, YARN-1166,YARN-1566,YARN-1689,YARN-1661 from branch-2.3. - Updated YARN's CHANGES.txt in trunk, branch-2 and branch-2.3. - Updated these JIRAs to have 2.4 as the fix-version. - Compiled branch-2.3. Let me know if you run into any issues caused by this revert. Thanks, +Vinod On Fri, Feb 7, 2014 at 11:41 AM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Haven't heard back from Jian. Reverting the set from branch-2.3 (only). Tx for the offline list. +Vinod On Fri, Feb 7, 2014 at 9:08 AM, Alejandro Abdelnur t...@cloudera.com wrote: Vinod, I have the patches to revert most of the JIRAs, the first batch, I'll send them off line to you. Thanks. On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli vino...@apache.orgwrote: Thanks. please post your findings, Jian wrote this part of the code and between him/me, we can take care of those issues. +1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow morning unless I hear otherwise from Jian. Thanks, +Vinod On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur t...@cloudera.com wrote: Hi Vinod, Nothing confidential, * With umanaged AMs I'm seeing the trace I've posted a couple of days ago in YARN-1577 ( https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853 ). * Also, Robert has been digging in Oozie testcases failing/getting suck with several token renewer threads, this failures happened consistently at different places around the same testcases (like some file descriptors leaking out), reverting YARN-1490 fixes the problem. The potential issue with this is that a long running client (oozie) my run into this situation thus becoming unstable. *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test hanging? After YARN-1493 YARN-1490 we have a couple of JIRAs trying to fix issues introduced by them, and we still didn't get them right. Because this, the improvements driven by YARN-1493 YARN-1490 seem that require more work before being stable. IMO, being conservative, we should do 2.3 without them and roll them with 2.4. If we want to do regular releases we will have to make this kind of calls, else we will start dragging the releases. Sounds like a plan? Thanks. On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli vino...@apache.orgwrote: Hey I am not against removing them from 2.3 if that is helpful for progress. But I want to understand what the issues are before we make that decision. There is the issue with unmanaged AM that is clearly known and I was thinking of coming to the past two days, but couldn't. What is this new issue that we (confidently?) pinned down to YARN-1490? Thanks +Vinod On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur t...@cloudera.com wrote: Thanks Robert, All, So it seems that YARN-1493 and YARN-1490 are introducing serious regressions. I would propose to revert them and the follow up JIRAs from the 2.3 branch and keep working on them on trunk/branch-2 until the are stable (I would even prefer reverting them from branch-2 not to block a 2.4 if they are not ready in time). As I've mentioned before, the list of JIRAs to revert were: YARN-1493 YARN-1490 YARN-1166 YARN-1041 YARN-1566 Plus 2 additional JIRAs committed since my email on this issue 2 days ago: *YARN-1661 *YARN-1689 (not sure if this JIRA is related in functionality to the previous ones but it is creating conflicts). I think we should hold on continuing work on top of something that is broken until the broken stuff is fixed. Quoting Arun, Committers - Henceforth, please use extreme caution while committing to branch-2.3. Please commit *only* blockers to 2.3. YARN-1661 YARN-1689 are not blockers. Unless there are objections, I'll revert all
Re: Re-swizzle 2.3
Just committed a fix for HDFS-5921 to branch-2.3. Fire away. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Feb 10, 2014 at 1:34 PM, Aaron T. Myers a...@cloudera.com wrote: OK. I think I should be able to get it in by 6pm PT, thanks to a quick +1 from Andrew, but certainly don't let it hold up the train if for some reason it takes longer than that. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Feb 10, 2014 at 12:04 PM, Arun C Murthy a...@hortonworks.comwrote: Looks like we are down to 0 blockers; I'll create rc0 tonight. ATM - Your call, you have until 6pm tonight to get this in. thanks, Arun On Feb 10, 2014, at 11:44 AM, Aaron T. Myers a...@cloudera.com wrote: I just filed an issue for the fact that browsing the FS from the NN is broken if you have a directory with the sticky bit set: https://issues.apache.org/jira/browse/HDFS-5921 I didn't set this to be targeted for 2.3 because it doesn't seem like a _blocker_ to me, but if we're not going to get 2.3 out today anyway, I'd like to put this in. It's a small fix, and since many people have the sticky bit set on /tmp, they won't be able to browse any of the FS hierarchy from the NN without this fix. -- Aaron T. Myers Software Engineer, Cloudera On Fri, Feb 7, 2014 at 12:45 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Heres what I've done: - Reverted YARN-1493,YARN-1490,YARN-1041, YARN-1166,YARN-1566,YARN-1689,YARN-1661 from branch-2.3. - Updated YARN's CHANGES.txt in trunk, branch-2 and branch-2.3. - Updated these JIRAs to have 2.4 as the fix-version. - Compiled branch-2.3. Let me know if you run into any issues caused by this revert. Thanks, +Vinod On Fri, Feb 7, 2014 at 11:41 AM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Haven't heard back from Jian. Reverting the set from branch-2.3 (only). Tx for the offline list. +Vinod On Fri, Feb 7, 2014 at 9:08 AM, Alejandro Abdelnur t...@cloudera.com wrote: Vinod, I have the patches to revert most of the JIRAs, the first batch, I'll send them off line to you. Thanks. On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli vino...@apache.orgwrote: Thanks. please post your findings, Jian wrote this part of the code and between him/me, we can take care of those issues. +1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow morning unless I hear otherwise from Jian. Thanks, +Vinod On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur t...@cloudera.com wrote: Hi Vinod, Nothing confidential, * With umanaged AMs I'm seeing the trace I've posted a couple of days ago in YARN-1577 ( https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853 ). * Also, Robert has been digging in Oozie testcases failing/getting suck with several token renewer threads, this failures happened consistently at different places around the same testcases (like some file descriptors leaking out), reverting YARN-1490 fixes the problem. The potential issue with this is that a long running client (oozie) my run into this situation thus becoming unstable. *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test hanging? After YARN-1493 YARN-1490 we have a couple of JIRAs trying to fix issues introduced by them, and we still didn't get them right. Because this, the improvements driven by YARN-1493 YARN-1490 seem that require more work before being stable. IMO, being conservative, we should do 2.3 without them and roll them with 2.4. If we want to do regular releases we will have to make this kind of calls, else we will start dragging the releases. Sounds like a plan? Thanks. On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli vino...@apache.orgwrote: Hey I am not against removing them from 2.3 if that is helpful for progress. But I want to understand what the issues are before we make that decision. There is the issue with unmanaged AM that is clearly known and I was thinking of coming to the past two days, but couldn't. What is this new issue that we (confidently?) pinned down to YARN-1490? Thanks +Vinod On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur t...@cloudera.com wrote: Thanks Robert, All, So it seems that YARN-1493 and YARN-1490 are introducing serious regressions. I would propose to revert them and the follow up JIRAs from the 2.3 branch and keep working on them on trunk/branch-2 until the are stable (I would even prefer reverting them from branch-2 not to block a 2.4 if they are not ready in time). As I've mentioned before, the list of JIRAs to revert were: YARN-1493 YARN-1490 YARN-1166 YARN-1041 YARN
Re: Re-swizzle 2.3
There's still ongoing discussion on HDFS-4858 and I don't think we should hold up 2.3.0 for that. IMO we should target that for 2.3.1 or 2.4.0. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Feb 10, 2014 at 5:53 PM, Konstantin Shvachko shv.had...@gmail.comwrote: Sorry for the last minute request. Can we add HDFS-4858 to the release, please? It solves pretty important bug related to failover. I can commit momentarily if there are no objections. Thanks, --Konstantin On Mon, Feb 10, 2014 at 4:50 PM, Aaron T. Myers a...@cloudera.com wrote: Just committed a fix for HDFS-5921 to branch-2.3. Fire away. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Feb 10, 2014 at 1:34 PM, Aaron T. Myers a...@cloudera.com wrote: OK. I think I should be able to get it in by 6pm PT, thanks to a quick +1 from Andrew, but certainly don't let it hold up the train if for some reason it takes longer than that. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Feb 10, 2014 at 12:04 PM, Arun C Murthy a...@hortonworks.com wrote: Looks like we are down to 0 blockers; I'll create rc0 tonight. ATM - Your call, you have until 6pm tonight to get this in. thanks, Arun On Feb 10, 2014, at 11:44 AM, Aaron T. Myers a...@cloudera.com wrote: I just filed an issue for the fact that browsing the FS from the NN is broken if you have a directory with the sticky bit set: https://issues.apache.org/jira/browse/HDFS-5921 I didn't set this to be targeted for 2.3 because it doesn't seem like a _blocker_ to me, but if we're not going to get 2.3 out today anyway, I'd like to put this in. It's a small fix, and since many people have the sticky bit set on /tmp, they won't be able to browse any of the FS hierarchy from the NN without this fix. -- Aaron T. Myers Software Engineer, Cloudera On Fri, Feb 7, 2014 at 12:45 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Heres what I've done: - Reverted YARN-1493,YARN-1490,YARN-1041, YARN-1166,YARN-1566,YARN-1689,YARN-1661 from branch-2.3. - Updated YARN's CHANGES.txt in trunk, branch-2 and branch-2.3. - Updated these JIRAs to have 2.4 as the fix-version. - Compiled branch-2.3. Let me know if you run into any issues caused by this revert. Thanks, +Vinod On Fri, Feb 7, 2014 at 11:41 AM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Haven't heard back from Jian. Reverting the set from branch-2.3 (only). Tx for the offline list. +Vinod On Fri, Feb 7, 2014 at 9:08 AM, Alejandro Abdelnur t...@cloudera.com wrote: Vinod, I have the patches to revert most of the JIRAs, the first batch, I'll send them off line to you. Thanks. On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli vino...@apache.orgwrote: Thanks. please post your findings, Jian wrote this part of the code and between him/me, we can take care of those issues. +1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow morning unless I hear otherwise from Jian. Thanks, +Vinod On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur t...@cloudera.com wrote: Hi Vinod, Nothing confidential, * With umanaged AMs I'm seeing the trace I've posted a couple of days ago in YARN-1577 ( https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853 ). * Also, Robert has been digging in Oozie testcases failing/getting suck with several token renewer threads, this failures happened consistently at different places around the same testcases (like some file descriptors leaking out), reverting YARN-1490 fixes the problem. The potential issue with this is that a long running client (oozie) my run into this situation thus becoming unstable. *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test hanging? After YARN-1493 YARN-1490 we have a couple of JIRAs trying to fix issues introduced by them, and we still didn't get them right. Because this, the improvements driven by YARN-1493 YARN-1490 seem that require more work before being stable. IMO, being conservative, we should do 2.3 without them and roll them with 2.4. If we want to do regular releases we will have to make this kind of calls, else we will start dragging the releases. Sounds like a plan? Thanks. On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli vino...@apache.orgwrote: Hey I am not against removing them
Re: [VOTE] Release Apache Hadoop 2.2.0
+1 (binding) Downloaded the release, built from tarball, tested a single node cluster. Everything worked as expected. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Oct 7, 2013 at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Resolved] (MAPREDUCE-5524) java.io.IOException: Task process exit with nonzero status of 255. how to fix it?
[ https://issues.apache.org/jira/browse/MAPREDUCE-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-5524. --- Resolution: Invalid Please email u...@hadoop.apache.org with this question. Apache JIRA is for reporting bugs and tracking features/improvements. It's not intended for user-level help. java.io.IOException: Task process exit with nonzero status of 255. how to fix it? --- Key: MAPREDUCE-5524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5524 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: hawkswood Task ..FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 255. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) -- This message was sent by Atlassian JIRA (v6.1#6144)
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha (RC1)
+1 (binding) I downloaded the bits, set up a 4-node cluster, and ran some example jobs. Looks good to me. -- Aaron T. Myers Software Engineer, Cloudera On Thu, Aug 15, 2013 at 10:29 PM, Konstantin Boudnik c...@apache.org wrote: All, I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1 The maven artifacts are available via repository.apache.org. The only difference between rc0 and rc1 is ASL added to releasenotes.html and updated release dates in CHANGES.txt files. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
[jira] [Created] (MAPREDUCE-5193) A few MR tests use block sizes which are smaller than the default minimum block size
Aaron T. Myers created MAPREDUCE-5193: - Summary: A few MR tests use block sizes which are smaller than the default minimum block size Key: MAPREDUCE-5193 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5193 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.5-beta Reporter: Aaron T. Myers Assignee: Aaron T. Myers HDFS-4305 introduced a new configurable minimum block size of 1MB. A few MR tests deliberately set much smaller block sizes. This JIRA is to update those tests to fix these failing tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5004) Somebody working on Genetic Algorithm library on Map Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-5004. --- Resolution: Invalid Hi Abhishek, I'm not quite sure what you were trying to get at with this JIRA, but I recommend emailing either u...@hadoop.apache.org or common-...@hadoop.apache.org with your question. Somebody working on Genetic Algorithm library on Map Reduce --- Key: MAPREDUCE-5004 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5004 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Abhishek Bajpai -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release hadoop-2.0.3-alpha
+1 (binding) I downloaded the src tar ball, built it with the native bits enabled, started up a little cluster, and ran some sample jobs. Things worked as expected. I also verified the signatures on the source artifact. I did bump into one little issue, but I don't think it should be considered a blocker. When I first tried to start up the RM, it failed to start with this error: 13/02/08 16:00:31 FATAL resourcemanager.ResourceManager: Error starting ResourceManager java.lang.IllegalStateException: Queue configuration missing child queue names for root at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:710) And then this on shutdown: 13/02/08 16:00:31 INFO service.CompositeService: Error stopping ResourceManager java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.stop(ResourceManager.java:590) at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) Presumably this is because I don't have the CapacityScheduler queues configured at all, and the default scheduler is now the CapacityScheduler. To work around this for my testing, I switched to the FairScheduler and the RM came up just fine. -- Aaron T. Myers Software Engineer, Cloudera On Wed, Feb 6, 2013 at 7:59 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like to release. This release contains several major enhancements such as QJM for HDFS HA, multi-resource scheduling for YARN, YARN ResourceManager restart etc. Also YARN has achieved significant stability at scale (more details from Y! folks here: http://s.apache.org/VYO). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/ The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
[jira] [Resolved] (MAPREDUCE-4390) java.io.IOException: File /user/XXXX/QuasiMonteCarlo_TMP_3_141592654/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datano
[ https://issues.apache.org/jira/browse/MAPREDUCE-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-4390. --- Resolution: Invalid Hi Srikanth, the Apache JIRA is for tracking established bugs or improvements. This looks like an operational issue. Specifically, it looks like you don't have any DNs running. I recommend you email mapreduce-u...@hadoop.apache.org to get more help with this. java.io.IOException: File /user//QuasiMonteCarlo_TMP_3_141592654/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. - Key: MAPREDUCE-4390 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4390 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples, job submission, mrv2 Affects Versions: 0.23.0 Environment: Ubuntu Server 11.04 Reporter: srikanth ayalasomayajulu Labels: hadoop Fix For: 0.23.0 Original Estimate: 2h Remaining Estimate: 2h Tried to run an example program on hadoop0.23.0 and getting the following error. error: java.io.IOException: File /user/X/QuasiMonteCarlo_TMP_3_141592654/in/part0 could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1181) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1486) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:390) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:365) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1490) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1486) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1152) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4391) datanode.DataNode (DataNode.java:handshake(820)) - Problem connecting to server: master/192.168.100.140:9000
[ https://issues.apache.org/jira/browse/MAPREDUCE-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-4391. --- Resolution: Invalid Hi Srikanth, Apache JIRA is for tracking confirmed bugs or improvements. This issue looks like a misconfiguration. You'll probably get more help by emailing hdfs-u...@hadoop.apache.org with a description of your issue. datanode.DataNode (DataNode.java:handshake(820)) - Problem connecting to server: master/192.168.100.140:9000 Key: MAPREDUCE-4391 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4391 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Environment: Ubuntu Server 11.04, Hadoop 0.23.0 Reporter: srikanth ayalasomayajulu Labels: datanode, hadoop Fix For: 0.23.0 Original Estimate: 2h Remaining Estimate: 2h datanode cannot able to connect to namenode, and in turn resulting in future errors during running examples. 2012-07-04 15:25:09,636 WARN datanode.DataNode (DataNode.java:handshake(820)) - Problem connecting to server: master/192.168.100.140:9000 2012-07-04 15:25:15,638 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 0 time(s). 2012-07-04 15:25:16,640 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 1 time(s). 2012-07-04 15:25:17,642 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 2 time(s). 2012-07-04 15:25:18,643 INFO ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: master/192.168.100.140:9000. Already tried 3 time(s). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: validating user IDs
-hdfs-dev@ +mapreduce-dev@ Moving to the more-relevant mapreduce-dev. -- Aaron T. Myers Software Engineer, Cloudera On Mon, Jun 11, 2012 at 4:12 PM, Alejandro Abdelnur t...@cloudera.comwrote: Colin, Would be possible using some kind of cmake config magic to set a macro to the current OS limit? Even if this means detecting the OS version and assuming its default limit. thx On Mon, Jun 11, 2012 at 3:57 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: Hi all, I recently pulled the latest source, and ran a full build. The command line was this: mvn compile -Pnative I was confronted with this: [INFO] Requested user cmccabe has id 500, which is below the minimum allowed 1000 [INFO] FAIL: test-container-executor [INFO] [INFO] 1 of 1 test failed [INFO] Please report to mapreduce-dev@hadoop.apache.org [INFO] [INFO] make[1]: *** [check-TESTS] Error 1 [INFO] make[1]: Leaving directory `/home/cmccabe/hadoop4/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/container-executor' Needless to say, it didn't do much to improve my mood. I was even less happy when I discovered that -DskipTests has no effect on native tests (they always run.) See HADOOP-8480. Unfortunately, it seems like this problem is popping up more and more in our native code. It first appeared in test-task-controller (see MAPREDUCE-2376) and then later in test-container-executor (HADOOP-8499). The basic problem seems to be the hardcoded assumption that all user IDs below 1000 are system IDs. It is true that there are configuration files that can be changed to alter the minimum user ID, but unfortunately these configuration files are not used by the unit tests. So anyone developing on a platform where the user IDs start at 500 is now a second-class citizen, unable to run unit tests. This includes anyone running Red Hat, MacOS, Fedora, etc. Personally, I can change my user ID. It's a time-consuming process, because I need to re-uid all files, but I can do it. This luxury may not be available to everyone, though-- developers who don't have root on their machines, or are using a pre-assigned user ID to connect to NFS come to mind. It's true that we could hack around this with environment variables. It might even be possible to have Maven set these environment variables automatically from the current user ID. However, the larger question I have here is whether this UID validation scheme even makes any sense. I have a user named nobody whose user ID is 65534. Surely I should not be able to run map-reduce jobs as this user? Yet, under the current system, I can do exactly that. The root of the problem seems to be that there is both a default minimum and a default maximum for automatic user IDs. This configuration seems to be stored in /etc/login.defs. On my system, it has: SYSTEM_UID_MIN100 SYSTEM_UID_MAX499 UID_MIN 500 UID_MAX 6 So that means that anything over 6 (like nobody) is not considered a valid user ID for regular users. We could potentially read this file (at least on Linux) and get more sensible defaults. I am also curious if we could simply check whether the user we're trying to run the job as has a valid login shell. System users are almost always set to have a login shell of /bin/false or /sbin/nologin. Thoughts? Colin -- Alejandro
[jira] [Created] (MAPREDUCE-4170) Move sleep and fail jobs from tests module to examples module
Aaron T. Myers created MAPREDUCE-4170: - Summary: Move sleep and fail jobs from tests module to examples module Key: MAPREDUCE-4170 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4170 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples, test Affects Versions: 2.0.0 Reporter: Aaron T. Myers Priority: Minor The sleep job used to be in the examples jar in MR1. I'm not quite sure when, but the sleep job has been moved to the tests module/jar in MR2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x
Hi Milind, On Tue, Apr 3, 2012 at 11:27 AM, milind.bhandar...@emc.com wrote: Here you say: Essentially 'trunk' is where incompatible changes *may* be committed (in future). We should allow for that. What I believe Arun is alluding to here is that we expect for compatibility to be maintained for the lifetime of a major release branch. On another thread, responding to Avner (re: MAPREDUCE-4049?) you say, We do expect 'new features' to make it to trunk before we can commit to either branch-1 or branch-2. Which one is it ? These two statements aren't mutually exclusive. We require that all new features go to trunk first so as to ensure that future releases are supersets of the functionality of previous releases, except in the case of explicit deprecation. Only once it's committed to trunk may it be back-ported to an earlier branch. Do you expect that new features will always remain compatible ? Not necessarily, but only if a feature is compatible may it be back-ported to major release branches. -- Aaron T. Myers Software Engineer, Cloudera
Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x
On Tue, Apr 3, 2012 at 2:37 PM, milind.bhandar...@emc.com wrote: What would be guideline for a new feature, such as https://issues.apache.org/jira/browse/MAPREDUCE-4049, which maintains compatibility for 1.x, but is not relevant to trunk, because the codebases have completely diverged, so cannot be committed to trunk ? Are you sure this isn't relevant to trunk? The target versions field of that JIRA lists both 1.0.x and 0.24.0 (trunk.) In the latest comment on that JIRA, the author appears to intend to do this work for both trunk and 1.0: I want to have the described plugin-ability (desired with same interface) for all future versions of Hadoop (as mentioned in the Target Version/s field). snip On the first phase, I am focusing on the existing 1.0 branch as I know it. In parallel, I'll try to learn what exists in 0.23 -- Aaron T. Myers Software Engineer, Cloudera
Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x
If that's the case then there doesn't seem to be any question here. The feature is in trunk, and an implementation could be done for an older release branch that would be compatible with that branch. Sure, the code to implement the feature is quite different between the two branches, but trunk will remain a superset of the functionality of the past release, so no issue. -- Aaron T. Myers Software Engineer, Cloudera On Tue, Apr 3, 2012 at 3:14 PM, milind.bhandar...@emc.com wrote: To my knowledge, shuffle is already pluggable in 0.23 onwards, as long as it is used only by mapreduce framework. That's why Avner says : In parallel, I'll try to *learn what exists* in 0.23. (Emphasize my own.) That's why I was wondering about the insistence of committing to trunk first. - Milind --- Milind Bhandarkar Greenplum Labs, EMC (Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any organization, past or present, the author might be affiliated with.) On 4/3/12 2:44 PM, Aaron T. Myers a...@cloudera.com wrote: On Tue, Apr 3, 2012 at 2:37 PM, milind.bhandar...@emc.com wrote: What would be guideline for a new feature, such as https://issues.apache.org/jira/browse/MAPREDUCE-4049, which maintains compatibility for 1.x, but is not relevant to trunk, because the codebases have completely diverged, so cannot be committed to trunk ? Are you sure this isn't relevant to trunk? The target versions field of that JIRA lists both 1.0.x and 0.24.0 (trunk.) In the latest comment on that JIRA, the author appears to intend to do this work for both trunk and 1.0: I want to have the described plugin-ability (desired with same interface) for all future versions of Hadoop (as mentioned in the Target Version/s field). snip On the first phase, I am focusing on the existing 1.0 branch as I know it. In parallel, I'll try to learn what exists in 0.23 -- Aaron T. Myers Software Engineer, Cloudera
Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x
Hey Arun, On Wed, Mar 28, 2012 at 12:28 AM, Arun C Murthy a...@hortonworks.com wrote: Done, all clear; I've also created jira revisions. Please let me know if you find any issues. Thanks a lot for making these changes. Two questions: - Now that we've renamed branch-0.23 to branch-2, and since there is as yet no branch-0.23.3, what should be done with JIRAs marked with the 0.23.3 version? Perhaps all of these should be updated to 2.0.0? - We still have the JIRA version 0.24.0, which is presumably still representative of trunk. Given that we will likely never release an 0.24.0, should we rename this version in JIRA as well? -- Aaron T. Myers Software Engineer, Cloudera
Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x
On Wed, Mar 28, 2012 at 11:53 AM, Todd Lipcon t...@cloudera.com wrote: Proposal: is it possible to call the JIRA fixVersion trunk, and then when we branch off trunk to make a release, rename it at that point to 2.1 or 3.0 or whatever it gets called? I like this idea. Just to be clear, I think the exact workflow would be: 1. Set the version fields to trunk if you're not committing the JIRA to any current versioned branch. 2. When a new release branch is made off of trunk, rename the trunk JIRA version to whatever the appropriate version number is. 3. At the same time as (2), create a new JIRA version also called trunk. 4. Go to 1. Is this what you were thinking, Todd? -- Aaron T. Myers Software Engineer, Cloudera
Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x
Hi Owen, On Wed, Mar 28, 2012 at 12:39 PM, Owen O'Malley omal...@apache.org wrote: Let's imagine that we already had a 2.0.0 release. Now we want to add features like HA. The only place to put that is in 2.1.0. On the other hand, you don't want to pull *ALL* of the changes from trunk. That is way too much scope. So the RM of the 2 branch needs to make the call of what should be 2.1 vs 3.0. I don't think anyone disagrees with this line of reasoning. It's certainly up to the RM what gets included in branch-2 and hence what gets put up for release votes under the 2.y.z version numbers. I don't think Todd was suggesting we rename the JIRA version 0.24.0 to 2.1.0. But, the question still remains of how to refer to the branch trunk in JIRA. I don't think it should be called 3.0.0, as that's not necessarily the next release that will come off of it, and using a version number for trunk that changes from time to time has other downsides as I described in my response to Arun. Given this, do you object to renaming the JIRA fix version that refers to the branch trunk to trunk ? -- Aaron T. Myers Software Engineer, Cloudera
[jira] [Created] (MAPREDUCE-3934) RetriableCommand in distcp should not use RetryPolicy classes
RetriableCommand in distcp should not use RetryPolicy classes - Key: MAPREDUCE-3934 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3934 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.24.0 Reporter: Aaron T. Myers It's generally not kosher for RetriableCommand to be using the RetryPolicy classes at all, since they're not really intended to be used except by RetryInvocationHandler. See HADOOP-8116 for an example of why. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3579) ConverterUtils should not include a port in a path for a URL with no port
ConverterUtils should not include a port in a path for a URL with no port - Key: MAPREDUCE-3579 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3579 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.23.1, 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers In {{ConverterUtils#getPathFromYarnURL}}, it's incorrectly assumed that if a URL includes a valid host it must also include a valid port. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Application/Job is hanging! validate
This sounds like https://issues.apache.org/jira/browse/MAPREDUCE-2324manifested in MR2 to me. Should we perhaps update the affects versions field there? -- Aaron T. Myers Software Engineer, Cloudera On Mon, Oct 17, 2011 at 9:55 AM, Mahadev Konar maha...@hortonworks.comwrote: Kamesh, This is what will happen if you do not have enough resource in the cluster to run a job. Though there could be an improvement that a job fails with not enough resource in the cluster but thats not what happens right now. thanks mahadev On Mon, Oct 17, 2011 at 6:25 AM, Kamesh kames...@imaginea.com wrote: Hi All, I submitted an application to YARN. But there are no containers available. In this case job got hanged indefinitely. I reproduced this in a pseudo cluster mode with the following steps. Steps to reproduce 1) let the amount of memory the MR app master needs be 2GB. 2) let the Nodemanager's capability be 1GB. In the above case, no of available containers is 0 and so the job could not able to get a chance to run due to containers unavailability. Please validate the above case. -- ThanksRegards, Bh.V.S.Kamesh.
Re: Application/Job is hanging! validate
On Mon, Oct 17, 2011 at 11:08 AM, Aaron T. Myers a...@cloudera.com wrote: This sounds like https://issues.apache.org/jira/browse/MAPREDUCE-2324manifested in MR2 to me. Should we perhaps update the affects versions field there? Never mind. I just found MAPREDUCE-2723https://issues.apache.org/jira/browse/MAPREDUCE-2723 . -- Aaron T. Myers Software Engineer, Cloudera
[jira] [Created] (MAPREDUCE-2934) MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process
MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup process -- Key: MAPREDUCE-2934 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2934 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.24.0 Once HADOOP-7607 goes in, {{ProtoOverHadoopRpcEngine.stopProxy}} will need to be removed or at least have its {{@Override}} annotation removed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Mavenizing the HDFS build
I believe these JIRAs should address those issues: https://issues.apache.org/jira/browse/HADOOP-7563 https://issues.apache.org/jira/browse/HDFS-2277 -- Aaron T. Myers Software Engineer, Cloudera On Mon, Aug 22, 2011 at 1:16 PM, Thomas Graves tgra...@yahoo-inc.comwrote: What is the expected way to run from the tarballs? It seems if I just untar them so they are in their separate directories ( hadoop-common-0.23.0-SNAPSHOT/, hadoop-hdfs-0.23.0-SNAPSHOT/, hadoop-mapreduce-1.0-SNAPSHOT/), setup the ENV vars for HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, etc.. It won't run like I would expect it to. Certain things like hadoop-config.sh are in libexec and it expects them in bin, things in hdfs bin it looks for in common bin, things are in sbin but it wants them in bin, it now can't seem to find the hdfs jars, etc.. Hopefully I'm missing something simple to make this work? Thanks, Tom Graves On 8/19/11 12:55 PM, Tom White t...@cloudera.com wrote: HDFS-2096 is now committed to trunk. The instructions for building HDFS can be found in the top-level BUILDING.txt file. I added a script to https://issues.apache.org/jira/browse/HADOOP-7500 to help with migrating HDFS patches to the new layout. There are a few follow-up patches that need doing soon (e.g. HADOOP-7498, HADOOP-7496, MAPREDUCE-2856), but these shouldn't stop folks from doing development as usual. Thanks to everyone who helped with this! Cheers, Tom On Thu, Aug 18, 2011 at 11:30 AM, Tom White t...@cloudera.com wrote: Now that MR-279 has been merged into trunk, I plan to commit the HDFS mavenization changes tomorrow (Friday) at 9am PDT. Cheers, Tom On Mon, Aug 15, 2011 at 1:24 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Tom. I'm running the final set of tests with the 'MR-279 rebased on trunk' and should be done by tmrw. Also, can you guys please ensure that secure HDFS works after mvn'ization? thanks, Arun On Aug 13, 2011, at 9:39 PM, Tom White wrote: Hi Arun, I'm fine with that. When do you expect to start the vote? Cheers, Tom On Fri, Aug 12, 2011 at 11:41 PM, Arun C Murthy a...@hortonworks.com wrote: Hi Tom, Can I request you to wait on this commit until we merge MR-279? As Vinod pointed out in his mail to mapreduce-dev@ we are very close to getting the merge done. We should call a vote asap. By holding off it the mvn patch it will save us a bit of time - we spent at more than a couple of days on resolving after the common mvn'ization. Thanks for understanding. Arun On Aug 12, 2011, at 4:18 PM, Tom White wrote: The work in https://issues.apache.org/jira/browse/HDFS-2096 is ready to be committed, so unless there are any objections I will do so on Monday at 5pm UTC (that's 10am PDT, http://s.apache.org/o6F). I'll also create a script to convert patches to the new layout, and switch over the Jenkins jobs that test and build HDFS. Cheers, Tom
Pre-commit Jenkins job disabled
https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/ Is there a reason the M/R pre-commit build is disabled, while Common and HDFS are enabled? Is this a vestige of the build slaves being offline? -- Aaron T. Myers Software Engineer, Cloudera
[jira] [Resolved] (MAPREDUCE-2109) Add support for reading multiple hadoop delegation token files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-2109. --- Resolution: Won't Fix Add support for reading multiple hadoop delegation token files -- Key: MAPREDUCE-2109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2109 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Affects Versions: 0.22.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: mapreduce-2109.0.txt, mapreduce-2109.1.txt, mapreduce-2109.2.txt, mapreduce-2109.3.txt This is the MR part of HADOOP-6988. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-2473) MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
MR portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent - Key: MAPREDUCE-2473 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2473 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Affects Versions: 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.23.0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Unable to build MR-279 due to unresolved ivy/maven dependencies ?
On Thu, Apr 14, 2011 at 11:31 AM, Arun C Murthy ar...@yahoo-inc.com wrote: 'ant -Dresolvers-=internal Sorry if I'm being pedantic, but it's actually ant -Dresolvers=internal. (Arun had an extra - in there.) -- Aaron T. Myers Software Engineer, Cloudera
[jira] Resolved: (MAPREDUCE-2276) Fix build failure introduced by HDFS-1547
[ https://issues.apache.org/jira/browse/MAPREDUCE-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved MAPREDUCE-2276. --- Resolution: Duplicate Resolving as duplicate of https://issues.apache.org/jira/browse/HDFS-1585 Fix build failure introduced by HDFS-1547 - Key: MAPREDUCE-2276 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2276 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.23.0 MiniDFSCluster#startDataNodes() method signature changes introduced by HDFS-1547 breaks the mapreduce build -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-2067) Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies
Distinct minicluster services (e.g. NN and JT) overwrite each other's service policies -- Key: MAPREDUCE-2067 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2067 Project: Hadoop Map/Reduce Issue Type: Bug Components: security Reporter: Aaron T. Myers MR portion of HADOOP-6951. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.