[jira] [Commented] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427134#comment-13427134 ] Alejandro Abdelnur commented on MAPREDUCE-4495: --- As I've mentioned to Arun during our chat, I think that, at least initially, the Workflow/DAG AM should come to life within Hadoop. Later we can, as it has been done with other projects, move it to a separate project once it gets traction/adoption. Thus we would not get distracted with the bureaucracy required to bootstrap an incubator project. Following are my reasons why Workflow/DAG AM should be (initially) part of Hadoop: * It is meant primarily to run workflows of MR jobs. It is not the intention implementing a general purpose workflow engine. * It is server version of the JobController. * It will most likely require changes in the MR AM (making it thread safe and multi MR job). Being in Hadoop will create the synergy to make this changes rapidly. * It may require changes in the YARN APIs. * Being in Hadoop it can be easily consumed by Pig/Hive/Oozie and MR developers. And if those projects require special actions other than MR jobs they can be easily added as the AM runs in user space via plugins. * Doing it in Oozie means that Pig/Hive would not be able to consume it easily as it would create a circular dependency among those projects. > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427128#comment-13427128 ] Alejandro Abdelnur commented on MAPREDUCE-4334: --- Arun, if somebody is willing to install cgrulesengd/cgexec in the nodes then there is no need for super-user privileges;p plus, any CE could be used (unmodified) with a ResourceEnforcer injecting cgexec to the launcher invocation. This has also the benefit that if we add more resource dimensions (last bullet above), CE implementations would not need to change, only the ResourceEnforcer. Which means, no code duplication, the cgroup configuration logic lives once, in the ResourceEnforcer, as opposed to every CE that wants to support cgroups. Finally, I like the fact that with the ResourceEnforcer we are doing a clean separation of responsibilities between the ResourceEnforcer (configures) and the ContainerExecutor (executes), IMO this separation will simplify making improvements in each one of them without risk of mixing these 2 responsibilities. > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-executor-v1.patch, > MAPREDUCE-4334-executor-v2.patch, MAPREDUCE-4334-executor-v3.patch, > MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, > mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427120#comment-13427120 ] Arun C Murthy commented on MAPREDUCE-4334: -- Alejandro - I'm thinking that since *only* LCE can use cgroups (due to necessary super-user privs etc.), it's simpler to do minimal changes to LCE to create/encapsulate into cgroups. Thoughts? > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-executor-v1.patch, > MAPREDUCE-4334-executor-v2.patch, MAPREDUCE-4334-executor-v3.patch, > MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, > mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427111#comment-13427111 ] eric baldeschwieler commented on MAPREDUCE-4495: +1 Arun, Chris - Yarn is about opening up Hadoop to allow a lot more innovation on open APIs. The whole point of open APIs is to let lots of different people try lots of different things in parallel without having to get those things added to the Hadoop core. So go build something excellent, share it and build a community. Use GitHub, sourceforge, apache extras or start an incubator project. You don't need our approval and its not fair or scalable to ask the Yarn folks to get involved in supporting everyone's ideas of what might be interesting to build on top of Yarn. Everyone building a tomcat app doesn't want or get to check it into the tomcat project. Another idea is to include this in Oozie if it fits well there... If that would let us break oozie down into a scheduler and a workflow library (two sub-projects), that would be a very cool refactoring of oozie. Look at the OpenMPI work... They are proposing to add their AM to MPI. > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427106#comment-13427106 ] Harsh J commented on MAPREDUCE-3223: Can someone look into this soon please? Helps avoid quite a few warnings and removes stale docs. Todd/Arun/others? > Remove MR1 configs from mapred-default.xml > -- > > Key: MAPREDUCE-3223 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: mr-3223.txt, mr-3223.txt > > > All of the MRv1 configs are still in mapred-default.xml. This is confusing > when trying to make config changes. Since a lot of the input/output format > tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml > for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4068) Jars in lib subdirectory of the submittable JAR are not added to the classpath
[ https://issues.apache.org/jira/browse/MAPREDUCE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427096#comment-13427096 ] Radim Kolar commented on MAPREDUCE-4068: I do not see it either. Its major problem while migrating from mrv1. Libs in lib/ are used only in hadoop driver during job submission, but not in map/reduce YarnChild.java > Jars in lib subdirectory of the submittable JAR are not added to the classpath > -- > > Key: MAPREDUCE-4068 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4068 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ahmed Radwan > Fix For: 0.23.2 > > > Prior to hadoop 0.23, users could add third party jars to the lib > subdirectory of the submitted job jar and they become available in the task's > classpath. I see this functionality was in TaskRunner.java, but I can't see > similar functionality in hadoop 0.23 (neither in MapReduceChildJVM.java nor > other places). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427052#comment-13427052 ] Arun C Murthy commented on MAPREDUCE-3289: -- +1 for the trunk patch, looks good - thanks Todd & Sid. And +1 for Sid's comment that we should keep trunk and branch-1 in sync. > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, > MR3289_trunk_3.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427044#comment-13427044 ] Hadoop QA commented on MAPREDUCE-3289: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538856/MAPREDUCE-3289.branch-1.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2698//console This message is automatically generated. > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, > MR3289_trunk_3.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427041#comment-13427041 ] Brandon Li commented on MAPREDUCE-3289: --- Updated the branch-1 patch to use the same parameter names as trunk patch. > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, > MR3289_trunk_3.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated MAPREDUCE-3289: -- Attachment: MAPREDUCE-3289.branch-1.patch > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MAPREDUCE-3289.branch-1.patch, MR3289_trunk.txt, MR3289_trunk_2.txt, > MR3289_trunk_3.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427033#comment-13427033 ] Hadoop QA commented on MAPREDUCE-3289: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538853/MR3289_trunk_3.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2697//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2697//console This message is automatically generated. > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3289: -- Attachment: MR3289_trunk_3.txt Fixes the findbugs warnings. I've left the configs in ShuffleHandler itself. MRConfig doesn't seem like the right place - since Shuffle is an Aux Service for yarn, which doesn't necessarily need MR classes to run. Regarding the parameter names - I think it's better to use the same in both patches, as against introducing a parameter in branch-1 and deprecating it immediately in the the branch-2 patch. Brandon, could you please make this change in the branch-1 patch you had posted. > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3289: -- Status: Patch Available (was: Open) > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MR3289_trunk.txt, MR3289_trunk_2.txt, MR3289_trunk_3.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3289: -- Status: Open (was: Patch Available) > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427020#comment-13427020 ] Arun C Murthy commented on MAPREDUCE-4327: -- The findbugs warnings are all from FairScheduler, sigh, I thought we had fixed them - anyway, unrelated to this patch. > Enhance CS to schedule accounting for both memory and cpu cores > --- > > Key: MAPREDUCE-4327 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, resourcemanager, scheduler >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, > MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, MAPREDUCE-4327.patch, > MAPREDUCE-4327.patch, MAPREDUCE-4327.patch > > > With YARN being a general purpose system, it would be useful for several > applications (MPI et al) to specify not just memory but also CPU (cores) for > their resource requirements. Thus, it would be useful to the > CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427018#comment-13427018 ] Hadoop QA commented on MAPREDUCE-4327: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538848/MAPREDUCE-4327.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2696//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2696//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2696//console This message is automatically generated. > Enhance CS to schedule accounting for both memory and cpu cores > --- > > Key: MAPREDUCE-4327 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, resourcemanager, scheduler >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, > MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, MAPREDUCE-4327.patch, > MAPREDUCE-4327.patch, MAPREDUCE-4327.patch > > > With YARN being a general purpose system, it would be useful for several > applications (MPI et al) to specify not just memory but also CPU (cores) for > their resource requirements. Thus, it would be useful to the > CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427015#comment-13427015 ] Andrew Ferguson commented on MAPREDUCE-4327: @acmurthy: you bet! I should have time this week to read this over. also, I'm happy to port pieces of my ginormous patch over to this if you'd like -- while the majority of the patch I posted is test cases (which may or may not match the semantics of your DRF implementation due to decisions about edge cases), other pieces such as the FIFO support, the web GUI, and the metrics code might save you some time. cheers, Andrew > Enhance CS to schedule accounting for both memory and cpu cores > --- > > Key: MAPREDUCE-4327 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, resourcemanager, scheduler >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, > MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, MAPREDUCE-4327.patch, > MAPREDUCE-4327.patch, MAPREDUCE-4327.patch > > > With YARN being a general purpose system, it would be useful for several > applications (MPI et al) to specify not just memory but also CPU (cores) for > their resource requirements. Thus, it would be useful to the > CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427004#comment-13427004 ] Arun C Murthy commented on MAPREDUCE-4495: -- Thanks Chris! I'm happy to participate - others like Bobby & Mayank would be interested too I bet! > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4327: - Status: Patch Available (was: Open) > Enhance CS to schedule accounting for both memory and cpu cores > --- > > Key: MAPREDUCE-4327 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, resourcemanager, scheduler >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, > MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, MAPREDUCE-4327.patch, > MAPREDUCE-4327.patch, MAPREDUCE-4327.patch > > > With YARN being a general purpose system, it would be useful for several > applications (MPI et al) to specify not just memory but also CPU (cores) for > their resource requirements. Thus, it would be useful to the > CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4327) Enhance CS to schedule accounting for both memory and cpu cores
[ https://issues.apache.org/jira/browse/MAPREDUCE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4327: - Attachment: MAPREDUCE-4327.patch Thanks for reviews Bobby. I've incorporated all except the CPU one - not sure if fraction is the right one to go for right now... Andrew - if you have time, could you pls take a look too? Thanks. > Enhance CS to schedule accounting for both memory and cpu cores > --- > > Key: MAPREDUCE-4327 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4327 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, resourcemanager, scheduler >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: MAPREDUCE-4327-v2.patch, MAPREDUCE-4327-v3.patch, > MAPREDUCE-4327-v4.patch, MAPREDUCE-4327-v5.patch, MAPREDUCE-4327.patch, > MAPREDUCE-4327.patch, MAPREDUCE-4327.patch > > > With YARN being a general purpose system, it would be useful for several > applications (MPI et al) to specify not just memory but also CPU (cores) for > their resource requirements. Thus, it would be useful to the > CapacityScheduler to account for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426992#comment-13426992
]
Hudson commented on MAPREDUCE-4504:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #2567 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2567/])
MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans via
tgraves) (Revision 1368294)
Result = FAILURE
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368294
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/SortValidator.java
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426978#comment-13426978 ] Alejandro Abdelnur commented on MAPREDUCE-4334: --- I like to introduce the ResourceEnforcer interface for the following reasons: * It provides clean lifecycle hooks for initializing/configuring/cleanup cgroups, leaving to the LCE just the the actual binding. * It will work with multiple container executors as oposed to LCE only. * Makes the changes in the LCE minimal (IMO, the less logic with put in native code the better). * taskset could easily be implemented as a ResourceEnforcer. * If we eventually want to control other resources via cgroups (such as memory/disk/network), only the ResourceEnforcer would require changes. Fair enough? > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-executor-v1.patch, > MAPREDUCE-4334-executor-v2.patch, MAPREDUCE-4334-executor-v3.patch, > MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, > mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426979#comment-13426979 ] Chris A. Mattmann commented on MAPREDUCE-4495: -- +1 Arun, that's my thinking here too. The ASF has shown a movement away from umbrella projects, and I think one great thing to consider here would be to start a new project in the Incubator for the Workflow App Master in the Incubator. Perhaps Alejandro, and Arun and others interested could make an Apache Incubator proposal, see here: http://incubator.apache.org/guides/proposal.html > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426970#comment-13426970
]
Hudson commented on MAPREDUCE-4504:
---
Integrated in Hadoop-Common-trunk-Commit #2549 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2549/])
MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans via
tgraves) (Revision 1368294)
Result = SUCCESS
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368294
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/SortValidator.java
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426964#comment-13426964
]
Hudson commented on MAPREDUCE-4504:
---
Integrated in Hadoop-Hdfs-trunk-Commit #2614 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2614/])
MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans via
tgraves) (Revision 1368294)
Result = SUCCESS
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1368294
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/SortValidator.java
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426961#comment-13426961 ] Arun C Murthy commented on MAPREDUCE-4334: -- Thanks tucu, this is getting close. Please help me understand if the following (simpler) proposal will work: # NM calls LCE.launchContainer with the cpu-set. # LCE will create the necessary cgroup if necessary # LCE will launch the process within the cgroup Pros: This way, we avoid new interfaces such as ResourceEnforcer and we can also use taskset if necessary. Taskset should also work for DefaultContianerExecutor. Thoughts? > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-executor-v1.patch, > MAPREDUCE-4334-executor-v2.patch, MAPREDUCE-4334-executor-v3.patch, > MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, > mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated MAPREDUCE-4504:
-
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
3.0.0
0.23.3
Status: Resolved (was: Patch Available)
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426957#comment-13426957 ] Alejandro Abdelnur commented on MAPREDUCE-4334: --- I was chatting offline with Arun about this JIRA. His key concern is that it should be possible to use cgroups without requiring the installation of additional packages and extra OS configuration. As the LinuxContainerExecutor already runs as root, we can leverage that to create the cgroup mounts. This means that the LinuxContainerExecutor is required to use cgroups with zero configuration. While typically the LinuxContainerExecutor is used in secure clusters, still it can be used in non-secure cluster always running as the mapred user (which would be the equivalent of the DefaultContainerExecutor). Given this how about the following proposal? This approach will not depend on cgexec binary being installed. * The LinuxContainerExecutor would have 2 new options. ** --cgroupsinit : This option will be used for initialization. When invoked with this option, the LCE will create the cgroup mount point would and give owmership of it to the yarn user. Then it will complete its execution. ** --cgroup : This option will be used for launching containers. When invoked with this option, the LCE will add the process to specified cgroup paramerer. * The ResourceEnforcer will have the following methods (exactly as in the latest patch): ** init(): called when the RM is initialized. ** preExecute(containerId, Resource): called before launching the container. ** wrapCommand(containerId, command): augments the execution command line before launching. ** postExecute(containerId): called after launching the container. * A default implementation of the ResourceEnforcer will do NOPs. * The CgroupsResourceEnforcer implementation will do the following: ** init(): call LCE --cgroupsinit ** preExecute(containerId, Resource): configure the cgroup with the assigned cpu resources. ** wrapCommand(containerId, command): augments regular LCE invocation with the -cgroup option. ** postExecute(containerId): any necessary cgroup clean up. > Add support for CPU isolation/monitoring of containers > -- > > Key: MAPREDUCE-4334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Arun C Murthy >Assignee: Andrew Ferguson > Attachments: MAPREDUCE-4334-executor-v1.patch, > MAPREDUCE-4334-executor-v2.patch, MAPREDUCE-4334-executor-v3.patch, > MAPREDUCE-4334-executor-v4.patch, MAPREDUCE-4334-pre1.patch, > MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, > MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch, > MAPREDUCE-4334-v1.patch, MAPREDUCE-4334-v2.patch, > mapreduce-4334-design-doc-v2.txt, mapreduce-4334-design-doc.txt > > > Once we get in MAPREDUCE-4327, it will be important to actually enforce > limits on CPU consumption of containers. > Several options spring to mind: > # taskset (RHEL5+) > # cgroups (RHEL6+) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426955#comment-13426955 ] Arun C Murthy commented on MAPREDUCE-4495: -- I had a brief discussion with Alejandro about this, for full disclosure I'll post it up here. I'll let Alejandro post his response of course. One of my concerns accepting this as a new module is that we are in the danger of turning YARN into an umbrella project here. The ASF is, very rightly, concerned about this. Hadoop YARN is, and should remain, merely as the framework. Over time we will, if successful, have several applications. We cannot become an aggregation project for them - this would rightly get us in the awkward situation of being a disparate set of communities. Hadoop already went through this with HBase, Zookeeper, Hive, Pig etc. and we have since remedied the concerns of the ASF by moving them out as independent TLPs. Furthermore, if the DAG-AM is to be successful it will need to churn fast in the early days to react to requirements of communities such as Pig, Hive, Oozie etc. and you don't want to be tied to release schedules of Hadoop... Also, technically, this project, if housed in Hadoop, will forever be limited to merely being able to run MapReduce jobs as part of the workflow and rules out Pig, Hive etc. since we cannot espouse a dependency on those projects. Hence, my suggestion is that we consider either starting this in Oozie (I'd love to start contributing to Oozie via this route) or we start this as a standalone project in either Apache Incubator or Apache Extras. Thoughts? PS: We added DistShell merely as an *example* application and if a community develops around it, I'm happy to support moving that out too. > Workflow Application Master in YARN > --- > > Key: MAPREDUCE-4495 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Affects Versions: 2.0.0-alpha >Reporter: Bo Wang >Assignee: Bo Wang > > It is useful to have a workflow application master, which will be capable of > running a DAG of jobs. The workflow client submits a DAG request to the AM > and then the AM will manage the life cycle of this application in terms of > requesting the needed resources from the RM, and starting, monitoring and > retrying the application's individual tasks. > Compared to running Oozie with the current MapReduce Application Master, > these are some of the advantages: > - Less number of consumed resources, since only one application master will > be spawned for the whole workflow. > - Reuse of resources, since the same resources can be used by multiple > consecutive jobs in the workflow (no need to request/wait for resources for > every individual job from the central RM). > - More optimization opportunities in terms of collective resource requests. > - Optimization opportunities in terms of rewriting and composing jobs in the > workflow (e.g. pushing down Mappers). > - This Application Master can be reused/extended by higher systems like Pig > and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426954#comment-13426954
]
Thomas Graves commented on MAPREDUCE-4504:
--
+1 looks good, thanks Bobby!
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426910#comment-13426910 ] Hadoop QA commented on MAPREDUCE-3289: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538835/MR3289_trunk_2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2695//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2695//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2695//console This message is automatically generated. > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3289: -- Status: Patch Available (was: Open) > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3289: -- Attachment: MR3289_trunk_2.txt Updated patch fro trunk with some fixes, and fadvise when using ssl. > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 3289-1.txt, 3289-2.txt, MAPREDUCE-3289.branch-1.patch, > MR3289_trunk.txt, MR3289_trunk_2.txt, mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4481) User Log Retention across TT restarts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426894#comment-13426894 ] Benoy Antony commented on MAPREDUCE-4481: - Good point related to porting MAPREDUCE-2415 to 0.22. Another related question will be porting MAPREDUCE-1213 to 1.1 > User Log Retention across TT restarts > - > > Key: MAPREDUCE-4481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4481 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Benoy Antony >Priority: Minor > > The tasktrackers cleanup the userlog directory when they restart. > This happens independent of value of mapred.userlog.retain.hours. > The feature is to add a configurable feature to respect > mapred.userlog.retain.hours across TT restarts -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4501) couldn't compile hadoop-2.0 successfully because of errors in build files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426854#comment-13426854 ] Mayank Bansal commented on MAPREDUCE-4501: -- I have looked at the trunk pom files and could not find out. Can you please give little more details? Thanks, Mayank > couldn't compile hadoop-2.0 successfully because of errors in build files > - > > Key: MAPREDUCE-4501 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4501 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Yan Liu > > hadoop-yarn-applications relies on is 2.0.1-SNAPSHOT, however, the commit > makes it 3.0.0-SNAPSHOT. This makes the compile fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4466) Using URI for yarn.nodemanager log dirs fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426852#comment-13426852 ] Mayank Bansal commented on MAPREDUCE-4466: -- This change effects UI and I have tested this in my environment and it works. Thanks, Mayank > Using URI for yarn.nodemanager log dirs fails > - > > Key: MAPREDUCE-4466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4466 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eli Collins >Assignee: Mayank Bansal >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-4466-trunk-v1.patch > > > If I use URIs (eg file:///home/eli/hadoop/dirs) for yarn.nodemanager.log-dirs > or yarn.nodemanager.remote-app-log-dir the container log servlet fails with > an NPE (works if I remove the "file" scheme). Using a URI for > yarn.nodemanager.local-dirs works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4466) Using URI for yarn.nodemanager log dirs fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426849#comment-13426849 ] Hadoop QA commented on MAPREDUCE-4466: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538823/MAPREDUCE-4466-trunk-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2694//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2694//console This message is automatically generated. > Using URI for yarn.nodemanager log dirs fails > - > > Key: MAPREDUCE-4466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4466 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eli Collins >Assignee: Mayank Bansal >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-4466-trunk-v1.patch > > > If I use URIs (eg file:///home/eli/hadoop/dirs) for yarn.nodemanager.log-dirs > or yarn.nodemanager.remote-app-log-dir the container log servlet fails with > an NPE (works if I remove the "file" scheme). Using a URI for > yarn.nodemanager.local-dirs works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4466) Using URI for yarn.nodemanager log dirs fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4466: - Fix Version/s: trunk Status: Patch Available (was: Open) > Using URI for yarn.nodemanager log dirs fails > - > > Key: MAPREDUCE-4466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4466 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eli Collins >Assignee: Mayank Bansal >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-4466-trunk-v1.patch > > > If I use URIs (eg file:///home/eli/hadoop/dirs) for yarn.nodemanager.log-dirs > or yarn.nodemanager.remote-app-log-dir the container log servlet fails with > an NPE (works if I remove the "file" scheme). Using a URI for > yarn.nodemanager.local-dirs works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4466) Using URI for yarn.nodemanager log dirs fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4466: - Attachment: MAPREDUCE-4466-trunk-v1.patch Attaching the initial patch. > Using URI for yarn.nodemanager log dirs fails > - > > Key: MAPREDUCE-4466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4466 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eli Collins >Assignee: Mayank Bansal >Priority: Minor > Fix For: trunk > > Attachments: MAPREDUCE-4466-trunk-v1.patch > > > If I use URIs (eg file:///home/eli/hadoop/dirs) for yarn.nodemanager.log-dirs > or yarn.nodemanager.remote-app-log-dir the container log servlet fails with > an NPE (works if I remove the "file" scheme). Using a URI for > yarn.nodemanager.local-dirs works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4466) Using URI for yarn.nodemanager log dirs fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal reassigned MAPREDUCE-4466: Assignee: Mayank Bansal > Using URI for yarn.nodemanager log dirs fails > - > > Key: MAPREDUCE-4466 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4466 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eli Collins >Assignee: Mayank Bansal >Priority: Minor > > If I use URIs (eg file:///home/eli/hadoop/dirs) for yarn.nodemanager.log-dirs > or yarn.nodemanager.remote-app-log-dir the container log servlet fails with > an NPE (works if I remove the "file" scheme). Using a URI for > yarn.nodemanager.local-dirs works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4505) Create a combiner bypass path for keys with a single value
[ https://issues.apache.org/jira/browse/MAPREDUCE-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426827#comment-13426827 ] Karthik Kambatla commented on MAPREDUCE-4505: - Given the overheads involved, it might not be a bad idea to skip the combiner for more than just a single value. The threshold for running the combiner can be number of values or a cumulative size of the values. > Create a combiner bypass path for keys with a single value > -- > > Key: MAPREDUCE-4505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4505 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: performance, task >Reporter: Owen O'Malley >Assignee: Arun C Murthy > > It would help optimize a lot of cases where there aren't a lot of replicated > keys if the framework would bypass the deserialize/combiner/serialize step > for keys that only have a single value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4505) Create a combiner bypass path for keys with a single value
[ https://issues.apache.org/jira/browse/MAPREDUCE-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley reassigned MAPREDUCE-4505: Assignee: Arun C Murthy > Create a combiner bypass path for keys with a single value > -- > > Key: MAPREDUCE-4505 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4505 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: performance, task >Reporter: Owen O'Malley >Assignee: Arun C Murthy > > It would help optimize a lot of cases where there aren't a lot of replicated > keys if the framework would bypass the deserialize/combiner/serialize step > for keys that only have a single value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4505) Create a combiner bypass path for keys with a single value
Owen O'Malley created MAPREDUCE-4505: Summary: Create a combiner bypass path for keys with a single value Key: MAPREDUCE-4505 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4505 Project: Hadoop Map/Reduce Issue Type: Bug Components: performance, task Reporter: Owen O'Malley It would help optimize a lot of cases where there aren't a lot of replicated keys if the framework would bypass the deserialize/combiner/serialize step for keys that only have a single value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4036) Streaming TestUlimit fails on CentOS 6
[ https://issues.apache.org/jira/browse/MAPREDUCE-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4036: -- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to branch-1. > Streaming TestUlimit fails on CentOS 6 > -- > > Key: MAPREDUCE-4036 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4036 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.0.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 1.2.0 > > Attachments: MAPREDUCE-4036.patch > > > CentOS 6 seems to have higher memory requirements than other distros and > together with the new MALLOC library makes the TestUlimit to fail with exit > status 134. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4481) User Log Retention across TT restarts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426757#comment-13426757 ] Harsh J commented on MAPREDUCE-4481: Ah yes, my bad. Would 0.22 benefit from MAPREDUCE-2415 though? If its not needed, we can probably close this out again. > User Log Retention across TT restarts > - > > Key: MAPREDUCE-4481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4481 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Benoy Antony >Priority: Minor > > The tasktrackers cleanup the userlog directory when they restart. > This happens independent of value of mapred.userlog.retain.hours. > The feature is to add a configurable feature to respect > mapred.userlog.retain.hours across TT restarts -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4481) User Log Retention across TT restarts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426756#comment-13426756 ] Benoy Antony commented on MAPREDUCE-4481: - This will not impact 0.22. MAPREDUCE -2415 was not ported to 0.22 . So userlogs directory will not be under the scratch directories. > User Log Retention across TT restarts > - > > Key: MAPREDUCE-4481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4481 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Benoy Antony >Priority: Minor > > The tasktrackers cleanup the userlog directory when they restart. > This happens independent of value of mapred.userlog.retain.hours. > The feature is to add a configurable feature to respect > mapred.userlog.retain.hours across TT restarts -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4036) Streaming TestUlimit fails on CentOS 6
[ https://issues.apache.org/jira/browse/MAPREDUCE-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426755#comment-13426755 ] Harsh J commented on MAPREDUCE-4036: +1, lgtm. > Streaming TestUlimit fails on CentOS 6 > -- > > Key: MAPREDUCE-4036 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4036 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.0.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-4036.patch > > > CentOS 6 seems to have higher memory requirements than other distros and > together with the new MALLOC library makes the TestUlimit to fail with exit > status 134. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4036) Streaming TestUlimit fails on CentOS 6
[ https://issues.apache.org/jira/browse/MAPREDUCE-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426745#comment-13426745 ] Karthik Kambatla commented on MAPREDUCE-4036: - Yes, the patch fixes it for Ubuntu 12.04 as well. > Streaming TestUlimit fails on CentOS 6 > -- > > Key: MAPREDUCE-4036 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4036 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.0.1 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-4036.patch > > > CentOS 6 seems to have higher memory requirements than other distros and > together with the new MALLOC library makes the TestUlimit to fail with exit > status 134. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426695#comment-13426695
]
Hadoop QA commented on MAPREDUCE-4504:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12538786/MR-4504.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test
files.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2693//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2693//console
This message is automatically generated.
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4503) Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles
[ https://issues.apache.org/jira/browse/MAPREDUCE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426670#comment-13426670 ] Hadoop QA commented on MAPREDUCE-4503: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538783/MR-4503.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 javac. The applied patch generated 2070 javac compiler warnings (more than the trunk's current 2069 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common: org.apache.hadoop.mapred.TestLocalDistributedCacheManager +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2692//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2692//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2692//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2692//console This message is automatically generated. > Should throw InvalidJobConfException if duplicates found in cacheArchives or > cacheFiles > --- > > Key: MAPREDUCE-4503 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4503 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-4503.txt > > > in 1.0 if a file was both in a jobs cache archives and cache files, and > InvalidJobConfException was thrown. We should replicate this behavior on > mrv2. We should also extend it so that if a cache archive or cache file is > not going to be downloaded at all because of conflicts in the names of the > symlinks a similar exception is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4504:
---
Status: Patch Available (was: Open)
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4504) SortValidator writes to wrong directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4504:
---
Attachment: MR-4504.txt
Adds in fix.
> SortValidator writes to wrong directory
> ---
>
> Key: MAPREDUCE-4504
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Attachments: MR-4504.txt
>
>
> SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
> is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4504) SortValidator writes to wrong directory
Robert Joseph Evans created MAPREDUCE-4504:
--
Summary: SortValidator writes to wrong directory
Key: MAPREDUCE-4504
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4504
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
SortValidator tries to write to jobConf.get("hadoop.tmp.dir", "/tmp"), but it
is not intended to be an HDFS directory. it should just be /tmp.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4503) Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles
[ https://issues.apache.org/jira/browse/MAPREDUCE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4503: --- Status: Patch Available (was: Open) > Should throw InvalidJobConfException if duplicates found in cacheArchives or > cacheFiles > --- > > Key: MAPREDUCE-4503 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4503 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-4503.txt > > > in 1.0 if a file was both in a jobs cache archives and cache files, and > InvalidJobConfException was thrown. We should replicate this behavior on > mrv2. We should also extend it so that if a cache archive or cache file is > not going to be downloaded at all because of conflicts in the names of the > symlinks a similar exception is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4503) Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles
[ https://issues.apache.org/jira/browse/MAPREDUCE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4503: --- Attachment: MR-4503.txt This adds in the checks and also some tests. > Should throw InvalidJobConfException if duplicates found in cacheArchives or > cacheFiles > --- > > Key: MAPREDUCE-4503 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4503 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-4503.txt > > > in 1.0 if a file was both in a jobs cache archives and cache files, and > InvalidJobConfException was thrown. We should replicate this behavior on > mrv2. We should also extend it so that if a cache archive or cache file is > not going to be downloaded at all because of conflicts in the names of the > symlinks a similar exception is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4234) SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426647#comment-13426647 ] Hudson commented on MAPREDUCE-4234: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-4234. SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name) (Robert Evans via jeagles) (Revision 1367789) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367789 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/SortValidator.java > SortValidator.java is incompatible with multi-user or parallel use (due to a > /tmp file with static name) > > > Key: MAPREDUCE-4234 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4234 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: examples >Affects Versions: 0.23.3, trunk >Reporter: Randy Clayton >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MAPREDUCE-4234.patch, MR-4234.txt, MR-4234.txt > > > The SortValidator.java file checkRecords method creates a file in the > /tmp/sortvalidator directory using a static filename. This can result in > failures due to name collisions when the > hadoop-mapreduce-client-jobclient-*-tests jar is used by more than one task > or one user simultaneously. We use this jar when testing compression codecs > and after we started running tests in parallel (four at a time to reduce > overall test time) we started experiencing random test failures due to name > collisions. Creating a random or unique per thread filename may resolve this > issue. We have developed a change to introduce per use unique file names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4456) LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426641#comment-13426641
]
Hudson commented on MAPREDUCE-4456:
---
Integrated in Hadoop-Mapreduce-trunk #1154 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/])
MAPREDUCE-4456. LocalDistributedCacheManager can get an
ArrayIndexOutOfBounds when creating symlinks (Robert Evans via tgraves)
(Revision 1367581)
Result = FAILURE
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367581
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
> LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating
> symlinks
>
>
> Key: MAPREDUCE-4456
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4456
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MR-4456.txt, MR-4456.txt
>
>
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:194)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:154)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:620)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:385)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:233)
> at java.lang.Thread.run(Thread.java:619)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4483) 2.0 build does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426638#comment-13426638 ] Hudson commented on MAPREDUCE-4483: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-4483. 2.0 build does not work (John George via bobby) (Revision 1367726) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367726 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml > 2.0 build does not work > > > Key: MAPREDUCE-4483 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4483 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: John George >Assignee: John George > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: MR-4483.patch, MR-4483.patch, MR-4483.trunk.patch > > > Seems like hadoop-yarn-applications-unmanaged-am-launcher/pom.xml is pointing > to the wrong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4493) Distibuted Cache Compatability Issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426640#comment-13426640 ] Hudson commented on MAPREDUCE-4493: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-4493. Distibuted Cache Compatability Issues (Robert Evans via tgraves) (Revision 1367713) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367713 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MRCaching.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRDFSCaching.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java > Distibuted Cache Compatability Issues > - > > Key: MAPREDUCE-4493 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4493 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4493.txt, MR-4493.txt, MR-4493.txt > > > The distributed cache does not work like it does in 1.0. > mapreduce.job.cache.symlink.create is completely ignored and symlinks are > always created no matter what. Files and archives without a fragment will > also have symlinks created. > If two cache archives or cache files happen to have the same name, or same > symlink fragment only the last one in the list is localized. > The localCacheArchives and LocalCacheFiles are not set correctly when these > duplicates happen causing off by one or more errors for anyone trying to use > them. > The reality is that use of symlinking is so common currently that these > incompatibilities are not that likely to show up, but we still need to fix > them. -- This message is automatically generated by JIR
[jira] [Commented] (MAPREDUCE-4496) AM logs link is missing user name
[ https://issues.apache.org/jira/browse/MAPREDUCE-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426643#comment-13426643 ] Hudson commented on MAPREDUCE-4496: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby) (Revision 1367601) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367601 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java > AM logs link is missing user name > - > > Key: MAPREDUCE-4496 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4496 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.3, 2.2.0-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4496.patch > > > The link to the ApplicationMaster's logs on the MRAppMaster's web page is > missing the user name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4457) mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED
[ https://issues.apache.org/jira/browse/MAPREDUCE-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426637#comment-13426637 ] Hudson commented on MAPREDUCE-4457: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-4457. mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED (Robert Evans via tgraves) (Revision 1367771) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367771 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java > mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED > - > > Key: MAPREDUCE-4457 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4457 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4457.txt > > > we saw a job go into the ERROR state from an invalid state transition. > 3,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_007743_0 TaskAttempt Transitioned from SUCCEEDED > to FAILED > 2012-07-16 08:49:53,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_008850_0 TaskAttempt Transitioned from SUCCEEDED > to FAILED > 2012-07-16 08:49:53,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_017344_1000 TaskAttempt Transitioned from RUNNING > to SUCCESS_CONTAINER_CLEANUP > 2012-07-16 08:49:53,601 ERROR [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this > event at current state for attempt_1342238829791_2501_m_27_0 > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > TA_TOO_MANY_FETCH_FAILURE at FAILED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:954) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:133) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:913) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:905) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126) > at > org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:285) > at > org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:281) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75) > at java.lang.Thread.run(Thread.java:619) > 2012-07-16 08:49:53,601 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_029091_1000 TaskAttempt Transitioned from RUNNING > to SUCCESS_CONTAINER_CLEANUP > 2012-07-16 08:49:53,601 INFO [IPC Server handler 17 on 47153] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from > attempt_1342238829791_2501_r_000461_1000 > It looks like we possibly got 2 TA_TOO_MANY_FETCH_FAILURE events. The first > one moved it to FAILED and then the second one failed because no valid > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4492) Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull
[ https://issues.apache.org/jira/browse/MAPREDUCE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426639#comment-13426639 ] Hudson commented on MAPREDUCE-4492: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull (Mayank Bansal via bobby) (Revision 1367719) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367719 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > Configuring total queue capacity between 100.5 and 99.5 at perticular level > is sucessfull > - > > Key: MAPREDUCE-4492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Nishan Shetty >Assignee: Mayank Bansal >Priority: Minor > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4492-trunk-v2.patch, MAPREDUCE-4492-trunk.patch > > > Scenario: > 1.Configure a,b queues with capacities 40.0 and 60.5 respectively under root > queue > 2.Start process > Observe that process is started sucessfully with configured queue capacity > though the total capacity is 100.5(40.0+60.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4444) nodemanager fails to start when one of the local-dirs is bad
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426635#comment-13426635 ] Hudson commented on MAPREDUCE-: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-. nodemanager fails to start when one of the local-dirs is bad (Jason Lowe via bobby) (Revision 1367783) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367783 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestDiskFailures.java > nodemanager fails to start when one of the local-dirs is bad > > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0 >Reporter: Nathan Roberts >Assignee: Jason Lowe >Priority: Blocker > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4375) Show Configuration Tracability in MR UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426636#comment-13426636 ] Hudson commented on MAPREDUCE-4375: --- Integrated in Hadoop-Mapreduce-trunk #1154 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1154/]) MAPREDUCE-4375. Show Configuration Tracability in MR UI (bobby via tgraves) (Revision 1367539) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367539 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfEntryInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm > Show Configuration Tracability in MR UI > --- > > Key: MAPREDUCE-4375 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4375 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4375.txt, MR-4375.txt > > > Once HADOOP-8525 goes in we should provide a way for the Configuration UI to > display the traceability information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4481) User Log Retention across TT restarts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4481: --- Affects Version/s: 0.22.0 Thanks very much Benoy! Reopen reason: This should still affect 0.22 due to MAPREDUCE-1213 being in it. Lets try to fix it for 0.22.x. > User Log Retention across TT restarts > - > > Key: MAPREDUCE-4481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4481 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Benoy Antony >Priority: Minor > > The tasktrackers cleanup the userlog directory when they restart. > This happens independent of value of mapred.userlog.retain.hours. > The feature is to add a configurable feature to respect > mapred.userlog.retain.hours across TT restarts -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4481) User Log Retention across TT restarts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reassigned MAPREDUCE-4481: -- Assignee: (was: Benoy Antony) > User Log Retention across TT restarts > - > > Key: MAPREDUCE-4481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4481 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Benoy Antony >Priority: Minor > > The tasktrackers cleanup the userlog directory when they restart. > This happens independent of value of mapred.userlog.retain.hours. > The feature is to add a configurable feature to respect > mapred.userlog.retain.hours across TT restarts -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4481) User Log Retention across TT restarts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reopened MAPREDUCE-4481: > User Log Retention across TT restarts > - > > Key: MAPREDUCE-4481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4481 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > > The tasktrackers cleanup the userlog directory when they restart. > This happens independent of value of mapred.userlog.retain.hours. > The feature is to add a configurable feature to respect > mapred.userlog.retain.hours across TT restarts -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4503) Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles
Robert Joseph Evans created MAPREDUCE-4503: -- Summary: Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles Key: MAPREDUCE-4503 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4503 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans in 1.0 if a file was both in a jobs cache archives and cache files, and InvalidJobConfException was thrown. We should replicate this behavior on mrv2. We should also extend it so that if a cache archive or cache file is not going to be downloaded at all because of conflicts in the names of the symlinks a similar exception is thrown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4234) SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426589#comment-13426589 ] Hudson commented on MAPREDUCE-4234: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-4234. SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name) (Robert Evans via jeagles) (Revision 1367789) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367789 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/SortValidator.java > SortValidator.java is incompatible with multi-user or parallel use (due to a > /tmp file with static name) > > > Key: MAPREDUCE-4234 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4234 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: examples >Affects Versions: 0.23.3, trunk >Reporter: Randy Clayton >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MAPREDUCE-4234.patch, MR-4234.txt, MR-4234.txt > > > The SortValidator.java file checkRecords method creates a file in the > /tmp/sortvalidator directory using a static filename. This can result in > failures due to name collisions when the > hadoop-mapreduce-client-jobclient-*-tests jar is used by more than one task > or one user simultaneously. We use this jar when testing compression codecs > and after we started running tests in parallel (four at a time to reduce > overall test time) we started experiencing random test failures due to name > collisions. Creating a random or unique per thread filename may resolve this > issue. We have developed a change to introduce per use unique file names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4496) AM logs link is missing user name
[ https://issues.apache.org/jira/browse/MAPREDUCE-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426585#comment-13426585 ] Hudson commented on MAPREDUCE-4496: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby) (Revision 1367601) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367601 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java > AM logs link is missing user name > - > > Key: MAPREDUCE-4496 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4496 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.3, 2.2.0-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4496.patch > > > The link to the ApplicationMaster's logs on the MRAppMaster's web page is > missing the user name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4456) LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426583#comment-13426583
]
Hudson commented on MAPREDUCE-4456:
---
Integrated in Hadoop-Hdfs-trunk #1122 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/])
MAPREDUCE-4456. LocalDistributedCacheManager can get an
ArrayIndexOutOfBounds when creating symlinks (Robert Evans via tgraves)
(Revision 1367581)
Result = FAILURE
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367581
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
> LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating
> symlinks
>
>
> Key: MAPREDUCE-4456
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4456
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MR-4456.txt, MR-4456.txt
>
>
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:194)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:154)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:620)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:385)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:233)
> at java.lang.Thread.run(Thread.java:619)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4375) Show Configuration Tracability in MR UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426578#comment-13426578 ] Hudson commented on MAPREDUCE-4375: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-4375. Show Configuration Tracability in MR UI (bobby via tgraves) (Revision 1367539) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367539 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfEntryInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm > Show Configuration Tracability in MR UI > --- > > Key: MAPREDUCE-4375 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4375 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4375.txt, MR-4375.txt > > > Once HADOOP-8525 goes in we should provide a way for the Configuration UI to > display the traceability information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4457) mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED
[ https://issues.apache.org/jira/browse/MAPREDUCE-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426579#comment-13426579 ] Hudson commented on MAPREDUCE-4457: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-4457. mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED (Robert Evans via tgraves) (Revision 1367771) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367771 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java > mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED > - > > Key: MAPREDUCE-4457 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4457 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4457.txt > > > we saw a job go into the ERROR state from an invalid state transition. > 3,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_007743_0 TaskAttempt Transitioned from SUCCEEDED > to FAILED > 2012-07-16 08:49:53,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_008850_0 TaskAttempt Transitioned from SUCCEEDED > to FAILED > 2012-07-16 08:49:53,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_017344_1000 TaskAttempt Transitioned from RUNNING > to SUCCESS_CONTAINER_CLEANUP > 2012-07-16 08:49:53,601 ERROR [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this > event at current state for attempt_1342238829791_2501_m_27_0 > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > TA_TOO_MANY_FETCH_FAILURE at FAILED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:954) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:133) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:913) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:905) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126) > at > org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:285) > at > org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:281) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75) > at java.lang.Thread.run(Thread.java:619) > 2012-07-16 08:49:53,601 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_029091_1000 TaskAttempt Transitioned from RUNNING > to SUCCESS_CONTAINER_CLEANUP > 2012-07-16 08:49:53,601 INFO [IPC Server handler 17 on 47153] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from > attempt_1342238829791_2501_r_000461_1000 > It looks like we possibly got 2 TA_TOO_MANY_FETCH_FAILURE events. The first > one moved it to FAILED and then the second one failed because no valid > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4483) 2.0 build does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426580#comment-13426580 ] Hudson commented on MAPREDUCE-4483: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-4483. 2.0 build does not work (John George via bobby) (Revision 1367726) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367726 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/pom.xml * /hadoop/common/trunk/hadoop-project/pom.xml > 2.0 build does not work > > > Key: MAPREDUCE-4483 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4483 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.2.0-alpha >Reporter: John George >Assignee: John George > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: MR-4483.patch, MR-4483.patch, MR-4483.trunk.patch > > > Seems like hadoop-yarn-applications-unmanaged-am-launcher/pom.xml is pointing > to the wrong -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4492) Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull
[ https://issues.apache.org/jira/browse/MAPREDUCE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426581#comment-13426581 ] Hudson commented on MAPREDUCE-4492: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull (Mayank Bansal via bobby) (Revision 1367719) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367719 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > Configuring total queue capacity between 100.5 and 99.5 at perticular level > is sucessfull > - > > Key: MAPREDUCE-4492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Nishan Shetty >Assignee: Mayank Bansal >Priority: Minor > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4492-trunk-v2.patch, MAPREDUCE-4492-trunk.patch > > > Scenario: > 1.Configure a,b queues with capacities 40.0 and 60.5 respectively under root > queue > 2.Start process > Observe that process is started sucessfully with configured queue capacity > though the total capacity is 100.5(40.0+60.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4493) Distibuted Cache Compatability Issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426582#comment-13426582 ] Hudson commented on MAPREDUCE-4493: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-4493. Distibuted Cache Compatability Issues (Robert Evans via tgraves) (Revision 1367713) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367713 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobContext.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MRCaching.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRDFSCaching.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java > Distibuted Cache Compatability Issues > - > > Key: MAPREDUCE-4493 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4493 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4493.txt, MR-4493.txt, MR-4493.txt > > > The distributed cache does not work like it does in 1.0. > mapreduce.job.cache.symlink.create is completely ignored and symlinks are > always created no matter what. Files and archives without a fragment will > also have symlinks created. > If two cache archives or cache files happen to have the same name, or same > symlink fragment only the last one in the list is localized. > The localCacheArchives and LocalCacheFiles are not set correctly when these > duplicates happen causing off by one or more errors for anyone trying to use > them. > The reality is that use of symlinking is so common currently that these > incompatibilities are not that likely to show up, but we still need to fix > them. -- This message is automatically generated by JIRA. If you
[jira] [Commented] (MAPREDUCE-4444) nodemanager fails to start when one of the local-dirs is bad
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426577#comment-13426577 ] Hudson commented on MAPREDUCE-: --- Integrated in Hadoop-Hdfs-trunk #1122 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1122/]) MAPREDUCE-. nodemanager fails to start when one of the local-dirs is bad (Jason Lowe via bobby) (Revision 1367783) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367783 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestDiskFailures.java > nodemanager fails to start when one of the local-dirs is bad > > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0 >Reporter: Nathan Roberts >Assignee: Jason Lowe >Priority: Blocker > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4234) SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426572#comment-13426572 ] Hudson commented on MAPREDUCE-4234: --- Integrated in Hadoop-Hdfs-0.23-Build #331 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/]) svn merge -c 1367789 FIXES: MAPREDUCE-4234. SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name) (Robert Evans via jeagles) (Revision 1367795) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367795 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/SortValidator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/SortValidator.java > SortValidator.java is incompatible with multi-user or parallel use (due to a > /tmp file with static name) > > > Key: MAPREDUCE-4234 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4234 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: examples >Affects Versions: 0.23.3, trunk >Reporter: Randy Clayton >Assignee: Robert Joseph Evans >Priority: Minor > Attachments: MAPREDUCE-4234.patch, MR-4234.txt, MR-4234.txt > > > The SortValidator.java file checkRecords method creates a file in the > /tmp/sortvalidator directory using a static filename. This can result in > failures due to name collisions when the > hadoop-mapreduce-client-jobclient-*-tests jar is used by more than one task > or one user simultaneously. We use this jar when testing compression codecs > and after we started running tests in parallel (four at a time to reduce > overall test time) we started experiencing random test failures due to name > collisions. Creating a random or unique per thread filename may resolve this > issue. We have developed a change to introduce per use unique file names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4457) mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED
[ https://issues.apache.org/jira/browse/MAPREDUCE-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426566#comment-13426566 ] Hudson commented on MAPREDUCE-4457: --- Integrated in Hadoop-Hdfs-0.23-Build #331 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/]) merge -r 1367771:1367772 from branch-2. FIXES: MAPREDUCE-4457 (Revision 1367774) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367774 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java > mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED > - > > Key: MAPREDUCE-4457 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4457 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3 >Reporter: Thomas Graves >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4457.txt > > > we saw a job go into the ERROR state from an invalid state transition. > 3,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_007743_0 TaskAttempt Transitioned from SUCCEEDED > to FAILED > 2012-07-16 08:49:53,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_008850_0 TaskAttempt Transitioned from SUCCEEDED > to FAILED > 2012-07-16 08:49:53,600 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_017344_1000 TaskAttempt Transitioned from RUNNING > to SUCCESS_CONTAINER_CLEANUP > 2012-07-16 08:49:53,601 ERROR [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this > event at current state for attempt_1342238829791_2501_m_27_0 > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > TA_TOO_MANY_FETCH_FAILURE at FAILED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:954) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:133) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:913) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:905) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126) > at > org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:285) > at > org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:281) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75) > at java.lang.Thread.run(Thread.java:619) > 2012-07-16 08:49:53,601 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1342238829791_2501_m_029091_1000 TaskAttempt Transitioned from RUNNING > to SUCCESS_CONTAINER_CLEANUP > 2012-07-16 08:49:53,601 INFO [IPC Server handler 17 on 47153] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from > attempt_1342238829791_2501_r_000461_1000 > It looks like we possibly got 2 TA_TOO_MANY_FETCH_FAILURE events. The first > one moved it to FAILED and then the second one failed because no valid > transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4496) AM logs link is missing user name
[ https://issues.apache.org/jira/browse/MAPREDUCE-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426570#comment-13426570 ] Hudson commented on MAPREDUCE-4496: --- Integrated in Hadoop-Hdfs-0.23-Build #331 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/]) svn merge -c 1367601 FIXES: MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby) (Revision 1367604) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367604 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java > AM logs link is missing user name > - > > Key: MAPREDUCE-4496 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4496 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.3, 2.2.0-alpha >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4496.patch > > > The link to the ApplicationMaster's logs on the MRAppMaster's web page is > missing the user name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4493) Distibuted Cache Compatability Issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426569#comment-13426569 ] Hudson commented on MAPREDUCE-4493: --- Integrated in Hadoop-Hdfs-0.23-Build #331 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/]) merge -r 1367714:1367715 from branch-2. FIXES: MAPREDUCE-4493 (Revision 1367717) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367717 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Submitter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobContext.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/DistributedCache.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MRCaching.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRDFSCaching.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecution.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java > Distibuted Cache Compatability Issues > - > > Key: MAPREDUCE-4493 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4493 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Critical > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4493.txt, MR-4493.txt, MR-4493.txt > > > The distributed cache does not work like it does in 1.0. > mapreduce.job.cache.symlink.create is completely ignored and symlinks are > always created no matter what. Files and archives without a fragment will > also have symlinks created. > If two cache archives or cache files happen to have the same name, or same > symlink fragment only the last one in the list is localized. > The localCacheArchives and LocalCacheFiles are not set correctly when these > duplicates happen
[jira] [Commented] (MAPREDUCE-4456) LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426567#comment-13426567
]
Hudson commented on MAPREDUCE-4456:
---
Integrated in Hadoop-Hdfs-0.23-Build #331 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/])
merge -r 1367584:1367585 from branch-2. FIXES: MAPREDUCE-4456 (Revision
1367587)
Result = SUCCESS
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367587
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestLocalDistributedCacheManager.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java
> LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating
> symlinks
>
>
> Key: MAPREDUCE-4456
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4456
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.3, 3.0.0, 2.2.0-alpha
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MR-4456.txt, MR-4456.txt
>
>
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:194)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:154)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:620)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:385)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:233)
> at java.lang.Thread.run(Thread.java:619)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4444) nodemanager fails to start when one of the local-dirs is bad
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426564#comment-13426564 ] Hudson commented on MAPREDUCE-: --- Integrated in Hadoop-Hdfs-0.23-Build #331 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/]) svn merge -c 1367783 FIXES: MAPREDUCE-. nodemanager fails to start when one of the local-dirs is bad (Jason Lowe via bobby) (Revision 1367785) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367785 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LocalDirsHandlerService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestDiskFailures.java > nodemanager fails to start when one of the local-dirs is bad > > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0 >Reporter: Nathan Roberts >Assignee: Jason Lowe >Priority: Blocker > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4375) Show Configuration Tracability in MR UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426563#comment-13426563 ] Hudson commented on MAPREDUCE-4375: --- Integrated in Hadoop-Hdfs-0.23-Build #331 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/]) merge -r 1367539:1367540 from branch-2. FIXES: MAPREDUCE-4375 (Revision 1367544) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367544 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfEntryInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm > Show Configuration Tracability in MR UI > --- > > Key: MAPREDUCE-4375 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4375 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 0.23.3 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MR-4375.txt, MR-4375.txt > > > Once HADOOP-8525 goes in we should provide a way for the Configuration UI to > display the traceability information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4492) Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull
[ https://issues.apache.org/jira/browse/MAPREDUCE-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426565#comment-13426565 ] Hudson commented on MAPREDUCE-4492: --- Integrated in Hadoop-Hdfs-0.23-Build #331 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/331/]) svn merge -c 1367719 FIXES: MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull (Mayank Bansal via bobby) (Revision 1367723) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1367723 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java > Configuring total queue capacity between 100.5 and 99.5 at perticular level > is sucessfull > - > > Key: MAPREDUCE-4492 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4492 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0-alpha >Reporter: Nishan Shetty >Assignee: Mayank Bansal >Priority: Minor > Fix For: 0.23.3, 3.0.0, 2.2.0-alpha > > Attachments: MAPREDUCE-4492-trunk-v2.patch, MAPREDUCE-4492-trunk.patch > > > Scenario: > 1.Configure a,b queues with capacities 40.0 and 60.5 respectively under root > queue > 2.Start process > Observe that process is started sucessfully with configured queue capacity > though the total capacity is 100.5(40.0+60.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426365#comment-13426365 ] Tsuyoshi OZAWA commented on MAPREDUCE-4502: --- We prototyped the combining the result of maps per node/rack. the the result of our benchmark is available at mapreduce-dev ML(http://goo.gl/wWBWQ). > Multi-level aggregation with combining the result of maps per node/rack > --- > > Key: MAPREDUCE-4502 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Tsuyoshi OZAWA > > The shuffle costs is expensive in Hadoop in spite of the existence of > combiner, because the scope of combining is limited within only one MapTask. > To solve this problem, it's a good way to aggregate the result of maps per > node/rack by launch combiner. > This JIRA is to implement the multi-level aggregation infrastructure, > including combining per container(MAPREDUCE-3902 is related), coordinating > containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-4502: -- Description: The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner. This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs. was: The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner. This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs. > Multi-level aggregation with combining the result of maps per node/rack > --- > > Key: MAPREDUCE-4502 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Tsuyoshi OZAWA > > The shuffle costs is expensive in Hadoop in spite of the existence of > combiner, because the scope of combining is limited within only one MapTask. > To solve this problem, it's a good way to aggregate the result of maps per > node/rack by launch combiner. > This JIRA is to implement the multi-level aggregation infrastructure, > including combining per container(MAPREDUCE-3902 is related), coordinating > containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
