[jira] [Commented] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223025#comment-13223025 ] Hadoop QA commented on MAPREDUCE-1740: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517192/MAPREDUCE-1740.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2010//console This message is automatically generated. > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.3 >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-1740: Status: Patch Available (was: Open) > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3976) TestRMContainerAllocator failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3976: - Target Version/s: 0.23.2 Affects Version/s: (was: 0.23.0) 0.23.2 Fix Version/s: (was: 0.23.0) Jason - can you please take a look at this asap? Thanks. > TestRMContainerAllocator failing > > > Key: MAPREDUCE-3976 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3976 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Bikas Saha >Assignee: Jason Lowe > > The following stack trace is being generated > > org.apache.hadoop.metrics2.MetricsException: Metrics source JvmMetrics > already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:126) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:107) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:216) > at > org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:80) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:58) > at > org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:54) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:186) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:170) > at org.apache.hadoop.mapreduce.v2.app.MRApp.(MRApp.java:160) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator$1.(TestRMContainerAllocator.java:372) > at > org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgress(TestRMContainerAllocator.java:371) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > > This test fails when git trunk is reset to - commit > 6689d99b38c7c562e8cae484207ad30ad7b56eb5 > but passes when git trunk is reset to - commit > f429fdaf78a02211c4faee54b1ee92822edc5741 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1740) NPE in getMatchingLevelForNodes when node locations are variable depth
[ https://issues.apache.org/jira/browse/MAPREDUCE-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Radwan updated MAPREDUCE-1740: Attachment: MAPREDUCE-1740.patch An updated patch for branch-1.0. Please review. > NPE in getMatchingLevelForNodes when node locations are variable depth > -- > > Key: MAPREDUCE-1740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.3 >Reporter: Todd Lipcon > Attachments: MAPREDUCE-1740.patch, mapreduce-1740.txt > > > In getMatchingLevelForNodes, we assume that both nodes have the same "depth" > (ie number of path components). If the user provides a topology script that > assigns one node a path like /foo/bar/baz and another node a path like > /foo/blah, this function will throw an NPE. > I'm not sure if there are other places where we assume that all node > locations have a constant number of paths. If so we should check the output > of the topology script aggressively to be sure this is the case. Otherwise I > think we simply need to add && n2 != null to the while loop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222967#comment-13222967 ] Aaron T. Myers commented on MAPREDUCE-3974: --- Thanks a lot for the quick review, Arun. Mind committing this as well? I'm unfortunately not an MR committer. > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222965#comment-13222965 ] Arun C Murthy commented on MAPREDUCE-3974: -- +1 > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3912) getPriority() method within the JobHistoryParser JobInfo class fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222961#comment-13222961
]
Viraj Bhat commented on MAPREDUCE-3912:
---
Hi Arun,
Can we just change this to return NORMAL as the default value right now?
Viraj
> getPriority() method within the JobHistoryParser JobInfo class fails
>
>
> Key: MAPREDUCE-3912
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3912
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Viraj Bhat
> Fix For: 0.24.0
>
>
> When you use the following set of statements:
> {code}
> JobHistoryParser.JobInfo jobInfo;
> jobInfo.getPriority(); //fails
> {code}
> The problem is that {code} public String getPriority() { return
> priority.toString(); } {code} will fail if the "priority" is null.
> I was expecting priority to return a "NORMAL" value
> Viraj
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222947#comment-13222947 ] Bikas Saha commented on MAPREDUCE-3353: --- The test failure is unrelated to the patch and happens on trunk. MAPREDUCE-3976. > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3976) TestRMContainerAllocator failing
TestRMContainerAllocator failing Key: MAPREDUCE-3976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3976 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Bikas Saha Assignee: Jason Lowe Fix For: 0.23.0 The following stack trace is being generated org.apache.hadoop.metrics2.MetricsException: Metrics source JvmMetrics already exists! at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:126) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:107) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:216) at org.apache.hadoop.metrics2.source.JvmMetrics.create(JvmMetrics.java:80) at org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:58) at org.apache.hadoop.mapreduce.v2.app.metrics.MRAppMetrics.create(MRAppMetrics.java:54) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:186) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.(MRAppMaster.java:170) at org.apache.hadoop.mapreduce.v2.app.MRApp.(MRApp.java:160) at org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator$1.(TestRMContainerAllocator.java:372) at org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgress(TestRMContainerAllocator.java:371) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) This test fails when git trunk is reset to - commit 6689d99b38c7c562e8cae484207ad30ad7b56eb5 but passes when git trunk is reset to - commit f429fdaf78a02211c4faee54b1ee92822edc5741 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222926#comment-13222926 ] Aaron T. Myers commented on MAPREDUCE-3974: --- I feel pretty confident that the test failures are unrelated. I just ran these tests on my machine both with and without this patch applied, and they all passed just fine. Would someone mind reviewing this ASAP? MR Jenkins runs will be broken until this is committed. > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3855) TestSubmitJob should use FileSystem instead of private HDFS classes.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222913#comment-13222913 ] Aaron T. Myers commented on MAPREDUCE-3855: --- Ah, sorry. Didn't know about this issue. Thanks for resolving this one, Uma. > TestSubmitJob should use FileSystem instead of private HDFS classes. > > > Key: MAPREDUCE-3855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3855 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, test >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Uma Maheswara Rao G >Priority: Minor > > TestSubmitJob uses ClientNamenodeProtocolTranslatorPB which is marked > InterfaceAudience.Private. This causes build failures in mapreduce when HDFS > changes internal/private classes. Test should be changed to use FileSystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3855) TestSubmitJob should use FileSystem instead of private HDFS classes.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved MAPREDUCE-3855. Resolution: Duplicate I just verified MAPREDUCE-3974, that will fix this issue. Thanks Aaron for the patch, marking this as duplicate. > TestSubmitJob should use FileSystem instead of private HDFS classes. > > > Key: MAPREDUCE-3855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3855 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, test >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Uma Maheswara Rao G >Priority: Minor > > TestSubmitJob uses ClientNamenodeProtocolTranslatorPB which is marked > InterfaceAudience.Private. This causes build failures in mapreduce when HDFS > changes internal/private classes. Test should be changed to use FileSystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3855) TestSubmitJob should use FileSystem instead of private HDFS classes.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated MAPREDUCE-3855: --- Component/s: (was: mrv2) mrv1 > TestSubmitJob should use FileSystem instead of private HDFS classes. > > > Key: MAPREDUCE-3855 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3855 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, test >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Uma Maheswara Rao G >Priority: Minor > > TestSubmitJob uses ClientNamenodeProtocolTranslatorPB which is marked > InterfaceAudience.Private. This causes build failures in mapreduce when HDFS > changes internal/private classes. Test should be changed to use FileSystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222905#comment-13222905 ] Hadoop QA commented on MAPREDUCE-3974: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517178/MAPREDUCE-3974.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobClientGetJob org.apache.hadoop.mapred.TestMRWithDistributedCache org.apache.hadoop.mapred.TestLocalModeWithNewApis +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2009//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2009//console This message is automatically generated. > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222894#comment-13222894 ] Hudson commented on MAPREDUCE-3034: --- Integrated in Hadoop-Common-0.23-Commit #643 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/643/]) Merge -c 1297310 from trunk to branch-0.23 to fix MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from ResourceManager. Contributed by Devaraj K & Eric Payne. (Revision 1297311) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297311 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222892#comment-13222892 ] Hudson commented on MAPREDUCE-3034: --- Integrated in Hadoop-Common-trunk-Commit #1839 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1839/]) MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from ResourceManager. Contributed by Devaraj K & Eric Payne. (Revision 1297310) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297310 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222888#comment-13222888 ] Hudson commented on MAPREDUCE-3034: --- Integrated in Hadoop-Hdfs-0.23-Commit #633 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/633/]) Merge -c 1297310 from trunk to branch-0.23 to fix MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from ResourceManager. Contributed by Devaraj K & Eric Payne. (Revision 1297311) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297311 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222887#comment-13222887 ] Hudson commented on MAPREDUCE-3034: --- Integrated in Hadoop-Hdfs-trunk-Commit #1913 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1913/]) MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from ResourceManager. Contributed by Devaraj K & Eric Payne. (Revision 1297310) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297310 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-3974: -- Status: Patch Available (was: Open) > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated MAPREDUCE-3974: -- Attachment: MAPREDUCE-3974.patch Here's a patch for trunk which addresses the issue. I tested it by running `ant -Dtestcase=TestSubmitJob test-core' What a blast from the past working with MR1, and compiling using ant! :) > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > Attachments: MAPREDUCE-3974.patch > > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222861#comment-13222861 ] Hudson commented on MAPREDUCE-3034: --- Integrated in Hadoop-Mapreduce-0.23-Commit #644 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/644/]) Merge -c 1297310 from trunk to branch-0.23 to fix MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from ResourceManager. Contributed by Devaraj K & Eric Payne. (Revision 1297311) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297311 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222860#comment-13222860 ] Hudson commented on MAPREDUCE-3034: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1846 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1846/]) MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from ResourceManager. Contributed by Devaraj K & Eric Payne. (Revision 1297310) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297310 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222848#comment-13222848 ] Hadoop QA commented on MAPREDUCE-3353: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517166/MAPREDUCE-3353-branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 29 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 503 javac compiler warnings (more than the trunk's current 501 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2008//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2008//console This message is automatically generated. > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3034: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Devaraj and Eric! > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222838#comment-13222838
]
Arun C Murthy commented on MAPREDUCE-3034:
--
+1, thanks for the update Vinod.
The MR1 test failures are unrelated and seem like a temp. blip... all are of
the form:
{noformat}
org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot access
`/tmp/hadoop-jenkins/mapred/staging/jenkins755067366/.staging/job_local_0001':
No such file or directory
{noformat}
> NM should act on a REBOOT command from RM
> -
>
> Key: MAPREDUCE-3034
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Affects Versions: 0.23.0
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Devaraj K
>Priority: Critical
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch,
> MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt,
> MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch,
> MAPREDUCE-3034.patch, MR-3034.txt
>
>
> RM sends a reboot command to NM in some cases, like when it gets lost and
> rejoins back. In such a case, NM should act on the command and
> reboot/reinitalize itself.
> This is akin to TT reinitialize on order from JT. We will need to shutdown
> all the services properly and reinitialize - this should automatically take
> care of killing of containers, cleaning up local temporary files etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222836#comment-13222836 ] Hadoop QA commented on MAPREDUCE-3961: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517164/MR3961v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 57 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobClientGetJob org.apache.hadoop.mapred.TestMRWithDistributedCache org.apache.hadoop.mapred.TestLocalModeWithNewApis +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2007//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2007//console This message is automatically generated. > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt, MR3961v2.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Attachment: MAPREDUCE-3353-branch-0.23.patch Adding new patch with latest changes pulled and merged. > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Status: Patch Available (was: Open) > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Status: Open (was: Patch Available) > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222823#comment-13222823 ] Hudson commented on MAPREDUCE-3952: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1845 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1845/]) MAPREDUCE-3952. In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu) (Revision 1297293) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297293 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Bhallamudi Venkata Siva Kamesh > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222824#comment-13222824 ] Hudson commented on MAPREDUCE-3952: --- Integrated in Hadoop-Common-trunk-Commit #1838 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1838/]) MAPREDUCE-3952. In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu) (Revision 1297293) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297293 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Bhallamudi Venkata Siva Kamesh > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3961: -- Target Version/s: 0.23.2 Affects Version/s: 0.23.0 Status: Patch Available (was: Open) > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt, MR3961v2.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222819#comment-13222819 ] Hudson commented on MAPREDUCE-3952: --- Integrated in Hadoop-Hdfs-0.23-Commit #632 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/632/]) Merge -r 1297292:1297293 from trunk to branch. FIXES: MAPREDUCE-3952 (Revision 1297299) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297299 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Bhallamudi Venkata Siva Kamesh > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222820#comment-13222820 ] Hudson commented on MAPREDUCE-3952: --- Integrated in Hadoop-Mapreduce-0.23-Commit #643 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/643/]) Merge -r 1297292:1297293 from trunk to branch. FIXES: MAPREDUCE-3952 (Revision 1297299) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297299 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Bhallamudi Venkata Siva Kamesh > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3961: -- Attachment: MR3961v2.txt The jenkins unit test failures aren't related. Noticed the new unit test failing occasionally - updating the patch to fix that. > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt, MR3961v2.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3961: -- Status: Open (was: Patch Available) > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned MAPREDUCE-3952: - Assignee: Bhallamudi Venkata Siva Kamesh (was: Bhallamudi Venkata Siva Kamesh) > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Bhallamudi Venkata Siva Kamesh > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222812#comment-13222812 ] Hudson commented on MAPREDUCE-3952: --- Integrated in Hadoop-Hdfs-trunk-Commit #1912 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1912/]) MAPREDUCE-3952. In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu) (Revision 1297293) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297293 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Bhallamudi Venkata Siva Kamesh > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned MAPREDUCE-3952: - Assignee: Bhallamudi Venkata Siva Kamesh (was: Zhenxiao Luo) by mistake I've assigned to the reporter, assigning it to Bhallamudi > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Bhallamudi Venkata Siva Kamesh > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222808#comment-13222808 ] Hudson commented on MAPREDUCE-3952: --- Integrated in Hadoop-Common-0.23-Commit #642 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/642/]) Merge -r 1297292:1297293 from trunk to branch. FIXES: MAPREDUCE-3952 (Revision 1297299) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297299 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Zhenxiao Luo > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3952: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Zhenxiao Luo. Committed to trunk and branch-0.23 > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Zhenxiao Luo > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3952: -- Affects Version/s: 0.23.3 0.24.0 Fix Version/s: 0.23.3 > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Zhenxiao Luo >Assignee: Zhenxiao Luo > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reassigned MAPREDUCE-3952: - Assignee: Zhenxiao Luo > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Zhenxiao Luo >Assignee: Zhenxiao Luo > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222796#comment-13222796 ] Hadoop QA commented on MAPREDUCE-3034: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517152/MAPREDUCE-3034-20120305.1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobClientGetJob org.apache.hadoop.mapred.TestMRWithDistributedCache org.apache.hadoop.mapred.TestLocalModeWithNewApis +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2006//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2006//console This message is automatically generated. > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3952) In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222791#comment-13222791 ] Alejandro Abdelnur commented on MAPREDUCE-3952: --- +1 > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > --- > > Key: MAPREDUCE-3952 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3952 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Zhenxiao Luo > Attachments: MAPREDUCE-3952-1.patch, MAPREDUCE-3952.patch > > > Hive get unexpected result when using MR2(When using MR1, always get expected > result). > In MR2, when Total input paths to process == 1, > CombinefileInputFormat.getSplits() returns 0 split. > The calling code in Hive, in Hadoop23Shims.java: > InputSplit[] splits = super.getSplits(job, numSplits); > this get splits.length == 0. > In MR1, everything goes fine, the calling code in Hive, in Hadoop20Shims.java: > CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, > numSplits); > this get splits.length == 1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222754#comment-13222754 ] Arun C Murthy commented on MAPREDUCE-3975: -- Good catch Eric! We should fix YarnChild to set this for back-compat. > Default value not set for Configuration parameter mapreduce.job.local.dir > - > > Key: MAPREDUCE-3975 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.23.2 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Blocker > > mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by > default. This is a regression from 0.20.205. > In 0.20.205, JobLocalizer.createWorkDir() constructs the > "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on > $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's > JobConf. > So far, I haven't found where this is done in 0.23. It could be that this is > what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am > still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3975) Default value not set for Configuration parameter mapreduce.job.local.dir
Default value not set for Configuration parameter mapreduce.job.local.dir - Key: MAPREDUCE-3975 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3975 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.23.2 Reporter: Eric Payne Assignee: Eric Payne Priority: Blocker mapreduce.job.local.dir (formerly job.local.dir in 0.20) is not set by default. This is a regression from 0.20.205. In 0.20.205, JobLocalizer.createWorkDir() constructs the "$mapred.local.dir/taskTracker/$user/jobcache/$jobid/work" path based on $user and $jobid, and then sets TaskTracker.JOB_LOCAL_DIR in the job's JobConf. So far, I haven't found where this is done in 0.23. It could be that this is what should be done by LocalJobRunner.setupChildMapredLocalDirs(), but I am still investigating. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3034: --- Attachment: MAPREDUCE-3034-20120305.1.txt Suppressing findBugs warning about sys.exit... > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3034: --- Status: Patch Available (was: Open) > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.1.txt, MAPREDUCE-3034-20120305.txt, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3969) ConcurrentModificationException in JobHistory.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-3969.
--
Resolution: Not A Problem
> ConcurrentModificationException in JobHistory.java
> --
>
> Key: MAPREDUCE-3969
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3969
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.20.2
> Environment: cdh3u1 hadoop distribution, centos 5.5.
>Reporter: Alexey Zotov
>
> {code}
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201202150320_3709_m_000148_0' has completed
> task_201202150320_3709_m_000148 successfully.
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.mapred.JobHistory: Logging
> failed for job job_201202150320_3709removing PrintWriter from FileManager
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 5 on 8021, call
> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@61069281, false, false,
> true, -21317) from :450
> 41: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.hadoop.mapred.JobHistory.log(JobHistory.java:591)
> at
> org.apache.hadoop.mapred.JobHistory$MapAttempt.logFinished(JobHistory.java:1735)
> at
> org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2515)
> at
> org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1200)
> at
> org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:4539)
> at
> org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3503)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3202)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
> Task task_201202150320_3709_m_000148 was being marked as failed (but had not
> been restarted) at that moment and job execution was being freezed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3969) ConcurrentModificationException in JobHistory.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222735#comment-13222735
]
Arun C Murthy commented on MAPREDUCE-3969:
--
Alexey, hadoop-0.20.2 is too old and hadoop-1 or even hadoop-0.20.203+ seem to
not have this.
You might have to check the CDH user-lists.
> ConcurrentModificationException in JobHistory.java
> --
>
> Key: MAPREDUCE-3969
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3969
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.20.2
> Environment: cdh3u1 hadoop distribution, centos 5.5.
>Reporter: Alexey Zotov
>
> {code}
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.mapred.JobInProgress: Task
> 'attempt_201202150320_3709_m_000148_0' has completed
> task_201202150320_3709_m_000148 successfully.
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.mapred.JobHistory: Logging
> failed for job job_201202150320_3709removing PrintWriter from FileManager
> 2012-03-01 04:24:47,479 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 5 on 8021, call
> heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@61069281, false, false,
> true, -21317) from :450
> 41: error: java.io.IOException: java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at
> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.hadoop.mapred.JobHistory.log(JobHistory.java:591)
> at
> org.apache.hadoop.mapred.JobHistory$MapAttempt.logFinished(JobHistory.java:1735)
> at
> org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2515)
> at
> org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1200)
> at
> org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:4539)
> at
> org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3503)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3202)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
> Task task_201202150320_3709_m_000148 was being marked as failed (but had not
> been restarted) at that moment and job execution was being freezed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3034: --- Status: Open (was: Patch Available) > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.txt, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, > MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reassigned MAPREDUCE-3974: - Assignee: Aaron T. Myers > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge > > > Key: MAPREDUCE-3974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3 >Reporter: Arun C Murthy >Assignee: Aaron T. Myers >Priority: Blocker > > TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. > 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222722#comment-13222722 ] Hadoop QA commented on MAPREDUCE-3034: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517129/MAPREDUCE-3034-20120305.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2004//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2004//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2004//console This message is automatically generated. > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.txt, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, > MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge Key: MAPREDUCE-3974 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3 Reporter: Arun C Murthy Priority: Blocker TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge. 'ant compile-tests' doesn't work (since it's ant for MR1). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3578) starting nodemanager as 'root' gives "Unknown -jvm option"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222716#comment-13222716
]
Hadoop QA commented on MAPREDUCE-3578:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517138/MAPREDUCE-3578.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestJobClientGetJob
org.apache.hadoop.mapred.TestMRWithDistributedCache
org.apache.hadoop.mapred.TestLocalModeWithNewApis
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2005//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2005//console
This message is automatically generated.
> starting nodemanager as 'root' gives "Unknown -jvm option"
> --
>
> Key: MAPREDUCE-3578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3578
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0
> Environment: jdk 1.6.0.26, ubuntu 11.04
>Reporter: Gilad Wolff
> Attachments: MAPREDUCE-3578.patch
>
>
> running "sudo HADOOP_ROOT/bin/yarn-daemon.sh start nodemanager" I get
> "unknown -jvm option" (jdk version 1.6.0.26). The problem seems to be with
> line 204 in yarn:
> elif [ "$COMMAND" = "nodemanager" ] ; then
> CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
> CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
> if [[ $EUID -eq 0 ]]; then
> YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS"
> else
> YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
> fi
> using -server seems to solve the problem for me.
> I tested using build 929 from
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3755) Add the equivalent of JobStatus to end of JobHistory file
[ https://issues.apache.org/jira/browse/MAPREDUCE-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3755: -- Issue Type: Sub-task (was: Improvement) Parent: MAPREDUCE-3973 > Add the equivalent of JobStatus to end of JobHistory file > -- > > Key: MAPREDUCE-3755 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3755 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Bikas Saha >Priority: Critical > Fix For: 0.23.2 > > > In MR1 we have the notion of CompletedJobStatus store to aid fast responses > to job.getStatus. We need the equivalent for MR2, an option is to add the > jobStatus to the end of the JobHistory file to which the JHS can easily jump > ahead to and serve the query, it should also cache this for a fair number of > recently completed jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-3837: - Attachment: PATCH-HADOOP-1-MAPREDUCE-3837-2.patch Incorporating review comments > Hadoop 22 Job tracker is not able to recover job in case of crash and after > that no user can submit job. > > > Key: MAPREDUCE-3837 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Fix For: 0.24.0, 0.22.1, 0.23.2 > > Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, > PATCH-HADOOP-1-MAPREDUCE-3837-2.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch, > PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch > > > If job tracker is crashed while running , and there were some jobs are > running , so if job tracker's property mapreduce.jobtracker.restart.recover > is true then it should recover the job. > However the current behavior is as follows > jobtracker try to restore the jobs but it can not . And after that jobtracker > closes its handle to hdfs and nobody else can submit job. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222683#comment-13222683 ] Hudson commented on MAPREDUCE-3964: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1843 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1843/]) MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via bobby) (Revision 1297248) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297248 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222682#comment-13222682 ] Hudson commented on MAPREDUCE-3964: --- Integrated in Hadoop-Mapreduce-0.23-Commit #640 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/640/]) svn merge -c 1297248 from trunk to branch-0.23 FIXES MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via bobby) (Revision 1297250) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297250 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3578) starting nodemanager as 'root' gives "Unknown -jvm option"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated MAPREDUCE-3578:
-
Attachment: MAPREDUCE-3578.patch
Patch to remove jsvc options since jsvc isn't used by node managers (as Vinod
pointed out).
> starting nodemanager as 'root' gives "Unknown -jvm option"
> --
>
> Key: MAPREDUCE-3578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3578
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0
> Environment: jdk 1.6.0.26, ubuntu 11.04
>Reporter: Gilad Wolff
> Attachments: MAPREDUCE-3578.patch
>
>
> running "sudo HADOOP_ROOT/bin/yarn-daemon.sh start nodemanager" I get
> "unknown -jvm option" (jdk version 1.6.0.26). The problem seems to be with
> line 204 in yarn:
> elif [ "$COMMAND" = "nodemanager" ] ; then
> CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
> CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
> if [[ $EUID -eq 0 ]]; then
> YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS"
> else
> YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
> fi
> using -server seems to solve the problem for me.
> I tested using build 929 from
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3578) starting nodemanager as 'root' gives "Unknown -jvm option"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated MAPREDUCE-3578:
-
Status: Patch Available (was: Open)
> starting nodemanager as 'root' gives "Unknown -jvm option"
> --
>
> Key: MAPREDUCE-3578
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3578
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.0
> Environment: jdk 1.6.0.26, ubuntu 11.04
>Reporter: Gilad Wolff
> Attachments: MAPREDUCE-3578.patch
>
>
> running "sudo HADOOP_ROOT/bin/yarn-daemon.sh start nodemanager" I get
> "unknown -jvm option" (jdk version 1.6.0.26). The problem seems to be with
> line 204 in yarn:
> elif [ "$COMMAND" = "nodemanager" ] ; then
> CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
> CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
> if [[ $EUID -eq 0 ]]; then
> YARN_OPTS="$YARN_OPTS -jvm server $YARN_NODEMANAGER_OPTS"
> else
> YARN_OPTS="$YARN_OPTS -server $YARN_NODEMANAGER_OPTS"
> fi
> using -server seems to solve the problem for me.
> I tested using build 929 from
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222677#comment-13222677 ] Hudson commented on MAPREDUCE-3964: --- Integrated in Hadoop-Common-0.23-Commit #640 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/640/]) svn merge -c 1297248 from trunk to branch-0.23 FIXES MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via bobby) (Revision 1297250) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297250 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3973) [Umbrella JIRA] JobHistoryServer performance improvements in YARN+MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3973: --- Summary: [Umbrella JIRA] JobHistoryServer performance improvements in YARN+MR (was: [Umbrella JIRA] JobHistoryServer improvements in YARN+MR) Focusing on performance issues only.. > [Umbrella JIRA] JobHistoryServer performance improvements in YARN+MR > > > Key: MAPREDUCE-3973 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3973 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli > > Few parallel efforts are happening w.r.t improving/fixing issues with > JobHistoryServer in MR over YARN. This is the umbrella ticket so we have the > complete picture. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222673#comment-13222673 ] Hudson commented on MAPREDUCE-3964: --- Integrated in Hadoop-Hdfs-0.23-Commit #630 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/630/]) svn merge -c 1297248 from trunk to branch-0.23 FIXES MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via bobby) (Revision 1297250) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297250 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222670#comment-13222670 ] Hudson commented on MAPREDUCE-3964: --- Integrated in Hadoop-Common-trunk-Commit #1836 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1836/]) MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via bobby) (Revision 1297248) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297248 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3966) Add separate cache for CompletedJobs with tasks loaded and without tasks loaded in JobHIstoryServer
[ https://issues.apache.org/jira/browse/MAPREDUCE-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3966: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-3973 > Add separate cache for CompletedJobs with tasks loaded and without tasks > loaded in JobHIstoryServer > --- > > Key: MAPREDUCE-3966 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3966 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.1 >Reporter: Vinod Kumar Vavilapalli > > After MAPREDUCE-3901, we should have separate caches for list of CompletedJob > with tasks loaded and list of CompletedJob without tasks loaded. Once we have > separate cache, we should increase the cache size for the list which doesn't > load tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3971) Job History web services need to have limits on the number of itmes they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3971: --- Issue Type: Sub-task (was: Improvement) Parent: MAPREDUCE-3973 > Job History web services need to have limits on the number of itmes they can > return. > > > Key: MAPREDUCE-3971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > > The Job History web services canput a very large load on the job history > server. We should put in a limit on the number of entries that can be > returned by the web service, and also add in the ability to modify the > starting location in the list, so that all entries can still be downlaoded. > Just not all at once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222667#comment-13222667 ] Hudson commented on MAPREDUCE-3964: --- Integrated in Hadoop-Hdfs-trunk-Commit #1910 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1910/]) MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via bobby) (Revision 1297248) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297248 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3972) Locking and exception issues in JobHistory Server.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3972: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-3973 > Locking and exception issues in JobHistory Server. > -- > > Key: MAPREDUCE-3972 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3972 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > The JobHistory server's locking is inconsistent and wrong in some cases. > This is not super critical because the issues would only show up if a job is > being cleaned up or moved from intermediate done to done, at the same time it > is being parsed into a CompletedJob. However the locking is slowing down the > server in some cases, and is a ticking time bomb that needs to be addressed. > As part of this too we need to be sure that the Cleaner and Intermediate to > Done migration threads handle exceptions properly. Now it appears that the > exception is logged, and the thread just shuts down. This means that the > history server could still be up and running for weeks and never remove old > jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3905) Allow per job log aggregation configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222663#comment-13222663 ] Siddharth Seth commented on MAPREDUCE-3905: --- In the initial patch, am planning to only support per job enable/disable aggregation. Along with this - an option to keep the logs on the NM for some time if the job disables aggregation. Something like FAILED_ONLY will not work very well rightnow - since container exit status is not necessarily an indication of whether a task completes successfully or not. > Allow per job log aggregation configuration > --- > > Key: MAPREDUCE-3905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > > Currently, if log aggregation is enabled for a cluster - logs for all jobs > will be aggregated - leading to a whole bunch of files on hdfs which users > may not want. > Users should be able to control this along with the aggregation policy - > failed only, all, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3944:
---
Issue Type: Sub-task (was: Bug)
Parent: MAPREDUCE-3973
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3973) [Umbrella JIRA] JobHistoryServer improvements in YARN+MR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3973: --- Summary: [Umbrella JIRA] JobHistoryServer improvements in YARN+MR (was: [] JobHistoryServer improvements in YARN+MR) > [Umbrella JIRA] JobHistoryServer improvements in YARN+MR > > > Key: MAPREDUCE-3973 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3973 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli > > Few parallel efforts are happening w.r.t improving/fixing issues with > JobHistoryServer in MR over YARN. This is the umbrella ticket so we have the > complete picture. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3973) [] JobHistoryServer improvements in YARN+MR
[] JobHistoryServer improvements in YARN+MR --- Key: MAPREDUCE-3973 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3973 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Few parallel efforts are happening w.r.t improving/fixing issues with JobHistoryServer in MR over YARN. This is the umbrella ticket so we have the complete picture. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3964: --- Resolution: Fixed Fix Version/s: 0.23.2 Assignee: Jason Lowe Target Version/s: 0.24.0, 0.23.2, 0.23.3 (was: 0.23.3, 0.23.2, 0.24.0) Status: Resolved (was: Patch Available) Thanks Jason, I just put this into trunk, branch-0.23 and branch-0.23.2 > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222653#comment-13222653 ] Hadoop QA commented on MAPREDUCE-3961: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517128/MR3961.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 57 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobClientGetJob org.apache.hadoop.mapred.TestMRWithDistributedCache org.apache.hadoop.mapred.TestLocalModeWithNewApis +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2003//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2003//console This message is automatically generated. > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222647#comment-13222647 ] Robert Joseph Evans commented on MAPREDUCE-3964: The patch looks good to me +1. I am OK with not having any new unit tests on this one. > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222644#comment-13222644
]
Siddharth Seth commented on MAPREDUCE-3944:
---
Bobby, looks like you missed including MockHistoryJobs in the patch.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3034: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.txt, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, > MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3034: --- Attachment: MAPREDUCE-3034-20120305.txt Taking liberty and making changes myself, this issue has been in a bit of development hell. Will push this in when Jenkins returns. > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.txt, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, > MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3034: --- Fix Version/s: 0.23.2 Target Version/s: 0.23.2 (was: 0.23.2, 0.24.0, 0.23.0) Affects Version/s: (was: 0.23.2) (was: 0.24.0) Status: Open (was: Patch Available) Looked through the patch, fine overall. Making trivial edit like adding comments to the code, renaming a few fields etc. Also we don't need to make changes to CompositeService, we just need to remove the previous shut-down hook, doing this too. Also the fact that we are creating a new NodeManager underneath when RM asks to reboot is a problem. This issue is because we cannot reinit a Service, will file a separate ticket. > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-20120305.txt, MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, > MAPREDUCE-3034-5.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3961: -- Status: Patch Available (was: Open) > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3961) Map/ReduceSlotMillis computation incorrect
[ https://issues.apache.org/jira/browse/MAPREDUCE-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3961: -- Attachment: MR3961.txt Have had to add ClusterInfo to the MR AppContext - so that's it's available in the TaskAttempts. > Map/ReduceSlotMillis computation incorrect > -- > > Key: MAPREDUCE-3961 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3961 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: MR3961.txt > > > Map/ReduceSlot millis are currently computed based on a fixed container size. > They should instead be based on the minimum container size offered by the > cluster. > There's another jira to rename these Counters - based on the resource type. > This jira isn't to do that - just to fix the values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222601#comment-13222601 ] Jason Lowe commented on MAPREDUCE-3964: --- I am unable to reproduce the test failures locally. Possibly a reoccurrence of MAPREDUCE-3729? There appears to be lingering directories on the Hudson server. > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222589#comment-13222589 ] Hadoop QA commented on MAPREDUCE-3964: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517116/MAPREDUCE-3964.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobClientGetJob org.apache.hadoop.mapred.TestMRWithDistributedCache org.apache.hadoop.mapred.TestLocalModeWithNewApis +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2002//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2002//console This message is automatically generated. > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222570#comment-13222570 ] Hudson commented on MAPREDUCE-3954: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1842 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1842/]) MAPREDUCE-3954. Added new envs to separate heap size for different daemons started via bin scripts. Contributed by Robert Joseph Evans. .--This line, and those below, will be ignored-- M hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Mhadoop-mapreduce-project/hadoop-yarn/bin/yarn Mhadoop-mapreduce-project/CHANGES.txt Mhadoop-mapreduce-project/bin/mapred (Revision 1297174) Result = ABORTED vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297174 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Clean up passing HEAPSIZE to yarn and mapred commands. > -- > > Key: MAPREDUCE-3954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3954-20120305.txt, MR-3954.txt, MR-3954.txt, > MR-3954.txt > > > Currently the heap size for all of these is set in yarn-env.sh. > JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set > it will override JAVA_HEAP_MAX. However, we do not always want to have the > RM, NM, and HistoryServer with the exact same heap size. It would be logical > to have inside of yarn and mapred to set JAVA_HEAP_MAX if > YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or > HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug > because it is easy to configure the history server to store more entires then > the heap can hold. It is also a performance issue if we do not allow the > history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3964: -- Target Version/s: 0.24.0, 0.23.2, 0.23.3 Status: Patch Available (was: Open) > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3964) ResourceManager does not have JVM metrics
[ https://issues.apache.org/jira/browse/MAPREDUCE-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3964: -- Attachment: MAPREDUCE-3964.patch Patch to start JvmMetrics instance when ResourceManager starts up. No automated test, but I manually verified via the metrics2 system. I hooked up our metrics gathering tool to the ResourceManager and verified JVM metrics were present after the patch. > ResourceManager does not have JVM metrics > - > > Key: MAPREDUCE-3964 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3964 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.24.0, 0.23.2 >Reporter: Jason Lowe >Priority: Critical > Attachments: MAPREDUCE-3964.patch > > > ResourceManager is not creating a JvmMetrics instance on startup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222544#comment-13222544
]
Hadoop QA commented on MAPREDUCE-3944:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517109/MR-3944.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2001//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2001//console
This message is automatically generated.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222538#comment-13222538 ] Hadoop QA commented on MAPREDUCE-3954: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12517108/MAPREDUCE-3954-20120305.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobClientGetJob org.apache.hadoop.mapred.TestMRWithDistributedCache org.apache.hadoop.mapred.TestLocalModeWithNewApis +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2000//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2000//console This message is automatically generated. > Clean up passing HEAPSIZE to yarn and mapred commands. > -- > > Key: MAPREDUCE-3954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3954-20120305.txt, MR-3954.txt, MR-3954.txt, > MR-3954.txt > > > Currently the heap size for all of these is set in yarn-env.sh. > JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set > it will override JAVA_HEAP_MAX. However, we do not always want to have the > RM, NM, and HistoryServer with the exact same heap size. It would be logical > to have inside of yarn and mapred to set JAVA_HEAP_MAX if > YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or > HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug > because it is easy to configure the history server to store more entires then > the heap can hold. It is also a performance issue if we do not allow the > history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222539#comment-13222539 ] Hudson commented on MAPREDUCE-3954: --- Integrated in Hadoop-Mapreduce-0.23-Commit #639 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/639/]) MAPREDUCE-3954. Added new envs to separate heap size for different daemons started via bin scripts. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1297174 ../../trunk/ (Revision 1297176) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297176 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Clean up passing HEAPSIZE to yarn and mapred commands. > -- > > Key: MAPREDUCE-3954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3954-20120305.txt, MR-3954.txt, MR-3954.txt, > MR-3954.txt > > > Currently the heap size for all of these is set in yarn-env.sh. > JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set > it will override JAVA_HEAP_MAX. However, we do not always want to have the > RM, NM, and HistoryServer with the exact same heap size. It would be logical > to have inside of yarn and mapred to set JAVA_HEAP_MAX if > YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or > HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug > because it is easy to configure the history server to store more entires then > the heap can hold. It is also a performance issue if we do not allow the > history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222535#comment-13222535
]
Robert Joseph Evans commented on MAPREDUCE-3944:
I filed MAPREDUCE-3971 for limiting the number of times that the web services
returns, and MAPREDUCE-3972 to fix the locking in the server.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222532#comment-13222532 ] Hudson commented on MAPREDUCE-3954: --- Integrated in Hadoop-Common-trunk-Commit #1835 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1835/]) MAPREDUCE-3954. Added new envs to separate heap size for different daemons started via bin scripts. Contributed by Robert Joseph Evans. .--This line, and those below, will be ignored-- M hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Mhadoop-mapreduce-project/hadoop-yarn/bin/yarn Mhadoop-mapreduce-project/CHANGES.txt Mhadoop-mapreduce-project/bin/mapred (Revision 1297174) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297174 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Clean up passing HEAPSIZE to yarn and mapred commands. > -- > > Key: MAPREDUCE-3954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3954-20120305.txt, MR-3954.txt, MR-3954.txt, > MR-3954.txt > > > Currently the heap size for all of these is set in yarn-env.sh. > JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set > it will override JAVA_HEAP_MAX. However, we do not always want to have the > RM, NM, and HistoryServer with the exact same heap size. It would be logical > to have inside of yarn and mapred to set JAVA_HEAP_MAX if > YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or > HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug > because it is easy to configure the history server to store more entires then > the heap can hold. It is also a performance issue if we do not allow the > history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3972) Locking and exception issues in JobHistory Server.
Locking and exception issues in JobHistory Server. -- Key: MAPREDUCE-3972 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3972 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans The JobHistory server's locking is inconsistent and wrong in some cases. This is not super critical because the issues would only show up if a job is being cleaned up or moved from intermediate done to done, at the same time it is being parsed into a CompletedJob. However the locking is slowing down the server in some cases, and is a ticking time bomb that needs to be addressed. As part of this too we need to be sure that the Cleaner and Intermediate to Done migration threads handle exceptions properly. Now it appears that the exception is logged, and the thread just shuts down. This means that the history server could still be up and running for weeks and never remove old jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222531#comment-13222531 ] Hudson commented on MAPREDUCE-3954: --- Integrated in Hadoop-Common-0.23-Commit #639 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/639/]) MAPREDUCE-3954. Added new envs to separate heap size for different daemons started via bin scripts. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1297174 ../../trunk/ (Revision 1297176) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297176 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Clean up passing HEAPSIZE to yarn and mapred commands. > -- > > Key: MAPREDUCE-3954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3954-20120305.txt, MR-3954.txt, MR-3954.txt, > MR-3954.txt > > > Currently the heap size for all of these is set in yarn-env.sh. > JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set > it will override JAVA_HEAP_MAX. However, we do not always want to have the > RM, NM, and HistoryServer with the exact same heap size. It would be logical > to have inside of yarn and mapred to set JAVA_HEAP_MAX if > YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or > HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug > because it is easy to configure the history server to store more entires then > the heap can hold. It is also a performance issue if we do not allow the > history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222524#comment-13222524 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3034: Looking at the patch for final review/commit.. > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0, 0.24.0, 0.23.2 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034-4.patch, MAPREDUCE-3034-5.patch, > MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3009) RM UI -> Applications -> Application(Job History) -> Map Tasks -> Task ID -> Node link is not working
[ https://issues.apache.org/jira/browse/MAPREDUCE-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222527#comment-13222527 ] Hudson commented on MAPREDUCE-3009: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1841 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1841/]) MAPREDUCE-3009. Fixed node link on JobHistory webapp. Contributed by hackaravarthy. (Revision 1297149) Result = ABORTED vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297149 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java > RM UI -> Applications -> Application(Job History) -> Map Tasks -> Task ID -> > Node link is not working > - > > Key: MAPREDUCE-3009 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3009 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: chackaravarthy >Assignee: chackaravarthy > Fix For: 0.23.2 > > Attachments: IE_node_link_not_working.png, MAPREDUCE-3009-1.patch, > MAPREDUCE-3009.patch, MAPREDUCE-3009.patch, chrome_node_link_working.png, > invalid_url.png, invalid_url_latest_code.png > > > RM UI -> Applications -> Application(Job History) -> Map Tasks -> Task ID -> > Node link is not working. The URL contains extra '/' which is causing the > problem. Please find in the attached screen shots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222526#comment-13222526 ] Hudson commented on MAPREDUCE-3954: --- Integrated in Hadoop-Hdfs-trunk-Commit #1909 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1909/]) MAPREDUCE-3954. Added new envs to separate heap size for different daemons started via bin scripts. Contributed by Robert Joseph Evans. .--This line, and those below, will be ignored-- M hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm Mhadoop-mapreduce-project/hadoop-yarn/bin/yarn Mhadoop-mapreduce-project/CHANGES.txt Mhadoop-mapreduce-project/bin/mapred (Revision 1297174) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297174 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Clean up passing HEAPSIZE to yarn and mapred commands. > -- > > Key: MAPREDUCE-3954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3954-20120305.txt, MR-3954.txt, MR-3954.txt, > MR-3954.txt > > > Currently the heap size for all of these is set in yarn-env.sh. > JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set > it will override JAVA_HEAP_MAX. However, we do not always want to have the > RM, NM, and HistoryServer with the exact same heap size. It would be logical > to have inside of yarn and mapred to set JAVA_HEAP_MAX if > YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or > HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug > because it is easy to configure the history server to store more entires then > the heap can hold. It is also a performance issue if we do not allow the > history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3954) Clean up passing HEAPSIZE to yarn and mapred commands.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222525#comment-13222525 ] Hudson commented on MAPREDUCE-3954: --- Integrated in Hadoop-Hdfs-0.23-Commit #629 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/629/]) MAPREDUCE-3954. Added new envs to separate heap size for different daemons started via bin scripts. Contributed by Robert Joseph Evans. svn merge --ignore-ancestry -c 1297174 ../../trunk/ (Revision 1297176) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297176 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm > Clean up passing HEAPSIZE to yarn and mapred commands. > -- > > Key: MAPREDUCE-3954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3954 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3954-20120305.txt, MR-3954.txt, MR-3954.txt, > MR-3954.txt > > > Currently the heap size for all of these is set in yarn-env.sh. > JAVA_HEAP_MAX is set to -Xmx1000m unless YARN_HEAPSIZE is set. If it is set > it will override JAVA_HEAP_MAX. However, we do not always want to have the > RM, NM, and HistoryServer with the exact same heap size. It would be logical > to have inside of yarn and mapred to set JAVA_HEAP_MAX if > YARN_RESOURCEMANAGER_HEAPSIZE, YARN_NODEMANAGER_HEAPSIZE or > HADOOP_JOB_HISTORYSERVER_HEAPSIZE are set respectively. This is a bug > because it is easy to configure the history server to store more entires then > the heap can hold. It is also a performance issue if we do not allow the > history server to cache many entries on a large cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3971) Job History web services need to have limits on the number of itmes they can return.
Job History web services need to have limits on the number of itmes they can return. Key: MAPREDUCE-3971 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans The Job History web services canput a very large load on the job history server. We should put in a limit on the number of entries that can be returned by the web service, and also add in the ability to modify the starting location in the list, so that all entries can still be downlaoded. Just not all at once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
