[jira] [Updated] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated MAPREDUCE-3818:
---
Attachment: MAPREDUCE-3818.txt
DFSClient, ClientProtocol etc. is is classified as private interface. The tests
should be using FileSystem interface. We should create a bug to track this.
Vinod, can you please run a quick validation for this patch and commit it. I do
not have my mapreduce env completely setup.
> Trunk MRV1 compilation is broken.
> -
>
> Key: MAPREDUCE-3818
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
>Affects Versions: 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3818.txt
>
>
> Seeing this:
> {code}
> [javac]
> /Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
> cannot find symbol
> [javac] symbol : class ClientNamenodeWireProtocol
> [javac] location: class org.apache.hadoop.mapred.TestSubmitJob
> [javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
> [javac]^
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201551#comment-13201551 ] Robert Joseph Evans commented on MAPREDUCE-3034: The patch looks good to me too. But it is a complex enough change that I don't feel comfortable committing it myself, I think Arun or Mahadev should look at the patch. Also could you look at the TestJobCounters test failure. It looks like the failure is sporadic, and not related to this patch, but I could not find an existing JIRA for it. Could you at least file a new JIRA for it? > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3628) DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201547#comment-13201547 ] Amol Kekre commented on MAPREDUCE-3628: --- Throughput decrease was 10% with 6% CPU slowdown a few wks ago. Waiting for new numbers later this wk. > DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 > nodes size cluster. > - > > Key: MAPREDUCE-3628 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3628 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Amol Kekre >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 0.23.1 > > > DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 > nodes size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201541#comment-13201541 ] Hadoop QA commented on MAPREDUCE-3709: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513463/MR-3709.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1790//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1790//console This message is automatically generated. > TestDistributedShell is failing > --- > > Key: MAPREDUCE-3709 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.1 >Reporter: Eli Collins >Assignee: Hitesh Shah > Attachments: MR-3709.1.patch > > > TestDistributedShell#testDSShell is failing the assert on line 90 on > branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201543#comment-13201543 ] Hadoop QA commented on MAPREDUCE-3773: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513479/mr-3773-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1792//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1792//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1792//console This message is automatically generated. > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3794: - Attachment: MAPREDUCE-3794.patch Thanks Mahadev. Here's a corrected patch. > Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for > compatibility > > > Key: MAPREDUCE-3794 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-3794.patch, MAPREDUCE-3794.patch > > > The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we > should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201534#comment-13201534 ] Hadoop QA commented on MAPREDUCE-3812: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513476/MAPREDUCE-3812-20120206.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1791//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1791//console This message is automatically generated. > Change default memory slot sizes to be 1.5GB > > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201528#comment-13201528 ] Arpit Gupta commented on MAPREDUCE-3817: no tests added as changes are to shell scripts. Ran distcp and archive jobs using bin/mapred with these changes successfully. > bin/mapred command cannot run distcp and archive jobs > - > > Key: MAPREDUCE-3817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: MAPREDUCE-3817.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201522#comment-13201522 ] Hadoop QA commented on MAPREDUCE-3817: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513459/MAPREDUCE-3817.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1789//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1789//console This message is automatically generated. > bin/mapred command cannot run distcp and archive jobs > - > > Key: MAPREDUCE-3817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: MAPREDUCE-3817.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3808:
---
Target Version/s: 0.23.1, 0.24.0
Affects Version/s: 0.24.0
Status: Patch Available (was: Open)
> NPE in FileOutputCommitter when running a 0 reduce job
> --
>
> Key: MAPREDUCE-3808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Siddharth Seth
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3808.txt
>
>
> This was while running LoadGen.
> {noformat}
> Error: java.lang.NullPointerException at
> org.apache.hadoop.fs.Path.(Path.java:67)
> at org.apache.hadoop.fs.Path.(Path.java:56)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
>
> at
> org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
>
> at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
> at org.apache.hadoop.mapred.Task.done(Task.java:912)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3808:
---
Attachment: MR-3808.txt
This patch should fix the issue. It adds in a check for null on the output in
several places when building paths and short curcits those methods to return
null. It also adds in the a null output check in recoverTask and causes it to
be a noop when output is null. Finally it adds in tests to validate that the
FileOutputCommitter handles that case correctly.
> NPE in FileOutputCommitter when running a 0 reduce job
> --
>
> Key: MAPREDUCE-3808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Siddharth Seth
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3808.txt
>
>
> This was while running LoadGen.
> {noformat}
> Error: java.lang.NullPointerException at
> org.apache.hadoop.fs.Path.(Path.java:67)
> at org.apache.hadoop.fs.Path.(Path.java:56)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
>
> at
> org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
>
> at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
> at org.apache.hadoop.mapred.Task.done(Task.java:912)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201516#comment-13201516 ] Mahadev konar commented on MAPREDUCE-3697: -- Thanks for doing this Tom. Ill take care of those! > Hadoop Counters API limits Oozie's working across different hadoop versions > --- > > Key: MAPREDUCE-3697 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: John George >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3697-facade.patch, > MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, > MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, > MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, > apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh > > > Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in > hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). > Its > parent class AbstractCounters implements it. This is different from hadoop20X. > As a result, Oozie compiled with either hadoop version does not work with the > other version. > A specific scenario, Oozie compiled with .23 and run against 205, does not > update job status owing to a Counters API exception. > Will explicit re-compilation against the relevant hadoop jars be required each > time? This will prevent launching a uniform Oozie version across different > clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201517#comment-13201517 ] Hadoop QA commented on MAPREDUCE-3697: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513480/apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1793//console This message is automatically generated. > Hadoop Counters API limits Oozie's working across different hadoop versions > --- > > Key: MAPREDUCE-3697 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: John George >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3697-facade.patch, > MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, > MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, > MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, > apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh > > > Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in > hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). > Its > parent class AbstractCounters implements it. This is different from hadoop20X. > As a result, Oozie compiled with either hadoop version does not work with the > other version. > A specific scenario, Oozie compiled with .23 and run against 205, does not > update job status owing to a Counters API exception. > Will explicit re-compilation against the relevant hadoop jars be required each > time? This will prevent launching a uniform Oozie version across different > clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3697: - Attachment: apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh I reviewed the patch for compatibility with Hadoop 1.x. I ran a SigTest compatibility test (like in HADOOP-7738) to see what the differences between the mapred counters are between 1.0.0 and trunk with this patch (see attached). Most are false positives due to the inheritance hierarchy changing. Here are the ones that I think need fixing: * Counters.getGroupNames returns an Iterable not a Collection as it does in 1.x. I think we should change it back to return Collection - we can always add a getGroupNamesIterable() later if needed. * method public java.lang.String org.apache.hadoop.mapred.Counters.makeCompactString() is missing * public static int org.apache.hadoop.mapred.Counters.MAX_COUNTER_LIMIT is missing (it's not used in 23, but we should still define it for compatibility) > Hadoop Counters API limits Oozie's working across different hadoop versions > --- > > Key: MAPREDUCE-3697 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: John George >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3697-facade.patch, > MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, > MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, > MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, > apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh > > > Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in > hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). > Its > parent class AbstractCounters implements it. This is different from hadoop20X. > As a result, Oozie compiled with either hadoop version does not work with the > other version. > A specific scenario, Oozie compiled with .23 and run against 205, does not > update job status owing to a Counters API exception. > Will explicit re-compilation against the relevant hadoop jars be required each > time? This will prevent launching a uniform Oozie version across different > clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Target Version/s: 0.23.1, 1.1.0 (was: 1.1.0, 0.23.1) Status: Open (was: Patch Available) > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Target Version/s: 0.23.1, 1.1.0 (was: 1.1.0, 0.23.1) Status: Patch Available (was: Open) resubmitting > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Attachment: mr-3773-trunk.patch Fixed findbugs warnings and add missing Configurable interface from CapacityScheduler. > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201506#comment-13201506
]
Aaron T. Myers commented on MAPREDUCE-3818:
---
Seems very likely this is due to the recent commit of HDFS-2895.
> Trunk MRV1 compilation is broken.
> -
>
> Key: MAPREDUCE-3818
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
>Affects Versions: 0.24.0
>Reporter: Vinod Kumar Vavilapalli
>Priority: Blocker
> Fix For: 0.24.0
>
>
> Seeing this:
> {code}
> [javac]
> /Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
> cannot find symbol
> [javac] symbol : class ClientNamenodeWireProtocol
> [javac] location: class org.apache.hadoop.mapred.TestSubmitJob
> [javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
> [javac]^
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.24.0
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Status: Open (was: Patch Available) > TestDistributedShell is failing > --- > > Key: MAPREDUCE-3709 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.1 >Reporter: Eli Collins >Assignee: Hitesh Shah > Attachments: MR-3709.1.patch > > > TestDistributedShell#testDSShell is failing the assert on line 90 on > branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Status: Patch Available (was: Open) > Change default memory slot sizes to be 1.5GB > > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS returns IPs in getFileBlockLocations
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201497#comment-13201497 ] Siddharth Seth commented on MAPREDUCE-3815: --- Both - will create a separate hdfs jira. This one is for a check in MR for such situations. > Data Locality suffers if HDFS returns IPs in getFileBlockLocations > -- > > Key: MAPREDUCE-3815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > > BlockLocation.getHosts() returns IP addresses occasionally. Data locality is > affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS returns IPs in getFileBlockLocations
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201498#comment-13201498 ] Siddharth Seth commented on MAPREDUCE-3815: --- Both - will create a separate hdfs jira. This one is for a check in MR for such situations. > Data Locality suffers if HDFS returns IPs in getFileBlockLocations > -- > > Key: MAPREDUCE-3815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Critical > > BlockLocation.getHosts() returns IP addresses occasionally. Data locality is > affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Status: Open (was: Patch Available) > Change default memory slot sizes to be 1.5GB > > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Attachment: MAPREDUCE-3812-20120206.txt Correct patch applicable to trunk. > Change default memory slot sizes to be 1.5GB > > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201480#comment-13201480 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3813: Allen, this is not for hostname resolution. This is the rack resolver plugin which finds out the racks for a given set of hosts. Not every rack resolver plugin is fast enough, so we need a cache to keep track of hosts whose racks are already resolved. > RackResolver should maintain a cache to avoid repetitive lookups. > - > > Key: MAPREDUCE-3813 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3813-20120205.txt > > > With the current code, during task creation, we repeatedly resolve hosts and > RackResolver doesn't cache any of the results. Caching will improve > performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201476#comment-13201476
]
Robert Joseph Evans commented on MAPREDUCE-3808:
Well looking at the rest of the code it looks like the FileOutputCommitter
becomes a NOOP when the output dir is not set, so I need to replicate that here
too.
> NPE in FileOutputCommitter when running a 0 reduce job
> --
>
> Key: MAPREDUCE-3808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Siddharth Seth
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> This was while running LoadGen.
> {noformat}
> Error: java.lang.NullPointerException at
> org.apache.hadoop.fs.Path.(Path.java:67)
> at org.apache.hadoop.fs.Path.(Path.java:56)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
>
> at
> org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
>
> at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
> at org.apache.hadoop.mapred.Task.done(Task.java:912)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201474#comment-13201474
]
Robert Joseph Evans commented on MAPREDUCE-3808:
OK I am a bit confused here. Why is it allowed to run a map only job that does
not have an output directory set? The FileOutputFormat blows up if there are
any reduces and the output directory is not set, but for a map only job it is
fine. What is supposed to happen in that situation? I can replicate it, but I
don't know what the previous behavior was.
> NPE in FileOutputCommitter when running a 0 reduce job
> --
>
> Key: MAPREDUCE-3808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Siddharth Seth
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> This was while running LoadGen.
> {noformat}
> Error: java.lang.NullPointerException at
> org.apache.hadoop.fs.Path.(Path.java:67)
> at org.apache.hadoop.fs.Path.(Path.java:56)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
>
> at
> org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
>
> at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
> at org.apache.hadoop.mapred.Task.done(Task.java:912)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201471#comment-13201471
]
Robert Joseph Evans commented on MAPREDUCE-3808:
Sure I can look into it. It appears that for some reason
mapreduce.output.fileoutputformat.outputdir is not set in the config.
> NPE in FileOutputCommitter when running a 0 reduce job
> --
>
> Key: MAPREDUCE-3808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Siddharth Seth
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> This was while running LoadGen.
> {noformat}
> Error: java.lang.NullPointerException at
> org.apache.hadoop.fs.Path.(Path.java:67)
> at org.apache.hadoop.fs.Path.(Path.java:56)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
>
> at
> org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
>
> at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
> at org.apache.hadoop.mapred.Task.done(Task.java:912)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Status: Patch Available (was: Open) > TestDistributedShell is failing > --- > > Key: MAPREDUCE-3709 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.1 >Reporter: Eli Collins >Assignee: Hitesh Shah > Attachments: MR-3709.1.patch > > > TestDistributedShell#testDSShell is failing the assert on line 90 on > branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Priority: Major (was: Blocker) > TestDistributedShell is failing > --- > > Key: MAPREDUCE-3709 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.1 >Reporter: Eli Collins >Assignee: Hitesh Shah > Attachments: MR-3709.1.patch > > > TestDistributedShell#testDSShell is failing the assert on line 90 on > branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Attachment: MR-3709.1.patch @Eli, given that MAPREDUCE-3765 is now in, the latest patch should be able to use smaller sized containers which hopefully should solve the problem. Could you please try the patch and let me know if it fixes the issue in your env. I am also downgrading this to a non-blocker as it is not reproducible in most environments. Please let me know if you see any issues with doing this. > TestDistributedShell is failing > --- > > Key: MAPREDUCE-3709 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.1 >Reporter: Eli Collins >Assignee: Hitesh Shah >Priority: Blocker > Attachments: MR-3709.1.patch > > > TestDistributedShell#testDSShell is failing the assert on line 90 on > branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3817: - Component/s: mrv2 > bin/mapred command cannot run distcp and archive jobs > - > > Key: MAPREDUCE-3817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: MAPREDUCE-3817.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3817: - Status: Patch Available (was: Open) > bin/mapred command cannot run distcp and archive jobs > - > > Key: MAPREDUCE-3817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: MAPREDUCE-3817.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reassigned MAPREDUCE-3808:
--
Assignee: Robert Joseph Evans
Rober, can you please take a crack at this one?
> NPE in FileOutputCommitter when running a 0 reduce job
> --
>
> Key: MAPREDUCE-3808
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Siddharth Seth
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> This was while running LoadGen.
> {noformat}
> Error: java.lang.NullPointerException at
> org.apache.hadoop.fs.Path.(Path.java:67)
> at org.apache.hadoop.fs.Path.(Path.java:56)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
>
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
>
> at
> org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
>
> at
> org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
>
> at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
> at org.apache.hadoop.mapred.Task.done(Task.java:912)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta updated MAPREDUCE-3817: --- Attachment: MAPREDUCE-3817.patch added same options available in bin/hadoop also to bin/mapred for distcp and archives > bin/mapred command cannot run distcp and archive jobs > - > > Key: MAPREDUCE-3817 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.1, 0.24.0 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: MAPREDUCE-3817.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201414#comment-13201414
]
Robert Joseph Evans commented on MAPREDUCE-3802:
I have been looking for differences in the jhist files between the initial
kill, and the second one, and I cannot find much as far as the content is
concerned. So I think it might have something to do with the order of the
events.
In the second jhist file there are two AMStarted events. I assume that is to
allow for the history server to show how many have been started.
The mapFinishTime of all the new MAP_ATTEMPT_FINISH_EVENTS is 0. The finishTime
for these events are all showing the same finish time which I think is when the
attempt was recovered, not the original event finish time. The state of the
event changed from "map" (which seems like a bug) to "SUCCEEDED" which looks
more correct to me. And finally the clockSplits counters are all 0 in the new
one as well.
Similarly the TASK_FINISH_EVENTS have the finish time of the recovery, not the
actual finish time.
I could not find anything else that is significantly different.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201386#comment-13201386 ] Hadoop QA commented on MAPREDUCE-3034: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513433/MAPREDUCE-3034-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobCounters +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1787//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1787//console This message is automatically generated. > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201383#comment-13201383 ] Hadoop QA commented on MAPREDUCE-3773: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513435/mr-3773-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits org.apache.hadoop.yarn.server.resourcemanager.TestRM org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1788//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1788//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1788//console This message is automatically generated. > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Target Version/s: 0.23.1, 1.1.0 Status: Patch Available (was: Open) > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Attachment: mr-3773-trunk.patch This patch applies to 0.23 and trunk. It renames the property to yarn.resourcemanager.metrics.runtime.buckets I had to make the CapacityScheduler Configurable so that it could get the YarnConfiguration. I also needed to add the YarnConfiguration to QueueMetrics.forQueue so that the QueueMetrics object would have access to the YarnConfiguration. > Add queue metrics with buckets for job run times > > > Key: MAPREDUCE-3773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobtracker >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: mr-3773-trunk.patch, mr-3773.patch > > > It would be nice to have queue metrics that reflect the number of jobs in > each queue that have been running for different ranges of time. > Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but > they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201360#comment-13201360 ] Eric Payne commented on MAPREDUCE-3034: --- +1 (non-binding) I approve the changes introduced in MAPREDUCE-3034-3.patch > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3034: - Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) Thanks a lot Robert for taking look at the patch. I have updated the patch with the above comments fixed. > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3034: - Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3034: - Attachment: MAPREDUCE-3034-3.patch > NM should act on a REBOOT command from RM > - > > Key: MAPREDUCE-3034 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, nodemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Devaraj K >Priority: Critical > Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, > MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt > > > RM sends a reboot command to NM in some cases, like when it gets lost and > rejoins back. In such a case, NM should act on the command and > reboot/reinitalize itself. > This is akin to TT reinitialize on order from JT. We will need to shutdown > all the services properly and reinitialize - this should automatically take > care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3816) capacity scheduler web ui bar graphs for used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3816: - Attachment: capacityschedulerusedcap.png In this screen shots, this is single node cluster 8 GB total memory. The default queue capacity is 20% which would be 1638MB allocated to it. You will see that its currently using 2G so the usage is 125% - which is the utilization. > capacity scheduler web ui bar graphs for used capacity wrong > > > Key: MAPREDUCE-3816 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3816 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: capacityschedulerusedcap.png > > > The capacity scheduler web ui has bar graphs showing the capacity/used > capacity/max capacity for each queue. The used capacity it is showing is > actually the % of its parents queue it is using, which doesn't make sense on > the bar graphs when compared to the capacity and max capacity of that > particular queue. The bar graphs should be using utilization so that the > user can see that its using x% or the y% allocated to that queue. > I will attach some screen shots showing the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3816) capacity scheduler web ui bar graphs for used capacity wrong
capacity scheduler web ui bar graphs for used capacity wrong Key: MAPREDUCE-3816 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3816 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical The capacity scheduler web ui has bar graphs showing the capacity/used capacity/max capacity for each queue. The used capacity it is showing is actually the % of its parents queue it is using, which doesn't make sense on the bar graphs when compared to the capacity and max capacity of that particular queue. The bar graphs should be using utilization so that the user can see that its using x% or the y% allocated to that queue. I will attach some screen shots showing the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201329#comment-13201329
]
Robert Joseph Evans commented on MAPREDUCE-3802:
That appears to be the case. We are getting an NPE which is caused by calling
RecoverService.getTaskAttemptInfo() and getting a null back.
RecoveryService.getTaskAttemptInfo() first gets a task info, and then gets a
task attempt info from inside that task. It looks like the task info is parsed
and populated just fine, but the task attempt info is not. That seems to be
caused by no TaskAttemptStarted events being put in the history log at all
during the recovery process. This also seems like no MapAttemptFinishedEvents,
ReduceAttemp0tFinishedEvents, TaskAttemptFailedEvents nor
TaskAttemptFinishedEvents are in the log either, or we would get null pointer
exceptions while parsing them too.
> If an MR AM dies twice it looks like the process freezes
> -
>
> Key: MAPREDUCE-3802
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Robert Joseph Evans
>Priority: Critical
> Attachments: syslog
>
>
> It looks like recovering from an RM AM dieing works very well on a single
> failure. But if it fails multiple times we appear to get into a live lock
> situation.
> {noformat}
> yarn jar
> hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
> wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
> input output
> 12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
> Instead, use fs.defaultFS
> 12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
> is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> 12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
> 17
> 12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
> 12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
> 12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
> application_1328302034486_0003 to ResourceManager at HOST/IP:8040
> 12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
> http://HOST:8088/proxy/application_1328302034486_0003/
> 12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
> 12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
> uber mode : false
> 12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
> 12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
> 12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
> #KILLED AM with kill -9 here
> 12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
> 12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
> 12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
> 12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
> 12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
> 12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
> 12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
> 12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
> 12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
> 12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
> 12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
> 12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
> 12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
> 12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
> 12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
> #killed AM with kill -9 here
> 12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 0 time(s).
> 12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 1 time(s).
> 12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
> Already tried 2 time(s).
> 12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
> #It never makes any more progress...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3747) Memory Total is not refreshed until an app is launched
[ https://issues.apache.org/jira/browse/MAPREDUCE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201295#comment-13201295 ] Hudson commented on MAPREDUCE-3747: --- Integrated in Hadoop-Mapreduce-trunk #981 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/981/]) MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240886 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java > Memory Total is not refreshed until an app is launched > -- > > Key: MAPREDUCE-3747 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3747 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ramya Sunil >Assignee: Arun C Murthy > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, > MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch > > > Memory Total on the RM UI is not refreshed until an application is launched. > This is a problem when the cluster is started for the first time or when > there are any lost/decommissioned NMs. > When the cluster is started for the first time, Active Nodes is > 0 but the > Memory Total=0. Also when there are any lost/decommissioned nodes, Memory > Total has wrong value. > This is a useful tool for cluster admins and has to be updated correctly > without having the need to submit an app each time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3746) Nodemanagers are not automatically shut down after decommissioning
[ https://issues.apache.org/jira/browse/MAPREDUCE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201266#comment-13201266 ] Hudson commented on MAPREDUCE-3746: --- Integrated in Hadoop-Mapreduce-0.23-Build #183 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/183/]) Merge -c 1240886 from trunk to branch-0.23 to fix MAPREDUCE-3746. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240887 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java > Nodemanagers are not automatically shut down after decommissioning > -- > > Key: MAPREDUCE-3746 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3746 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.1 > > > Nodemanagers are not automatically shutdown after decommissioning. > MAPREDUCE-2775 does not seem to fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3746) Nodemanagers are not automatically shut down after decommissioning
[ https://issues.apache.org/jira/browse/MAPREDUCE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201247#comment-13201247 ] Hudson commented on MAPREDUCE-3746: --- Integrated in Hadoop-Hdfs-0.23-Build #161 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/161/]) Merge -c 1240886 from trunk to branch-0.23 to fix MAPREDUCE-3746. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240887 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java > Nodemanagers are not automatically shut down after decommissioning > -- > > Key: MAPREDUCE-3746 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3746 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Critical > Fix For: 0.23.1 > > > Nodemanagers are not automatically shutdown after decommissioning. > MAPREDUCE-2775 does not seem to fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3747) Memory Total is not refreshed until an app is launched
[ https://issues.apache.org/jira/browse/MAPREDUCE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201244#comment-13201244 ] Hudson commented on MAPREDUCE-3747: --- Integrated in Hadoop-Hdfs-trunk #948 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/948/]) MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240886 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java > Memory Total is not refreshed until an app is launched > -- > > Key: MAPREDUCE-3747 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3747 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Ramya Sunil >Assignee: Arun C Murthy > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, > MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch > > > Memory Total on the RM UI is not refreshed until an application is launched. > This is a problem when the cluster is started for the first time or when > there are any lost/decommissioned NMs. > When the cluster is started for the first time, Active Nodes is > 0 but the > Memory Total=0. Also when there are any lost/decommissioned nodes, Memory > Total has wrong value. > This is a useful tool for cluster admins and has to be updated correctly > without having the need to submit an app each time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
