[jira] [Updated] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3762:
-
Resolution: Fixed
Fix Version/s: 0.23.1
Status: Resolved (was: Patch Available)
I just committed this. Thanks Mahadev!
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197664#comment-13197664
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Hdfs-0.23-Commit #449 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/449/])
Merge -c 1238987 from trunk to branch-0.23 to fix MAPREDUCE-3762. Fixed
default CapacityScheduler configs.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238989
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197665#comment-13197665
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1702 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1702/])
MAPREDUCE-3762. Fixed default CapacityScheduler configs. Contributed by
Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238987
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197666#comment-13197666
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Common-trunk-Commit #1631 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1631/])
MAPREDUCE-3762. Fixed default CapacityScheduler configs. Contributed by
Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238987
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197667#comment-13197667
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Common-0.23-Commit #459 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/459/])
Merge -c 1238987 from trunk to branch-0.23 to fix MAPREDUCE-3762. Fixed
default CapacityScheduler configs.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238989
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197668#comment-13197668 ] Hadoop QA commented on MAPREDUCE-3640: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512724/MAPREDUCE-3640.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 3 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1733//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1733//console This message is automatically generated. AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch, MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3640: - Attachment: MAPREDUCE-3640.patch Fixed javadoc warnings. AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch, MAPREDUCE-3640.patch, MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3640: - Status: Open (was: Patch Available) AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch, MAPREDUCE-3640.patch, MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3640: - Status: Patch Available (was: Open) AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch, MAPREDUCE-3640.patch, MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3768) MR-2450 introduced a significant performance regression
[
https://issues.apache.org/jira/browse/MAPREDUCE-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197680#comment-13197680
]
Arun C Murthy commented on MAPREDUCE-3768:
--
Eli, the question was, did you look into the root cause. Yes, it could have
caused a regression or just exposed a bug in Hive - I've seen several examples
of both.
MR-2450 introduced a significant performance regression
---
Key: MAPREDUCE-3768
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3768
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Eli Collins
Priority: Blocker
MAPREDUCE-2450 introduced, or at least triggers, a significant performance
regression in Hive. With MR-2450 the execution time of TestCliDriver.skewjoin
goes from 2 minutes to 15 minutes. Reverting this change from the build fixes
the issue.
Here's the relevant query:
{noformat}
FROM src src1 JOIN src src2 ON (src1.key = src2.key)
INSERT OVERWRITE TABLE dest_j1 SELECT src1.key, src2.value;
{noformat}
You can reproduce this by running the following from Hive 8.0 against Hadoop
built from branch-23.
{noformat}
ant very-clean package test -Dtestcase=TestCliDriver -Dqfile=skewjoin.q
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3427:
-
Status: Open (was: Patch Available)
Hitesh, just use @Ignore for the tests?
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1, 0.24.0
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197694#comment-13197694
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1647 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1647/])
MAPREDUCE-3762. Fixed default CapacityScheduler configs. Contributed by
Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238987
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197704#comment-13197704
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #473 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/473/])
Merge -c 1238987 from trunk to branch-0.23 to fix MAPREDUCE-3762. Fixed
default CapacityScheduler configs.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238989
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197718#comment-13197718 ] Hadoop QA commented on MAPREDUCE-3640: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512731/MAPREDUCE-3640.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1734//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1734//console This message is automatically generated. AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch, MAPREDUCE-3640.patch, MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3582: Attachment: 3582.Missing_Rumen_tests.patch Some Rumen unit tests are missing in hadoop-tools/ in trunk and 0.23, which were passing. Is this commit has some problem ? Missing rumen unit tests are: TestRumenJobTraces.java TestParsedLine.java TestRumenAnonymization.java TestZombieJob.java TestConcurrentRead.java TestRumenFolder.java And all these tests were passing. Am attaching a patch with these files copied to hadoop-tools/hadoop-rumen/src/test/java/org/apache/hadoop/tools/rumen/. Am not sure if any other files from other folders are also missing. Please recheck once. Thanks. Move successfully passing MR1 tests to MR2 maven tree. -- Key: MAPREDUCE-3582 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Ahmed Radwan Assignee: Ahmed Radwan Fix For: 0.23.1 Attachments: 3582.Missing_Rumen_tests.patch, MAPREDUCE-3582-part1_rev3.patch, MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part1_rev5.patch, MAPREDUCE-3582-part1_rev6.patch, MAPREDUCE-3582-part1_rev7.patch, MAPREDUCE-3582-part1_rev8.patch, MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, MAPREDUCE-3582-part2_rev5.patch, MAPREDUCE-3582-part2_rev6.patch, MAPREDUCE-3582-part2_rev7.patch, MAPREDUCE-3582-part2_rev8-0.23.patch, MAPREDUCE-3582-part2_rev8.patch, MAPREDUCE-3582-prepatch-0.23.patch, MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, MAPREDUCE-3582-whole_rev5.patch, MAPREDUCE-3582-whole_rev6.patch, MAPREDUCE-3582-whole_rev7.patch, MAPREDUCE-3582-whole_rev8.patch, MAPREDUCE-3582.patch, MAPREDUCE-3582_rev2.patch, MAPREDUCE-3597_branch-0.23.patch, MAPREDUCE-778_branch-0.23.patch, MR-3582rev7fixes.patch, mv_script_MR-3582.sh, mv_script_MR-3582_rev2.sh, mv_script_MR-3582_rev3.sh, mv_script_MR-3582_rev4.sh, mv_script_MR-3582_rev5.sh, mv_script_MR-3582_rev6.sh, mv_script_MR-3582_rev7.sh, mv_script_MR-3582_rev8.sh, mv_script_MR-3582_rev8_fixed.sh This ticket will track moving mr1 tests that are passing successfully to mr2 maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-3034:
-
Attachment: MAPREDUCE-3034-1.patch
{quote}
* Major:
** {{NodeManager.stateChanged()}}:
*** In the case of {{STATE.STOPPED}} *both* {{stop()}} and {{reboot()}} need to
be called.
*** I don't see anywhere in the call hierarchy where {{NodeManager.stop()}} is
ever called in the reboot case.
*** Following the logic through, it looks like this is the call sequence from
your patch:
{{NodeStatusUpdaterImpl.startStatusUpdater().run()}}
* calls: {{NodeStatusUpdaterImpl.reboot()}}
* calls: {{AbstractService.stop()}}
* calls: {{AbstractService.changeState()}}
* calls: {{NodeManager.stateChanged()}}
* Which *does not* call {{NodeManager.stop()}} if
{{NodeStatusUpdaterImpl.isRebooted}} is set.
{quote}
stop() was calling from reboot method. Anyway now I have moved it to common for
SHUTDOWN and REBOOT cases.
{quote}
** {{NodeManager}}:
*** In order for the new private static variable {{nodeManager}} to be fully
effective, you should take out the local {{NodeManager}} declaration within
{{main()}}. Otherwise, the new {{getNodeManager()}} method won't always return
the NodeManager instance.
{quote}
It was added only for reboot testability purpose. Now I have refactored the
code.
{quote}
** Please look into the new findbugs and core test failures.
{quote}
These test failures are not related to this patch.
{quote}
* Minor:
** {{NodeManager.main()}} / {{NodeManager.reboot()}}:
*** Rather than copy the same code from {{main()}} into the new {{reboot()}}
method, I would create a common method (call it {{commonNMMain}}, for example).
You will probably need a parameter flag to tell you whether or not this is
called from {{main()}} or from {{reboot()}}.
{quote}
I have refactored the code accordingly.
{quote}
** {{NodeManager.reboot()}}:
*** You should handle the case where {{nodeManagerShutdownHook}} is {{null}}.
Even though it should not be {{null}}, if it is {{null}}, you should create a
new instance of it.
{quote}
There will not be a case where it will be null. This condition is added only
for test purpose.
{quote}
** {{NodeManager.reboot()}}:
*** Rather than calling {{getConfig()}} when calling {{nodeManager.init()}}, I
would argue that it would be better to create a new instance of conf so that
any updates to the NM configs will be picked up. I guess it could be argued the
other way, too. That is, you could make the case that the grid owner doesn't
want the NM configs to change just because the RM was restarted. However, I
think it should be a new NM config because the RM will have a new config.
{quote}
I don't feel it is a good idea to reload the configuration when the NM gets
rebooted. If the user want to update any property then they should restart the
NM manually.
{quote}
** {{NodeStatusUpdaterImpl}}:
*** {{YarnRPC rpc;}} is declared but never used.
{quote}
It was my mistake, I was doing some thing and forgot to remove. I have updated
now.
{quote}
** {{TestNodeStatusUpdater}}:
*** I don't see where it is testing the RM-restart / NM-reboot case.
{quote}
This test case verifies that when NM gets reboot signal, current instance will
be stopped and new instance will be started successfully.
NM should act on a REBOOT command from RM
-
Key: MAPREDUCE-3034
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0, 0.24.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Devaraj K
Priority: Critical
Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034.patch, MR-3034.txt
RM sends a reboot command to NM in some cases, like when it gets lost and
rejoins back. In such a case, NM should act on the command and
reboot/reinitalize itself.
This is akin to TT reinitialize on order from JT. We will need to shutdown
all the services properly and reinitialize - this should automatically take
care of killing of containers, cleaning up local temporary files etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3034: - Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) I updated the patch by incorporating the above comments. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197792#comment-13197792 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Hdfs-trunk #943 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/]) MAPREDUCE-3749. ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238734 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197794#comment-13197794 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Hdfs-trunk #943 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/]) MAPREDUCE-3756. Made single shuffle limit configurable. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238721 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197797#comment-13197797 ] Hudson commented on MAPREDUCE-3754: --- Integrated in Hadoop-Hdfs-trunk #943 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/]) MAPREDUCE-3754. Modified RM UI to filter applications based on state of the applications. (vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238887 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsList.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197799#comment-13197799
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Hdfs-trunk #943 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/943/])
MAPREDUCE-3762. Fixed default CapacityScheduler configs. Contributed by
Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238987
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197803#comment-13197803 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Hdfs-0.23-Build #156 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/]) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238735 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197805#comment-13197805 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Hdfs-0.23-Build #156 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/]) Merge -c 1238721 from trunk to branch-0.23 to fix MAPREDUCE-3756. Made single shuffle limit configurable. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238723 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197810#comment-13197810
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Hdfs-0.23-Build #156 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/])
Merge -c 1238987 from trunk to branch-0.23 to fix MAPREDUCE-3762. Fixed
default CapacityScheduler configs.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238989
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197808#comment-13197808 ] Hudson commented on MAPREDUCE-3754: --- Integrated in Hadoop-Hdfs-0.23-Build #156 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/156/]) MAPREDUCE-3754. Modified RM UI to filter applications based on state of the applications. (vinodkv) svn merge --ignore-ancestry -c 1238887 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=123 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsList.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197839#comment-13197839 ] Hadoop QA commented on MAPREDUCE-3034: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512746/MAPREDUCE-3034-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1735//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1735//console This message is automatically generated. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197844#comment-13197844 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Mapreduce-0.23-Build #178 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/]) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238735 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197846#comment-13197846 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Mapreduce-0.23-Build #178 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/]) Merge -c 1238721 from trunk to branch-0.23 to fix MAPREDUCE-3756. Made single shuffle limit configurable. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238723 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197849#comment-13197849 ] Hudson commented on MAPREDUCE-3754: --- Integrated in Hadoop-Mapreduce-0.23-Build #178 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/]) MAPREDUCE-3754. Modified RM UI to filter applications based on state of the applications. (vinodkv) svn merge --ignore-ancestry -c 1238887 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=123 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsList.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197851#comment-13197851
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Mapreduce-0.23-Build #178 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/])
Merge -c 1238987 from trunk to branch-0.23 to fix MAPREDUCE-3762. Fixed
default CapacityScheduler configs.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238989
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197860#comment-13197860 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Mapreduce-trunk #976 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/]) MAPREDUCE-3756. Made single shuffle limit configurable. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238721 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197858#comment-13197858 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Mapreduce-trunk #976 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/]) MAPREDUCE-3749. ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238734 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3762) Resource Manager fails to come up with default capacity scheduler configs.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197865#comment-13197865
]
Hudson commented on MAPREDUCE-3762:
---
Integrated in Hadoop-Mapreduce-trunk #976 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/])
MAPREDUCE-3762. Fixed default CapacityScheduler configs. Contributed by
Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238987
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
Resource Manager fails to come up with default capacity scheduler configs.
--
Key: MAPREDUCE-3762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3762.patch
Thanks to [~harip] for pointing out the issue. This is the stack trace for
bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01
used in call to setMaxCapacity for queue default
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:210)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.init(LeafQueue.java:184)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197863#comment-13197863 ] Hudson commented on MAPREDUCE-3754: --- Integrated in Hadoop-Mapreduce-trunk #976 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/976/]) MAPREDUCE-3754. Modified RM UI to filter applications based on state of the applications. (vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238887 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/YarnWebParams.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsList.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/NavBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebApp.java RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3499: --- Attachment: MR-3499.patch Sorry for that. Attaching a patch with all files included. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1, 0.24.0 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197916#comment-13197916 ] Alejandro Abdelnur commented on MAPREDUCE-3499: --- +1 New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1, 0.24.0 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3499: -- Resolution: Fixed Fix Version/s: (was: 0.24.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks John Mahadev for the work on this one. Committed to trunk and branch-0.23 New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197960#comment-13197960 ] Hudson commented on MAPREDUCE-3499: --- Integrated in Hadoop-Hdfs-trunk-Commit #1703 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1703/]) MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239207 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197967#comment-13197967 ] Hudson commented on MAPREDUCE-3499: --- Integrated in Hadoop-Common-0.23-Commit #460 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/460/]) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239210 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197971#comment-13197971 ] Hudson commented on MAPREDUCE-3499: --- Integrated in Hadoop-Hdfs-0.23-Commit #450 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/450/]) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239210 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197973#comment-13197973 ] Hudson commented on MAPREDUCE-3499: --- Integrated in Hadoop-Common-trunk-Commit #1632 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1632/]) MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239207 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197986#comment-13197986
]
Hitesh Shah commented on MAPREDUCE-3427:
@Arun - the patch does use @Ignore for the ones that use junit annotations. For
TestStreamingBadRecords that derives off another class ( which in turn extends
TestCase ), I had to comment out the tests themselves.
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1, 0.24.0
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198001#comment-13198001 ] Hudson commented on MAPREDUCE-3499: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1648 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1648/]) MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work. (johnvijoe via tucu) tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239207 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198003#comment-13198003 ] Hudson commented on MAPREDUCE-3499: --- Integrated in Hadoop-Mapreduce-0.23-Commit #474 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/474/]) Merge -r 1239206:1239207 from trunk to branch. FIXES: MAPREDUCE-3499 tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239210 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/MiniMRClientClusterFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/MiniMRYarnCluster.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMiniMRProxyUser.java New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3768) MR-2450 introduced a significant performance regression
[
https://issues.apache.org/jira/browse/MAPREDUCE-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198007#comment-13198007
]
Eli Collins commented on MAPREDUCE-3768:
Here's a jstack during test execution. Possible that it could be a bug in Hive
but more likely an issue of 2450 interacting with the LocalJobRunner.
{noformat}
LocalJobRunner Map Task Executor #0 prio=5 tid=102804800 nid=0x10ef1e000 in
Object.wait() [10ef1d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 7bd6c6890 (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.mapred.Task$TaskReporter.stopCommunicationThread(Task.java:726)
- locked 7bd6c6890 (a java.lang.Object)
at org.apache.hadoop.mapred.Task.done(Task.java:935)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:232)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
{noformat}
MR-2450 introduced a significant performance regression
---
Key: MAPREDUCE-3768
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3768
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Eli Collins
Priority: Blocker
MAPREDUCE-2450 introduced, or at least triggers, a significant performance
regression in Hive. With MR-2450 the execution time of TestCliDriver.skewjoin
goes from 2 minutes to 15 minutes. Reverting this change from the build fixes
the issue.
Here's the relevant query:
{noformat}
FROM src src1 JOIN src src2 ON (src1.key = src2.key)
INSERT OVERWRITE TABLE dest_j1 SELECT src1.key, src2.value;
{noformat}
You can reproduce this by running the following from Hive 8.0 against Hadoop
built from branch-23.
{noformat}
ant very-clean package test -Dtestcase=TestCliDriver -Dqfile=skewjoin.q
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3779) Create hard and soft limits for job counters
Create hard and soft limits for job counters Key: MAPREDUCE-3779 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3779 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker, tasktracker Affects Versions: 1.0.0, 0.23.0 Reporter: Dave Shine Priority: Minor The mapreduce.job.counters.limit is not overridable at the job level. While it is necessary to limit the number of counters to reduce overhead, there are times when exceeding the limit is required. Currently, the only solution is to increase the limit cluster wide. I would like to see a soft limit set in the mapred-site.xml that can be overridden at the job level, in addition to the hard limit that exists today. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3768) MR-2450 introduced a significant performance regression
[
https://issues.apache.org/jira/browse/MAPREDUCE-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198022#comment-13198022
]
Arun C Murthy commented on MAPREDUCE-3768:
--
Thanks, I'll take a look.
MR-2450 introduced a significant performance regression
---
Key: MAPREDUCE-3768
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3768
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Eli Collins
Priority: Blocker
MAPREDUCE-2450 introduced, or at least triggers, a significant performance
regression in Hive. With MR-2450 the execution time of TestCliDriver.skewjoin
goes from 2 minutes to 15 minutes. Reverting this change from the build fixes
the issue.
Here's the relevant query:
{noformat}
FROM src src1 JOIN src src2 ON (src1.key = src2.key)
INSERT OVERWRITE TABLE dest_j1 SELECT src1.key, src2.value;
{noformat}
You can reproduce this by running the following from Hive 8.0 against Hadoop
built from branch-23.
{noformat}
ant very-clean package test -Dtestcase=TestCliDriver -Dqfile=skewjoin.q
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3667) Gridmix jobs are failing with OOM in reduce shuffle phase.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198079#comment-13198079 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3667: bq. Hadoop 0.20.204 reducers does not goes OOM, I have seen this in Hadoop.Next only. That is right, Karam. We would ideally want YARN+MR to be able to run with the same configuration. Unfortunately mrv2 reduce runtime has changed a lot, and it performs better in so many other situations but worse in this particular case. The implementation has changed in such a way that supporting 0.7 is not possible with the current set of jobs in gridmix without running into OOM. I agree that it is a regression but for now this should be fine. We'd ideally want to fix this, will open a separate ticket, but the timelines for that fix are uncertain at this point. Gridmix jobs are failing with OOM in reduce shuffle phase. -- Key: MAPREDUCE-3667 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3667 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Amol Kekre Priority: Blocker Fix For: 0.23.1 Roll up bug for gridmix3 benchmark -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3711) AppMaster recovery for Medium to large jobs take long time
[ https://issues.apache.org/jira/browse/MAPREDUCE-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198083#comment-13198083 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3711: bq. for saving hdfs trips, a simple check of non-zero reduces to skip maps output recovery in RecoveryService should be sufficient. True. Robert, any update on how the patch is coming out? Can help if you need anything. Thanks! AppMaster recovery for Medium to large jobs take long time -- Key: MAPREDUCE-3711 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3711 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Robert Joseph Evans Priority: Blocker Reported by [~karams] yarn.resourcemanager.am.max-retries=2 Ran test cases with sort job on 350 scale having 16800 maps and 680 reduces -: 1. After 70 secs of Job Sumbission Am is killed using kill -9, around 3900 maps were completed and 680 reduces were scheduled, Second AM got restart. Job got completed in 980 secs. AM took very less time to recover. 2. After 150 secs of Job Sumbission AM is killed using kill -9, around 90% maps were completed and 680 reduces were scheduled , Second AM got restart Job got completed in 1000 secs. AM got revocer. 3. After 150 secs of Job Sumbission AM as killed using kill -9, almost all maps were completed and only 680 reduces were running, Recovery was too slow, AM was still revocering after 1hr :40 mis when I killed the run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3780) RM assigns containers to killed applications
RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198092#comment-13198092 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3774: +1. hadoop-yarn-common is the correct destination. I am pushing this in. yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3774: --- Resolution: Fixed Release Note: MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just did an svn move on trunk and branch-0.23. This is done. Thanks Mahadev. yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3780) RM assigns containers to killed applications
[ https://issues.apache.org/jira/browse/MAPREDUCE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned MAPREDUCE-3780: -- Assignee: Hitesh Shah RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Hitesh Shah Fix For: 0.23.1 RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Attachment: mr-3773.patch This is the patch for branch-1. I still need to forward port it to trunk. Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3476) Optimize YARN API calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3476: --- Priority: Major (was: Blocker) We've been running several runs of gridmix (via MAPREDUCE-3719), and we are now using REPLAY mode because stress mode has submission related issues. Given that, this is no longer a blocker, I am downgrading it. Optimize YARN API calls --- Key: MAPREDUCE-3476 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3476 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Vinod Kumar Vavilapalli Several YARN API calls are taking inordinately long. This might be a performance blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3780) RM assigns containers to killed applications
[ https://issues.apache.org/jira/browse/MAPREDUCE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3780: --- Priority: Blocker (was: Major) RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3781) Fix history for apps which were terminated before the AM launch
Fix history for apps which were terminated before the AM launch --- Key: MAPREDUCE-3781 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3781 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Currently the history for applications which were terminated/killed/failed before the AM was launched redirects to a page that does not exist. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198115#comment-13198115 ] Mahadev konar commented on MAPREDUCE-3696: -- Thanks John for testing it out. The patch looks good to me. Ill go ahead and commit it. MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3711) AppMaster recovery for Medium to large jobs take long time
[ https://issues.apache.org/jira/browse/MAPREDUCE-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3711: --- Attachment: MR-3711.txt I have attached the first go at my patch. The unit tests all pass, and I have cleaned up the code a lot. I still want to deploy it to a single node cluster and verify that everything is working just fine, especially with recovery. I am posting it now in hopes of getting some feedback on it while I run the tests. AppMaster recovery for Medium to large jobs take long time -- Key: MAPREDUCE-3711 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3711 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Robert Joseph Evans Priority: Blocker Attachments: MR-3711.txt Reported by [~karams] yarn.resourcemanager.am.max-retries=2 Ran test cases with sort job on 350 scale having 16800 maps and 680 reduces -: 1. After 70 secs of Job Sumbission Am is killed using kill -9, around 3900 maps were completed and 680 reduces were scheduled, Second AM got restart. Job got completed in 980 secs. AM took very less time to recover. 2. After 150 secs of Job Sumbission AM is killed using kill -9, around 90% maps were completed and 680 reduces were scheduled , Second AM got restart Job got completed in 1000 secs. AM got revocer. 3. After 150 secs of Job Sumbission AM as killed using kill -9, almost all maps were completed and only 680 reduces were running, Recovery was too slow, AM was still revocering after 1hr :40 mis when I killed the run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3711) AppMaster recovery for Medium to large jobs take long time
[ https://issues.apache.org/jira/browse/MAPREDUCE-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3711: --- Target Version/s: 0.23.1, 0.24.0 Affects Version/s: 0.24.0 Status: Patch Available (was: Open) AppMaster recovery for Medium to large jobs take long time -- Key: MAPREDUCE-3711 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3711 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Siddharth Seth Assignee: Robert Joseph Evans Priority: Blocker Attachments: MR-3711.txt Reported by [~karams] yarn.resourcemanager.am.max-retries=2 Ran test cases with sort job on 350 scale having 16800 maps and 680 reduces -: 1. After 70 secs of Job Sumbission Am is killed using kill -9, around 3900 maps were completed and 680 reduces were scheduled, Second AM got restart. Job got completed in 980 secs. AM took very less time to recover. 2. After 150 secs of Job Sumbission AM is killed using kill -9, around 90% maps were completed and 680 reduces were scheduled , Second AM got restart Job got completed in 1000 secs. AM got revocer. 3. After 150 secs of Job Sumbission AM as killed using kill -9, almost all maps were completed and only 680 reduces were running, Recovery was too slow, AM was still revocering after 1hr :40 mis when I killed the run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3696: - Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks John! MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Fix For: 0.23.1 Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198125#comment-13198125 ] Hudson commented on MAPREDUCE-3774: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1650 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1650/]) MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239291 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
teragen terasort jobs fail when using webhdfs:// - Key: MAPREDUCE-3782 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta When running a teragen job with a webhdfs:// url the delegation token that is retrieved is an hdfs delegation token. And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198128#comment-13198128 ] Hudson commented on MAPREDUCE-3774: --- Integrated in Hadoop-Mapreduce-0.23-Commit #476 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/476/]) MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. svn merge --ignore-ancestry -c 1239291 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239292 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[ https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3782: - Component/s: mrv2 teragen terasort jobs fail when using webhdfs:// - Key: MAPREDUCE-3782 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta When running a teragen job with a webhdfs:// url the delegation token that is retrieved is an hdfs delegation token. And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198133#comment-13198133 ] Hudson commented on MAPREDUCE-3696: --- Integrated in Hadoop-Common-trunk-Commit #1635 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1635/]) MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239310 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Fix For: 0.23.1 Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198132#comment-13198132 ] Hudson commented on MAPREDUCE-3774: --- Integrated in Hadoop-Common-trunk-Commit #1635 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1635/]) MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239291 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[ https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta updated MAPREDUCE-3782: --- Priority: Critical (was: Major) teragen terasort jobs fail when using webhdfs:// - Key: MAPREDUCE-3782 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Priority: Critical When running a teragen job with a webhdfs:// url the delegation token that is retrieved is an hdfs delegation token. And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
[
https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198136#comment-13198136
]
Arpit Gupta commented on MAPREDUCE-3782:
here are the steps to reproduce the issue (make sure webhdfs is enabled)
1. run teragen
bin/hadoop --config HADOOP_CONF_DIR jar
hadoop-mapreduce-examples-0.23.1-SNAPSHOT.jar teragen 1
webhdfs://NN:50070/user/path
prints the following
{code}
12/02/01 20:03:33 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 158
for hrt_qa on 98.137.234.231:8020
12/02/01 20:03:33 INFO security.TokenCache: Got dt for
hdfs://NN:8020;uri=IP:8020;t.service=IP:8020
{code}
Since a webhdfs url was provided a webhdfs delegation token should have been
used.
2. Run terasort
bin/hadoop --config HADOOP_CONF_DIR jar
hadoop-mapreduce-examples-0.23.1-SNAPSHOT.jar terasort
webhdfs://NN:50070/user/path webhdfs://NN:50070/user/path2
This gets 2 delegation tokens one webhdfs and the other hdfs
{code}
12/02/01 20:03:48 INFO terasort.TeraSort: starting
12/02/01 20:03:49 INFO security.TokenCache: Got dt for
webhdfs://NN:50070;uri=IP:50070;t.service=IP:50070
Spent 65ms computing base-splits.
Spent 2ms computing TeraScheduler splits.
Computing input splits took 67ms
Sampling 2 splits of 2
Making 1 from 1 sampled records
Computing parititions took 1668ms
Spent 1740ms computing partitions.
12/02/01 20:03:51 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 161
for USER on 98.137.234.231:8020
12/02/01 20:03:51 INFO security.TokenCache: Got dt for
hdfs://NN:8020;uri=IP:8020;t.service=IP:8020
{code}
Both the tokens should be webhdfs delegation tokens.
Then we see a java io exception and the job fails. Below is the stack trace
{code}
2/02/01 20:03:54 INFO mapreduce.Job: Job job_1328054538421_0037 failed with
state FAILED due to: Application application_1328054538421_0037 failed 1 times
due to AM Container for appattempt_1328054538421_0037_01 exited with
exitCode: -1000 due to: RemoteTrace:
java.io.IOException: Offset=0 out of the range [0, 0); OPEN,
path=/path/terajobs/output/_partition.lst
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:167)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:267)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$000(WebHdfsFileSystem.java:105)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.checkResponseCode(WebHdfsFileSystem.java:676)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:106)
at
org.apache.hadoop.hdfs.ByteRangeInputStream.read(ByteRangeInputStream.java:130)
at java.io.InputStream.read(InputStream.java:154)
at java.io.DataInputStream.read(DataInputStream.java:83)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:75)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:49)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:109)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:260)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:232)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:183)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1837)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1806)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1782)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:95)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:49)
at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:157)
at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:155)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:153)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Offset=0 out of the range [0, 0); OPEN,
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198140#comment-13198140 ] Hudson commented on MAPREDUCE-3696: --- Integrated in Hadoop-Common-0.23-Commit #463 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/463/]) MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239312 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Fix For: 0.23.1 Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198139#comment-13198139 ] Hudson commented on MAPREDUCE-3774: --- Integrated in Hadoop-Common-0.23-Commit #463 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/463/]) MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. svn merge --ignore-ancestry -c 1239291 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239292 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198145#comment-13198145 ] Hudson commented on MAPREDUCE-3696: --- Integrated in Hadoop-Hdfs-trunk-Commit #1706 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1706/]) MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239310 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Fix For: 0.23.1 Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198144#comment-13198144 ] Hudson commented on MAPREDUCE-3774: --- Integrated in Hadoop-Hdfs-trunk-Commit #1706 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1706/]) MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239291 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198149#comment-13198149 ] Hudson commented on MAPREDUCE-3774: --- Integrated in Hadoop-Hdfs-0.23-Commit #453 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/453/]) MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from hadoop-server-common. svn merge --ignore-ancestry -c 1239291 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239292 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198150#comment-13198150 ] Hudson commented on MAPREDUCE-3696: --- Integrated in Hadoop-Hdfs-0.23-Commit #453 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/453/]) MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239312 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Fix For: 0.23.1 Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3711) AppMaster recovery for Medium to large jobs take long time
[ https://issues.apache.org/jira/browse/MAPREDUCE-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3711: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) I am getting some errors when I try to kill a job that has 100% of the mappers done. I need ot investigate it further to see what is happening. AppMaster recovery for Medium to large jobs take long time -- Key: MAPREDUCE-3711 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3711 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Siddharth Seth Assignee: Robert Joseph Evans Priority: Blocker Attachments: MR-3711.txt Reported by [~karams] yarn.resourcemanager.am.max-retries=2 Ran test cases with sort job on 350 scale having 16800 maps and 680 reduces -: 1. After 70 secs of Job Sumbission Am is killed using kill -9, around 3900 maps were completed and 680 reduces were scheduled, Second AM got restart. Job got completed in 980 secs. AM took very less time to recover. 2. After 150 secs of Job Sumbission AM is killed using kill -9, around 90% maps were completed and 680 reduces were scheduled , Second AM got restart Job got completed in 1000 secs. AM got revocer. 3. After 150 secs of Job Sumbission AM as killed using kill -9, almost all maps were completed and only 680 reduces were running, Recovery was too slow, AM was still revocering after 1hr :40 mis when I killed the run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3783) Fix queue -list to display the correct capacity of queues
Fix queue -list to display the correct capacity of queues --- Key: MAPREDUCE-3783 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3783 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Occasionally, the capacity of the queue as displayed by queue -list has incorrect values. For e.g: yarn.scheduler.capacity.root.queues=a,b yarn.scheduler.capacity.root.b.queues=b1,b2,b3 yarn.scheduler.capacity.root.b.b1.capacity=30 yarn.scheduler.capacity.root.b.b2.capacity=30 yarn.scheduler.capacity.root.b.b3.capacity=40 $ mapred queue -list Queue Name : b Queue State : running Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0 == Queue Name : b1 Queue State : running Scheduling Info : Capacity: *30.02*, MaximumCapacity: 1.0, CurrentCapacity: 0.0 == Queue Name : b2 Queue State : running Scheduling Info : Capacity: *30.02*, MaximumCapacity: 1.0, CurrentCapacity: 0.0 ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198173#comment-13198173 ] Hudson commented on MAPREDUCE-3696: --- Integrated in Hadoop-Mapreduce-0.23-Commit #477 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/477/]) MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) - Merging r1239310 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239312 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Fix For: 0.23.1 Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198172#comment-13198172 ] Hudson commented on MAPREDUCE-3696: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1651 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1651/]) MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. (John George via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239310 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Fix For: 0.23.1 Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198180#comment-13198180 ] Eric Payne commented on MAPREDUCE-3034: --- Hi Devaraj, Thanks for updating the patch. It looks good to me with one exeption. The more I think about it, the more I think we should re-read the namenode configs when the NM restarts. My reason for this opinion is that, in this particular use case, the RM will have a new version of the configs, and you usually want the RMs configs and the NMs configs to match. Other than that, I am happy with the patch. I downloaded it and tested it in both a one-node simple cluster and in a 10-node security cluster. I restarted the RM several times and checked the heap dump to look for memory leaks, and I didn't see any. I also ran about 100 wordcount tests after restarting the RM/NMs. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3725) Hadoop 22 hadoop job -list returns user name as NULL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-3725: - Fix Version/s: 0.22.0 Release Note: Submitting the patch after setting the user name at the client side Status: Patch Available (was: Open) setting the user name at the client side Hadoop 22 hadoop job -list returns user name as NULL Key: MAPREDUCE-3725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3725 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.22.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.22.0 Hadoop 22 hadoop job -list returns user name as NULL -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3725) Hadoop 22 hadoop job -list returns user name as NULL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-3725: - Attachment: patch-MAPREDUCE-3725.patch Hadoop 22 hadoop job -list returns user name as NULL Key: MAPREDUCE-3725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3725 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.22.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.22.0 Attachments: patch-MAPREDUCE-3725.patch Hadoop 22 hadoop job -list returns user name as NULL -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3583) ProcfsBasedProcessTree#constructProcessInfo() may throw NumberFormatException
[
https://issues.apache.org/jira/browse/MAPREDUCE-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198184#comment-13198184
]
Evan Pollan commented on MAPREDUCE-3583:
I yanked the regexp and capture group logic out of the CDH3U3 version of
ProcfsBasedProcessTree, and ran against all pid stat files on a 64 bit Ubuntu
10.04 system in EC2. Sure enough, the namenode's JVM stat file parses out to:
{code}
name: (java)
ppid: 2616
gid:2276
sess: 2276
utime: 32025597350190191
stime: 18446744073709551581
vsize: 1438765056
rss:28505
{code}
The 18446744073709551581 is definitely in the 15th field, documented per the
Ubuntu {{proc}} man page as stime. It just happens to not fit into a signed 64
bit integer...
Here's the actual contents of the stat file:
{code}
2625 (java) S 2616 2276 2276 0 -1 4202496 36120 35150 0 1 32025597350190191
18446744073709551581 55 2 20 0 41 0 4286 1438765056 28567 18446744073709551615
1073741824 1073778416 0 0 0 0 0 1 16800974 18446744073709551615 0 0 17 3 0 0 0
0 0
{code}
ProcfsBasedProcessTree#constructProcessInfo() may throw NumberFormatException
-
Key: MAPREDUCE-3583
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3583
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.20.205.0
Environment: 64-bit Linux:
asf011.sp2.ygridcore.net
Linux asf011.sp2.ygridcore.net 2.6.32-33-server #71-Ubuntu SMP Wed Jul 20
17:42:25 UTC 2011 x86_64 GNU/Linux
Reporter: Zhihong Yu
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Attachments: mapreduce-3583.txt
HBase PreCommit builds frequently gave us NumberFormatException.
From
https://builds.apache.org/job/PreCommit-HBASE-Build/553//testReport/org.apache.hadoop.hbase.mapreduce/TestHFileOutputFormat/testMRIncrementalLoad/:
{code}
2011-12-20 01:44:01,180 WARN [main] mapred.JobClient(784): No job jar file
set. User classes may not be found. See JobConf(Class) or
JobConf#setJar(String).
java.lang.NumberFormatException: For input string: 18446743988060683582
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Long.parseLong(Long.java:422)
at java.lang.Long.parseLong(Long.java:468)
at
org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:413)
at
org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:148)
at
org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
at org.apache.hadoop.mapred.Task.initialize(Task.java:536)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
{code}
From hadoop 0.20.205 source code, looks like ppid was 18446743988060683582,
causing NFE:
{code}
// Set (name) (ppid) (pgrpId) (session) (utime) (stime) (vsize) (rss)
pinfo.updateProcessInfo(m.group(2), Integer.parseInt(m.group(3)),
{code}
You can find information on the OS at the beginning of
https://builds.apache.org/job/PreCommit-HBASE-Build/553/console:
{code}
asf011.sp2.ygridcore.net
Linux asf011.sp2.ygridcore.net 2.6.32-33-server #71-Ubuntu SMP Wed Jul 20
17:42:25 UTC 2011 x86_64 GNU/Linux
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 6
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2048
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
6
Running in Jenkins mode
{code}
From Nicolas Sze:
{noformat}
It looks like that the ppid is a 64-bit positive integer but Java long is
signed and so only works with 63-bit positive integers. In your case,
2^64 18446743988060683582 2^63.
Therefore, there is a NFE.
{noformat}
I propose changing allProcessInfo to MapString, ProcessInfo so that we
don't encounter this problem by avoiding parsing
[jira] [Created] (MAPREDUCE-3784) maxActiveApplications(|PerUser) per queue is too low for small clusters
maxActiveApplications(|PerUser) per queue is too low for small clusters --- Key: MAPREDUCE-3784 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3784 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil We ran into this issue while testing on small clusters. On a 7node cluster with 8G per node, for a queue with absolute capacity 30%, user limit 100%, maxActiveApplications and maxActiveApplicationsPerUser is calculated to be 1. This means that even though the queue has 17GB(0.3*8*7), only 1 user can run 1 app at a given time queuing up rest of the apps/users. This hurts performance on small clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3711) AppMaster recovery for Medium to large jobs take long time
[ https://issues.apache.org/jira/browse/MAPREDUCE-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3711: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) AppMaster recovery for Medium to large jobs take long time -- Key: MAPREDUCE-3711 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3711 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Siddharth Seth Assignee: Robert Joseph Evans Priority: Blocker Attachments: MR-3711.txt, MR-3711.txt Reported by [~karams] yarn.resourcemanager.am.max-retries=2 Ran test cases with sort job on 350 scale having 16800 maps and 680 reduces -: 1. After 70 secs of Job Sumbission Am is killed using kill -9, around 3900 maps were completed and 680 reduces were scheduled, Second AM got restart. Job got completed in 980 secs. AM took very less time to recover. 2. After 150 secs of Job Sumbission AM is killed using kill -9, around 90% maps were completed and 680 reduces were scheduled , Second AM got restart Job got completed in 1000 secs. AM got revocer. 3. After 150 secs of Job Sumbission AM as killed using kill -9, almost all maps were completed and only 680 reduces were running, Recovery was too slow, AM was still revocering after 1hr :40 mis when I killed the run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3711) AppMaster recovery for Medium to large jobs take long time
[ https://issues.apache.org/jira/browse/MAPREDUCE-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3711: --- Attachment: MR-3711.txt OK I figured out the issue. It appears that fs.rename on the local file system will create parent directories for you, where as on HDFS it does not. Also I found out that if there is an error durring recover, it can cause the AM to fail, which does not result in another retry. I will try to reproduce the issue again, and file a JIRA for it. AppMaster recovery for Medium to large jobs take long time -- Key: MAPREDUCE-3711 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3711 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Siddharth Seth Assignee: Robert Joseph Evans Priority: Blocker Attachments: MR-3711.txt, MR-3711.txt Reported by [~karams] yarn.resourcemanager.am.max-retries=2 Ran test cases with sort job on 350 scale having 16800 maps and 680 reduces -: 1. After 70 secs of Job Sumbission Am is killed using kill -9, around 3900 maps were completed and 680 reduces were scheduled, Second AM got restart. Job got completed in 980 secs. AM took very less time to recover. 2. After 150 secs of Job Sumbission AM is killed using kill -9, around 90% maps were completed and 680 reduces were scheduled , Second AM got restart Job got completed in 1000 secs. AM got revocer. 3. After 150 secs of Job Sumbission AM as killed using kill -9, almost all maps were completed and only 680 reduces were running, Recovery was too slow, AM was still revocering after 1hr :40 mis when I killed the run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3780) RM assigns containers to killed applications
[ https://issues.apache.org/jira/browse/MAPREDUCE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3780: --- Status: Patch Available (was: Open) RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3780.1.patch RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3780) RM assigns containers to killed applications
[ https://issues.apache.org/jira/browse/MAPREDUCE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3780: --- Attachment: MR-3780.1.patch RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3780.1.patch RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3725) Hadoop 22 hadoop job -list returns user name as NULL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198290#comment-13198290 ] Hadoop QA commented on MAPREDUCE-3725: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512810/patch-MAPREDUCE-3725.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1739//console This message is automatically generated. Hadoop 22 hadoop job -list returns user name as NULL Key: MAPREDUCE-3725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3725 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.22.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: 0.22.0 Attachments: patch-MAPREDUCE-3725.patch Hadoop 22 hadoop job -list returns user name as NULL -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198294#comment-13198294
]
Arun C Murthy commented on MAPREDUCE-3427:
--
Got it, +1.
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-3427.
--
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
I just committed this. Thanks Hitesh!
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3780) RM assigns containers to killed applications
[ https://issues.apache.org/jira/browse/MAPREDUCE-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198301#comment-13198301 ] Arun C Murthy commented on MAPREDUCE-3780: -- Nice catch Hitesh. +1 RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Hitesh Shah Priority: Blocker Fix For: 0.23.1 Attachments: MR-3780.1.patch RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198302#comment-13198302
]
Hudson commented on MAPREDUCE-3427:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1707 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1707/])
MAPREDUCE-3427. Fix streaming unit tests broken after mavenization.
Contributed by Hitesh Shah.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239391
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-tools/hadoop-streaming/pom.xml
*
/hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingBadRecords.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestUlimit.java
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198313#comment-13198313 ] Siddharth Seth commented on MAPREDUCE-3640: --- +1. Looks good. Will commit the patch shortly. AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch, MAPREDUCE-3640.patch, MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198315#comment-13198315
]
Hudson commented on MAPREDUCE-3427:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #478 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/478/])
Merge -c 1239391 from trunk to branch-0.23 to fix MAPREDUCE-3427. Fix
streaming unit tests broken after mavenization.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239392
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/pom.xml
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingBadRecords.java
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestUlimit.java
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3725) Hadoop 22 hadoop job -list returns user name as NULL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-3725: --- Fix Version/s: (was: 0.22.0) Target Version/s: 0.22.1 Status: Open (was: Patch Available) Mayank, could you please update the jira on the running of the test target. +1 The patch looks good Hadoop 22 hadoop job -list returns user name as NULL Key: MAPREDUCE-3725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3725 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.22.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: patch-MAPREDUCE-3725.patch Hadoop 22 hadoop job -list returns user name as NULL -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3725) Hadoop 22 hadoop job -list returns user name as NULL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198321#comment-13198321 ] Mayank Bansal commented on MAPREDUCE-3725: -- Tests ran fine on 22 Hadoop 22 hadoop job -list returns user name as NULL Key: MAPREDUCE-3725 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3725 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 0.22.1 Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: patch-MAPREDUCE-3725.patch Hadoop 22 hadoop job -list returns user name as NULL -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3640: -- Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-0.23. Thanks Arun. AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3640.patch, MAPREDUCE-3640.patch, MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198323#comment-13198323
]
Hudson commented on MAPREDUCE-3427:
---
Integrated in Hadoop-Common-0.23-Commit #464 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/464/])
Merge -c 1239391 from trunk to branch-0.23 to fix MAPREDUCE-3427. Fix
streaming unit tests broken after mavenization.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239392
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/pom.xml
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingBadRecords.java
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestUlimit.java
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13198328#comment-13198328
]
Hudson commented on MAPREDUCE-3427:
---
Integrated in Hadoop-Hdfs-0.23-Commit #454 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/454/])
Merge -c 1239391 from trunk to branch-0.23 to fix MAPREDUCE-3427. Fix
streaming unit tests broken after mavenization.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1239392
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/pom.xml
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamingBadRecords.java
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestUlimit.java
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
