[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196771#comment-13196771
]
Hadoop QA commented on MAPREDUCE-3764:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512540/MAPREDUCE-3764.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1721//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1721//console
This message is automatically generated.
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3756: --- Attachment: MR-3756.1.patch Trivial patch. Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3756: --- Status: Patch Available (was: Open) Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3765) FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3765: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Open (was: Patch Available) FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting Key: MAPREDUCE-3765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3765 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Attachments: MR-3765.1.patch FifoScheduler uses default min 1 GB regardless of the configuration value set for minimum memory allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3771) Port MAPREDUCE-1735 to trunk/0.23
Port MAPREDUCE-1735 to trunk/0.23 - Key: MAPREDUCE-3771 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3771 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3771.patch Per discussion in general@, we should port MAPREDUCE-1735 to 0.23 trunk to 'undeprecate' old mapred api: http://s.apache.org/undeprecate-mapred-apis -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3771) Port MAPREDUCE-1735 to trunk/0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3771: - Status: Patch Available (was: Open) Port MAPREDUCE-1735 to trunk/0.23 - Key: MAPREDUCE-3771 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3771 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3771.patch Per discussion in general@, we should port MAPREDUCE-1735 to 0.23 trunk to 'undeprecate' old mapred api: http://s.apache.org/undeprecate-mapred-apis -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3771) Port MAPREDUCE-1735 to trunk/0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3771: - Attachment: MAPREDUCE-3771.patch Patch to undeprecate mapred apis. Port MAPREDUCE-1735 to trunk/0.23 - Key: MAPREDUCE-3771 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3771 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3771.patch Per discussion in general@, we should port MAPREDUCE-1735 to 0.23 trunk to 'undeprecate' old mapred api: http://s.apache.org/undeprecate-mapred-apis -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3764:
-
Resolution: Fixed
Fix Version/s: 0.23.1
Status: Resolved (was: Patch Available)
Thanks for the review Mahadev. I just committed this.
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3640) AMRecovery should pick completed task form partial JobHistory files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196788#comment-13196788 ] Arun C Murthy commented on MAPREDUCE-3640: -- Thanks for the review Mahadev. The api is needed for a test in a different package, hence it's public (but I've marked it @Private). AMRecovery should pick completed task form partial JobHistory files --- Key: MAPREDUCE-3640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3640 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Siddharth Seth Assignee: Arun C Murthy Priority: Blocker Attachments: MAPREDUCE-3640.patch Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196790#comment-13196790
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Common-trunk-Commit #1622 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1622/])
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238255
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196791#comment-13196791
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1694 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1694/])
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238255
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196793#comment-13196793
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Common-0.23-Commit #451 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/451/])
Merge -c 1238255 from trunk to branch-0.23 to fix MAPREDUCE-3764. Fixed
resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238256
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196795#comment-13196795
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Hdfs-0.23-Commit #442 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/442/])
Merge -c 1238255 from trunk to branch-0.23 to fix MAPREDUCE-3764. Fixed
resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238256
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196806#comment-13196806
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1639 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1639/])
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238255
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196810#comment-13196810 ] Hadoop QA commented on MAPREDUCE-3756: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512542/MR-3756.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1723//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1723//console This message is automatically generated. Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196812#comment-13196812
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #466 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/466/])
Merge -c 1238255 from trunk to branch-0.23 to fix MAPREDUCE-3764. Fixed
resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238256
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3597) Provide a way to access other info of history file from Rumentool
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3597: Attachment: 3597.branch-1.v2.2.patch Attaching new patch by changing testcase a little bit. Provide a way to access other info of history file from Rumentool - Key: MAPREDUCE-3597 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 0.24.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.24.0 Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access other info available in history file which is not dumped to trace file. This is useful for components which want to parse history files and get info. These components can directly use/leverage Rumen's parsing of history files across hadoop releases and get history info in a consistent way for further analysis/processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other info of history file from Rumentool
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196816#comment-13196816 ] Amar Kamat commented on MAPREDUCE-3597: --- +1. Looks good to me. Provide a way to access other info of history file from Rumentool - Key: MAPREDUCE-3597 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 0.24.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.24.0 Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access other info available in history file which is not dumped to trace file. This is useful for components which want to parse history files and get info. These components can directly use/leverage Rumen's parsing of history files across hadoop releases and get history info in a consistent way for further analysis/processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other info of history file from Rumentool
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196817#comment-13196817 ] Ravi Gummadi commented on MAPREDUCE-3597: - Unit tests and test-patch passed for the patch for branch-1. Provide a way to access other info of history file from Rumentool - Key: MAPREDUCE-3597 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 0.24.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.24.0 Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access other info available in history file which is not dumped to trace file. This is useful for components which want to parse history files and get info. These components can directly use/leverage Rumen's parsing of history files across hadoop releases and get history info in a consistent way for further analysis/processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3771) Port MAPREDUCE-1735 to trunk/0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196820#comment-13196820 ] Hadoop QA commented on MAPREDUCE-3771: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512543/MAPREDUCE-3771.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1724//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1724//console This message is automatically generated. Port MAPREDUCE-1735 to trunk/0.23 - Key: MAPREDUCE-3771 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3771 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3771.patch Per discussion in general@, we should port MAPREDUCE-1735 to 0.23 trunk to 'undeprecate' old mapred api: http://s.apache.org/undeprecate-mapred-apis -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3597) Provide a way to access other info of history file from Rumentool
[ https://issues.apache.org/jira/browse/MAPREDUCE-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196831#comment-13196831 ] Ravi Gummadi commented on MAPREDUCE-3597: - I just committed this to branch-1. Provide a way to access other info of history file from Rumentool - Key: MAPREDUCE-3597 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3597 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 0.24.0 Reporter: Ravi Gummadi Assignee: Ravi Gummadi Fix For: 0.24.0 Attachments: 3597.branch-0.23.patch, 3597.branch-1.v1.patch, 3597.branch-1.v2.1.patch, 3597.branch-1.v2.2.patch, 3597.branch-1.v2.patch, 3597.v0.patch, 3597.v1.patch, MAPREDUCE-3597_branch-0.23.patch As the trace file generated by Rumen TraceBuilder is skipping some of the info like job counters, task counters, etc. we need a way to access other info available in history file which is not dumped to trace file. This is useful for components which want to parse history files and get info. These components can directly use/leverage Rumen's parsing of history files across hadoop releases and get history info in a consistent way for further analysis/processing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196868#comment-13196868
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Hdfs-trunk #942 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/942/])
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238255
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3748) Move CS related nodeUpdate log messages to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196866#comment-13196866 ] Hudson commented on MAPREDUCE-3748: --- Integrated in Hadoop-Hdfs-trunk #942 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/942/]) MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug. Contributed by Ramya Sunil. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238157 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java Move CS related nodeUpdate log messages to DEBUG Key: MAPREDUCE-3748 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3748 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Ramya Sunil Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3748.patch, MAPREDUCE-3748.patch Currently, the RM has nodeUpdate logs per NM per second such as the following: 2012-01-27 21:51:32,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 2012-01-27 21:51:32,510 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager2:port2 clusterResources: memory: 57344 2012-01-27 21:51:33,094 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 Debugging is difficult with huge amount of logs such as this. These logs need to be moved to DEBUG. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3679) AM logs and others should not automatically refresh after every 1 second.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196867#comment-13196867 ] Hudson commented on MAPREDUCE-3679: --- Integrated in Hadoop-Hdfs-trunk #942 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/942/]) MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1 second. (Vinod KV via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238146 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/CountersPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ApplicationPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java AM logs and others should not automatically refresh after every 1 second. - Key: MAPREDUCE-3679 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3679 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3679-20120129.txt If you are looking through the logs for AM or containers, the page is automatically refreshed after 1 second or so which makes it problematic to search through the page or debug using the content on the page. We should not refresh the logs page. There should be a button to manually refresh if the user needs to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide node lists in JMX output
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196869#comment-13196869 ] Hudson commented on MAPREDUCE-3703: --- Integrated in Hadoop-Hdfs-trunk #942 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/942/]) MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1237920 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java ResourceManager should provide node lists in JMX output --- Key: MAPREDUCE-3703 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.23.1 Reporter: Eric Payne Assignee: Eric Payne Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3703.1.txt In 0.20.*, the JMX UI for the JobTracker (http://JobTrackerHost:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section. In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes. At least the list of active nodes is needed in JSON format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile fails in map/reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196870#comment-13196870 ] Hudson commented on MAPREDUCE-3716: --- Integrated in Hadoop-Hdfs-trunk #942 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/942/]) MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238136 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/mapred-default.xml java.io.File.createTempFile fails in map/reduce tasks - Key: MAPREDUCE-3716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3772) MultipleOutputs output lost if baseOutputPath starts with ../
MultipleOutputs output lost if baseOutputPath starts with ../ - Key: MAPREDUCE-3772 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3772 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.22.0 Environment: FreeBSD Reporter: Radim Kolar Priority: Minor Lets say you have output directory set: FileOutputFormat.setOutputPath(job, /tmp/multi1/out); and want to place output from MultipleOutputs into /tmp/multi1/extra I expect following code to work: mos = new MultipleOutputsText, IntWritable(context); mos.write(new Text(zrr), value, ../extra/); but no Exception is throw and expected output directory /tmp/multi1/extra does not even exists. All data written to this output vanish without trace. To make it work fullpath must be used mos.write(new Text(zrr), value, /tmp/multi1/extra/); Output is listed in statistics from MultipleOutputs correctly: org.apache.hadoop.mapreduce.lib.output.MultipleOutputs ../gaja1/=1 (* everything is lost *) /tmp/multi1/out/../ksd34/=1 (* this using full path works *) list1=6667 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide node lists in JMX output
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196881#comment-13196881 ] Hudson commented on MAPREDUCE-3703: --- Integrated in Hadoop-Hdfs-0.23-Build #155 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/155/]) MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) - Merge r1237920 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1237921 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java ResourceManager should provide node lists in JMX output --- Key: MAPREDUCE-3703 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.23.1 Reporter: Eric Payne Assignee: Eric Payne Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3703.1.txt In 0.20.*, the JMX UI for the JobTracker (http://JobTrackerHost:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section. In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes. At least the list of active nodes is needed in JSON format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196880#comment-13196880
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Hdfs-0.23-Build #155 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/155/])
Merge -c 1238255 from trunk to branch-0.23 to fix MAPREDUCE-3764. Fixed
resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238256
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3748) Move CS related nodeUpdate log messages to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196878#comment-13196878 ] Hudson commented on MAPREDUCE-3748: --- Integrated in Hadoop-Hdfs-0.23-Build #155 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/155/]) Merge -c 1238157 from trunk to branch-0.23 to fix MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238159 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java Move CS related nodeUpdate log messages to DEBUG Key: MAPREDUCE-3748 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3748 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Ramya Sunil Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3748.patch, MAPREDUCE-3748.patch Currently, the RM has nodeUpdate logs per NM per second such as the following: 2012-01-27 21:51:32,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 2012-01-27 21:51:32,510 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager2:port2 clusterResources: memory: 57344 2012-01-27 21:51:33,094 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 Debugging is difficult with huge amount of logs such as this. These logs need to be moved to DEBUG. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3679) AM logs and others should not automatically refresh after every 1 second.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196879#comment-13196879 ] Hudson commented on MAPREDUCE-3679: --- Integrated in Hadoop-Hdfs-0.23-Build #155 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/155/]) MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1 second. (Vinod KV via mahadev) - Merging r1238146 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238148 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/CountersPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ApplicationPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java AM logs and others should not automatically refresh after every 1 second. - Key: MAPREDUCE-3679 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3679 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3679-20120129.txt If you are looking through the logs for AM or containers, the page is automatically refreshed after 1 second or so which makes it problematic to search through the page or debug using the content on the page. We should not refresh the logs page. There should be a button to manually refresh if the user needs to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile fails in map/reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196882#comment-13196882 ] Hudson commented on MAPREDUCE-3716: --- Integrated in Hadoop-Hdfs-0.23-Build #155 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/155/]) MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles. svn merge --ignore-ancestry -c 1238136 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238138 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/mapred-default.xml java.io.File.createTempFile fails in map/reduce tasks - Key: MAPREDUCE-3716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196926#comment-13196926
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Mapreduce-0.23-Build #177 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/177/])
Merge -c 1238255 from trunk to branch-0.23 to fix MAPREDUCE-3764. Fixed
resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238256
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide node lists in JMX output
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196927#comment-13196927 ] Hudson commented on MAPREDUCE-3703: --- Integrated in Hadoop-Mapreduce-0.23-Build #177 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/177/]) MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) - Merge r1237920 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1237921 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java ResourceManager should provide node lists in JMX output --- Key: MAPREDUCE-3703 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.23.1 Reporter: Eric Payne Assignee: Eric Payne Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3703.1.txt In 0.20.*, the JMX UI for the JobTracker (http://JobTrackerHost:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section. In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes. At least the list of active nodes is needed in JSON format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3748) Move CS related nodeUpdate log messages to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196924#comment-13196924 ] Hudson commented on MAPREDUCE-3748: --- Integrated in Hadoop-Mapreduce-0.23-Build #177 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/177/]) Merge -c 1238157 from trunk to branch-0.23 to fix MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238159 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java Move CS related nodeUpdate log messages to DEBUG Key: MAPREDUCE-3748 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3748 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Ramya Sunil Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3748.patch, MAPREDUCE-3748.patch Currently, the RM has nodeUpdate logs per NM per second such as the following: 2012-01-27 21:51:32,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 2012-01-27 21:51:32,510 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager2:port2 clusterResources: memory: 57344 2012-01-27 21:51:33,094 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 Debugging is difficult with huge amount of logs such as this. These logs need to be moved to DEBUG. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3748) Move CS related nodeUpdate log messages to DEBUG
[ https://issues.apache.org/jira/browse/MAPREDUCE-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196931#comment-13196931 ] Hudson commented on MAPREDUCE-3748: --- Integrated in Hadoop-Mapreduce-trunk #975 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/975/]) MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug. Contributed by Ramya Sunil. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238157 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java Move CS related nodeUpdate log messages to DEBUG Key: MAPREDUCE-3748 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3748 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Ramya Sunil Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3748.patch, MAPREDUCE-3748.patch Currently, the RM has nodeUpdate logs per NM per second such as the following: 2012-01-27 21:51:32,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 2012-01-27 21:51:32,510 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager2:port2 clusterResources: memory: 57344 2012-01-27 21:51:33,094 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: nodemanager1:port1 clusterResources: memory: 57344 Debugging is difficult with huge amount of logs such as this. These logs need to be moved to DEBUG. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3703) ResourceManager should provide node lists in JMX output
[ https://issues.apache.org/jira/browse/MAPREDUCE-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196934#comment-13196934 ] Hudson commented on MAPREDUCE-3703: --- Integrated in Hadoop-Mapreduce-trunk #975 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/975/]) MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. (Eric Payne via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1237920 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestRMNMInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMNMInfoBeans.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java ResourceManager should provide node lists in JMX output --- Key: MAPREDUCE-3703 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3703 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, resourcemanager Affects Versions: 0.23.1 Reporter: Eric Payne Assignee: Eric Payne Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3703.1.txt In 0.20.*, the JMX UI for the JobTracker (http://JobTrackerHost:50030/jmx) showed lists of Live and BlackListed Nodes under the JobTrackerInfo section. In 0.23, the ResourceManager JMX UI shows the number of active, decommissioned, lost, unhealthy, and rebooted nodes under the ClusterMetrics section, but does not give the list of nodes. At least the list of active nodes is needed in JSON format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3764) AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
[
https://issues.apache.org/jira/browse/MAPREDUCE-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196933#comment-13196933
]
Hudson commented on MAPREDUCE-3764:
---
Integrated in Hadoop-Mapreduce-trunk #975 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/975/])
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238255
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
--
Key: MAPREDUCE-3764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3764
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Arun C Murthy
Priority: Critical
Fix For: 0.23.1
Attachments: MAPREDUCE-3764.patch, MAPREDUCE-3764.patch
MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB *
containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each
increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68%
utilized, depending on high ram requests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile fails in map/reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196928#comment-13196928 ] Hudson commented on MAPREDUCE-3716: --- Integrated in Hadoop-Mapreduce-0.23-Build #177 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/177/]) MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles. svn merge --ignore-ancestry -c 1238136 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238138 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java/mapred-default.xml java.io.File.createTempFile fails in map/reduce tasks - Key: MAPREDUCE-3716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3679) AM logs and others should not automatically refresh after every 1 second.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196932#comment-13196932 ] Hudson commented on MAPREDUCE-3679: --- Integrated in Hadoop-Mapreduce-trunk #975 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/975/]) MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1 second. (Vinod KV via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238146 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/CountersPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ApplicationPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java AM logs and others should not automatically refresh after every 1 second. - Key: MAPREDUCE-3679 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3679 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3679-20120129.txt If you are looking through the logs for AM or containers, the page is automatically refreshed after 1 second or so which makes it problematic to search through the page or debug using the content on the page. We should not refresh the logs page. There should be a button to manually refresh if the user needs to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3716) java.io.File.createTempFile fails in map/reduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196935#comment-13196935 ] Hudson commented on MAPREDUCE-3716: --- Integrated in Hadoop-Mapreduce-trunk #975 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/975/]) MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks. Contributed by Jonathan Eagles. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238136 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/MapReduceChildJVM.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/mapred-default.xml java.io.File.createTempFile fails in map/reduce tasks - Key: MAPREDUCE-3716 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3716 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3716.patch, MAPREDUCE-3716.patch, TestRandomWriter.patch container-launch.sh specifies java option java.io.tmpdir when executing the java process for the child container but fails to create the tmpdir. All uses of createTempFile and other commands relying on java.io.tmpdir will fail when called from child container jvms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3679) AM logs and others should not automatically refresh after every 1 second.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196925#comment-13196925 ] Hudson commented on MAPREDUCE-3679: --- Integrated in Hadoop-Mapreduce-0.23-Build #177 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/177/]) MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1 second. (Vinod KV via mahadev) - Merging r1238146 from trunk. mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238148 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/CountersPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/JobPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ApplicationPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NodePage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java AM logs and others should not automatically refresh after every 1 second. - Key: MAPREDUCE-3679 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3679 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3679-20120129.txt If you are looking through the logs for AM or containers, the page is automatically refreshed after 1 second or so which makes it problematic to search through the page or debug using the content on the page. We should not refresh the logs page. There should be a button to manually refresh if the user needs to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3760) Blacklisted NMs should not appear in Active nodes list
[ https://issues.apache.org/jira/browse/MAPREDUCE-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196959#comment-13196959 ] Robert Joseph Evans commented on MAPREDUCE-3760: It does conflict with it a bit. Also the Web Service API may also need to be updated so that it outputs data in the same way. Blacklisted NMs should not appear in Active nodes list -- Key: MAPREDUCE-3760 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3760 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Vinod Kumar Vavilapalli Priority: Minor Fix For: 0.23.1 Attachments: MAPREDUCE-3760-20120130.txt Blacklisted NMs appear in both Active Nodes and Unhealthy nodes on the RM UI. This should be fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[
https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197021#comment-13197021
]
Alejandro Abdelnur commented on MAPREDUCE-3499:
---
Mahadev,
I was missing something, now I got things working. I think we are good with
this patch for now.
I'm able to successfully submit job to MiniCluster from Oozie, I'm now hitting
the Counter's API change, MAPREDUCE-3697
{code}
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.Counters.getGroup(Ljava/lang/String;)Lorg/apache/hadoop/mapred/Counters$Group;
at
org.apache.oozie.action.hadoop.LauncherMapper.hasIdSwap(LauncherMapper.java:273)
at
org.apache.oozie.action.hadoop.TestMapReduceActionExecutor.testSetExecutionStats_when_user_has_specified_stats_write_FALSE(TestMapReduceActionExecutor.java:562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
com.intellij.junit3.JUnit3IdeaTestRunner.doRun(JUnit3IdeaTestRunner.java:139)
at
com.intellij.junit3.JUnit3IdeaTestRunner.startRunnerWithArgs(JUnit3IdeaTestRunner.java:52)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:199)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
{code}
Thanks.
Alejandro
New MiniMR does not setup proxyuser configuration correctly, thus tests using
doAs do not work
--
Key: MAPREDUCE-3499
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, test
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: John George
Priority: Blocker
Labels: bigtop
Fix For: 0.23.1, 0.24.0
Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch,
TestForMAPREDUCE-3499.patch
The new MiniMR implementation is not taking proxyuser settings.
Because of this, testcases using/testing doAs functionality fail.
This affects all Oozie testcases that use MiniMR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3499: --- Attachment: MR-3499.patch New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1, 0.24.0 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3499: --- Status: Patch Available (was: Open) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1, 0.24.0 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197060#comment-13197060 ] Mahadev konar commented on MAPREDUCE-3499: -- Thanks Alejandro. I am working on a patch for MAPREDUCE-3697. Will update soon. Can you please verify John's latest patch? New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1, 0.24.0 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3756: - Resolution: Fixed Fix Version/s: 0.23.1 Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2980) Fetch failures and other related issues in Jetty 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197079#comment-13197079 ] Kihwal Lee commented on MAPREDUCE-2980: --- Todd, any update on the use of jetty 6.1.27? It seems far less riskier than moving to jetty 7. We also have users complaining about this. The new health_check helps a bit but is not ideal. Fetch failures and other related issues in Jetty 6.1.26 --- Key: MAPREDUCE-2980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2980 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Todd Lipcon Priority: Critical Since upgrading Jetty from 6.1.14 to 6.1.26 we've had a ton of HTTP-related issues, including: - Much higher incidence of fetch failures - A few strange file-descriptor related bugs (eg MAPREDUCE-2389) - A few unexplained issues where long fscks on the NameNode drop out halfway through with a ClosedChannelException Stress tests with 1Map x 1Reduce sleep jobs reliably reproduce fetch failures at a rate of about 1 per million on a 25 node test cluster. These problems are all new since the upgrade from 6.1.14. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197084#comment-13197084 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Common-trunk-Commit #1624 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1624/]) MAPREDUCE-3756. Made single shuffle limit configurable. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238721 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3765) FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3765: --- Attachment: MR-3765.2.patch FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting Key: MAPREDUCE-3765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3765 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Attachments: MR-3765.1.patch, MR-3765.2.patch FifoScheduler uses default min 1 GB regardless of the configuration value set for minimum memory allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3765) FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3765: --- Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1) Status: Patch Available (was: Open) FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting Key: MAPREDUCE-3765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3765 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Attachments: MR-3765.1.patch, MR-3765.2.patch FifoScheduler uses default min 1 GB regardless of the configuration value set for minimum memory allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197088#comment-13197088 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Common-0.23-Commit #453 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/453/]) Merge -c 1238721 from trunk to branch-0.23 to fix MAPREDUCE-3756. Made single shuffle limit configurable. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238723 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3749: - Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. We can revisit the more concurrent approach (e.g. with ConcurrentSkipListMap) in the future. ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197095#comment-13197095 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Hdfs-trunk-Commit #1696 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1696/]) MAPREDUCE-3756. Made single shuffle limit configurable. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238721 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197094#comment-13197094 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Hdfs-trunk-Commit #1696 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1696/]) MAPREDUCE-3749. ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238734 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197100#comment-13197100 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Hdfs-0.23-Commit #444 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/444/]) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238735 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197101#comment-13197101 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Hdfs-0.23-Commit #444 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/444/]) Merge -c 1238721 from trunk to branch-0.23 to fix MAPREDUCE-3756. Made single shuffle limit configurable. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238723 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2980) Fetch failures and other related issues in Jetty 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197107#comment-13197107 ] Todd Lipcon commented on MAPREDUCE-2980: Hey Kihwal. 6.1.27 still hasn't been released. We've been shipping a patched version of 6.1.26 with some fixes provided by Greg Wilkins - the tag is here: https://github.com/toddlipcon/jetty-hadoop-fix/tree/6.1.26.cloudera.1 The problems aren't 100% gone with this build but they seem to be improved -- at least nothing's been escalated to me in a few months, so I'm assuming it's a good sign! The other patch we've recently added is MAPREDUCE-3184, which is similar to the health check script approach - it just suicides the TT if it detects the problem. Fetch failures and other related issues in Jetty 6.1.26 --- Key: MAPREDUCE-2980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2980 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Todd Lipcon Priority: Critical Since upgrading Jetty from 6.1.14 to 6.1.26 we've had a ton of HTTP-related issues, including: - Much higher incidence of fetch failures - A few strange file-descriptor related bugs (eg MAPREDUCE-2389) - A few unexplained issues where long fscks on the NameNode drop out halfway through with a ClosedChannelException Stress tests with 1Map x 1Reduce sleep jobs reliably reproduce fetch failures at a rate of about 1 per million on a 25 node test cluster. These problems are all new since the upgrade from 6.1.14. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197110#comment-13197110 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Common-0.23-Commit #454 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/454/]) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238735 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197112#comment-13197112 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Common-trunk-Commit #1625 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1625/]) MAPREDUCE-3749. ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238734 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3766) dumptb fails with filenotfound
[ https://issues.apache.org/jira/browse/MAPREDUCE-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antonio Piccolboni resolved MAPREDUCE-3766. --- Resolution: Not A Problem Seems to be reading each part just fine. Maybe it should get a more getmerge like functionality, maybe it should be documented, but most likely this is not a bug. dumptb fails with filenotfound -- Key: MAPREDUCE-3766 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3766 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming Affects Versions: 0.20.2 Environment: AWS, ubuntu, cdh3u3 Reporter: Antonio Piccolboni dumptb can't find an existing file antonio@ip-10-9-65-123:~$ hadoop jar /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar dumptb /tmp/Rtmp5h8lIO/file179e4fc 12/01/31 00:27:48 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing. 12/01/31 00:27:49 WARN snappy.LoadSnappy: Snappy native library is available 12/01/31 00:27:49 INFO util.NativeCodeLoader: Loaded the native-hadoop library 12/01/31 00:27:49 INFO snappy.LoadSnappy: Snappy native library loaded Exception in thread main java.io.FileNotFoundException: File does not exist: /tmp/Rtmp5h8lIO/file179e4fc/_logs at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456) at org.apache.hadoop.streaming.AutoInputFormat.getRecordReader(AutoInputFormat.java:56) at org.apache.hadoop.streaming.DumpTypedBytes.dumpTypedBytes(DumpTypedBytes.java:102) at org.apache.hadoop.streaming.DumpTypedBytes.run(DumpTypedBytes.java:83) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:41) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) But the file is there antonio@ip-10-9-65-123:~$ hadoop dfs -ls /tmp/Rtmp5h8lIO/file179e4fc/_logs 12/01/31 00:28:47 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing. Found 1 items drwxrwxrwx - antonio supergroup 0 2012-01-31 00:18 /tmp/Rtmp5h8lIO/file179e4fc/_logs/history -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197133#comment-13197133 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1641 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1641/]) MAPREDUCE-3756. Made single shuffle limit configurable. Contributed by Hitesh Shah. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238721 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3759) ClassCastException thrown in -list-active-trackers when there are a few unhealthy nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned MAPREDUCE-3759: -- Assignee: Vinod Kumar Vavilapalli ClassCastException thrown in -list-active-trackers when there are a few unhealthy nodes --- Key: MAPREDUCE-3759 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3759 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 When there are a few blacklisted nodes in the cluster, bin/mapred job -list-active-trackers throws java.lang.ClassCastException: org.apache.hadoop.yarn.server.resourcemanager.resource.Resources$1 cannot be cast to org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197138#comment-13197138 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Mapreduce-0.23-Commit #468 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/468/]) Merge -r 1238733:1238734 from trunk to branch-0.23. Fixes: MAPREDUCE-3749 tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238735 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3756) Make single shuffle limit configurable
[ https://issues.apache.org/jira/browse/MAPREDUCE-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197139#comment-13197139 ] Hudson commented on MAPREDUCE-3756: --- Integrated in Hadoop-Mapreduce-0.23-Commit #468 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/468/]) Merge -c 1238721 from trunk to branch-0.23 to fix MAPREDUCE-3756. Made single shuffle limit configurable. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238723 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestReduceFetchFromPartialMem.java Make single shuffle limit configurable -- Key: MAPREDUCE-3756 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3756 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3756.1.patch Make single shuffle limit configurable, currently it's hard-coded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3749) ConcurrentModificationException in counter groups
[ https://issues.apache.org/jira/browse/MAPREDUCE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197145#comment-13197145 ] Hudson commented on MAPREDUCE-3749: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1642 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1642/]) MAPREDUCE-3749. ConcurrentModificationException in counter groups. tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1238734 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounterGroup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/AbstractCounters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java ConcurrentModificationException in counter groups - Key: MAPREDUCE-3749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3749 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3749-concurrent.patch, MAPREDUCE-3749-test.patch, MAPREDUCE-3749.patch, MAPREDUCE-3749.patch Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException. This was found while running Hive unit tests against a recent 0.23 version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197150#comment-13197150 ] Hadoop QA commented on MAPREDUCE-3499: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512598/MR-3499.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 3721 javac compiler warnings (more than the trunk's current 3706 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1725//console This message is automatically generated. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1, 0.24.0 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3754: --- Attachment: MAPREDUCE-3754-20120131.txt Setting appropriate titles for the filtered pages. Tested on single node. RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3754: --- Status: Patch Available (was: Open) RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3754: --- Status: Open (was: Patch Available) RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3765) FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting
[ https://issues.apache.org/jira/browse/MAPREDUCE-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197163#comment-13197163 ] Hadoop QA commented on MAPREDUCE-3765: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512610/MR-3765.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1726//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1726//console This message is automatically generated. FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting Key: MAPREDUCE-3765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3765 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Attachments: MR-3765.1.patch, MR-3765.2.patch FifoScheduler uses default min 1 GB regardless of the configuration value set for minimum memory allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3744) Unable to retrieve application logs via yarn logs or mapred job -logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3744: -- Attachment: MAPREDUCE-3744.patch Patch to address both the yarn logs issue and better error messages when log aggregation not enabled. yarn logs problem was a copy-n-paste error grabbing the wrong property when computing the log dir suffix. To improve the error messages, it catches FileNotFound to print an error about aggregation rather than pass along the exception to the end user. Unable to retrieve application logs via yarn logs or mapred job -logs - Key: MAPREDUCE-3744 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3744 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-3744.patch Trying to retrieve application logs via the yarn logs shell command results in an error similar to this: Exception in thread main java.io.FileNotFoundException: File /tmp/logs/application_1327694122989_0001 does not exist. at org.apache.hadoop.fs.Hdfs$DirListingIterator.init(Hdfs.java:226) at org.apache.hadoop.fs.Hdfs$DirListingIterator.init(Hdfs.java:217) at org.apache.hadoop.fs.Hdfs$2.init(Hdfs.java:192) at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:192) at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1371) at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1) at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2319) at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1373) at org.apache.hadoop.yarn.logaggregation.LogDumper.dumpAllContainersLogs(LogDumper.java:191) at org.apache.hadoop.yarn.logaggregation.LogDumper.run(LogDumper.java:107) at org.apache.hadoop.yarn.logaggregation.LogDumper.main(LogDumper.java:226) Trying to grab the logs via the mapred jobs -logs command results in this error: 2012-01-27 14:05:52,040 INFO mapred.ClientServiceDelegate (ClientServiceDelegate.java:getProxy(246)) - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 2012-01-27 14:05:52,041 WARN mapred.ClientServiceDelegate (ClientServiceDelegate.java:checkAndGetHSProxy(257)) - Job History Server is not configured. Unable to get log information for job: job_1327694122989_0001 Even though the historyserver process is running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3744) Unable to retrieve application logs via yarn logs or mapred job -logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-3744: -- Target Version/s: 0.23.1, 0.24.0 Status: Patch Available (was: Open) Unable to retrieve application logs via yarn logs or mapred job -logs - Key: MAPREDUCE-3744 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3744 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Blocker Attachments: MAPREDUCE-3744.patch Trying to retrieve application logs via the yarn logs shell command results in an error similar to this: Exception in thread main java.io.FileNotFoundException: File /tmp/logs/application_1327694122989_0001 does not exist. at org.apache.hadoop.fs.Hdfs$DirListingIterator.init(Hdfs.java:226) at org.apache.hadoop.fs.Hdfs$DirListingIterator.init(Hdfs.java:217) at org.apache.hadoop.fs.Hdfs$2.init(Hdfs.java:192) at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:192) at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1371) at org.apache.hadoop.fs.FileContext$20.next(FileContext.java:1) at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2319) at org.apache.hadoop.fs.FileContext.listStatus(FileContext.java:1373) at org.apache.hadoop.yarn.logaggregation.LogDumper.dumpAllContainersLogs(LogDumper.java:191) at org.apache.hadoop.yarn.logaggregation.LogDumper.run(LogDumper.java:107) at org.apache.hadoop.yarn.logaggregation.LogDumper.main(LogDumper.java:226) Trying to grab the logs via the mapred jobs -logs command results in this error: 2012-01-27 14:05:52,040 INFO mapred.ClientServiceDelegate (ClientServiceDelegate.java:getProxy(246)) - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server 2012-01-27 14:05:52,041 WARN mapred.ClientServiceDelegate (ClientServiceDelegate.java:checkAndGetHSProxy(257)) - Job History Server is not configured. Unable to get log information for job: job_1327694122989_0001 Even though the historyserver process is running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197185#comment-13197185 ] Robert Joseph Evans commented on MAPREDUCE-3754: I finally got around to running this and taking a look at the actually web UI. I think the change as it is now, is fine +1. For consistency I would prefer to see something more like the history server, where we allow for column by column filtering in the javascript. If you would rather leave the patch as is that is fine and I am happy to commit it for you. RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3758) NPE while submitting job through Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated MAPREDUCE-3758: --- Resolution: Invalid Status: Resolved (was: Patch Available) After adding hadoop-yarn-server-common*.jar on oozie's list of libraries and trying with the latest build, this works. NPE while submitting job through Oozie -- Key: MAPREDUCE-3758 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3758 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Attachments: MR-3758.patch NPE while submitting job through oozie. Caused by: java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:212) at -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3758) NPE while submitting job through Oozie
[ https://issues.apache.org/jira/browse/MAPREDUCE-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197287#comment-13197287 ] Mahadev konar commented on MAPREDUCE-3758: -- I think, we need a fix here. Ill open a new jira for that. We need to move yarn-default.xml to the lowest common denominator for the jars like: hadoop-yarn-common. NPE while submitting job through Oozie -- Key: MAPREDUCE-3758 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3758 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Attachments: MR-3758.patch NPE while submitting job through oozie. Caused by: java.lang.NullPointerException at org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:212) at -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3774:
-
Attachment: MAPREDUCE-3774.patch
This patch should fix it.
Here is a simple svn command to run when committing:
{code}
svn mv
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/
{code}
yarn-default.xml should be moved to hadoop-yarn-common.
---
Key: MAPREDUCE-3774
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3774.patch
yarn-default.xml right now resides in hadoop-yarn-server-common jars which is
not the right thing to do since this jar might not be needed in some cases
when depending upon yarn. We should move it to hadoop-yarn-common which is a
required dependency for all the yarn components (client/server).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3774: - Status: Patch Available (was: Open) yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3768) MR-2450 introduced a significant performance regression
[
https://issues.apache.org/jira/browse/MAPREDUCE-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197303#comment-13197303
]
Eli Collins commented on MAPREDUCE-3768:
I don't. This is a pretty straightforward query and it runs 7x faster on MR1
and the same MR2 build w/o MR-2450 so I suspect MR-2450 had an unintended
side-effect, making a thousand jobs faster doesn't imply it didn't introduce a
new bug.
MR-2450 introduced a significant performance regression
---
Key: MAPREDUCE-3768
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3768
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Eli Collins
Priority: Blocker
MAPREDUCE-2450 introduced, or at least triggers, a significant performance
regression in Hive. With MR-2450 the execution time of TestCliDriver.skewjoin
goes from 2 minutes to 15 minutes. Reverting this change from the build fixes
the issue.
Here's the relevant query:
{noformat}
FROM src src1 JOIN src src2 ON (src1.key = src2.key)
INSERT OVERWRITE TABLE dest_j1 SELECT src1.key, src2.value;
{noformat}
You can reproduce this by running the following from Hive 8.0 against Hadoop
built from branch-23.
{noformat}
ant very-clean package test -Dtestcase=TestCliDriver -Dqfile=skewjoin.q
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197324#comment-13197324
]
Hitesh Shah commented on MAPREDUCE-3427:
TestUlimit seems to be failing as ulimit is not supported in mrv2. Likewise for
StreamingBadRecords as skipping bad records is also not supported.
Will open a separate jira for the above with respect to either cleaning up the
configuration settings or more clearly documenting which configs are not
supported. Will provide a patch shortly for commenting out the tests -
depending on whether ulimit or skip bad records will/will not be supported, the
test files could just be removed from svn.
TestStreamingCombiner seems to work in my local env. Anyone else still seeing
it fail?
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1, 0.24.0
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3775) Change MiniYarnCluster to escape special chars in testname
[ https://issues.apache.org/jira/browse/MAPREDUCE-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197329#comment-13197329 ] Hitesh Shah commented on MAPREDUCE-3775: Seen creating test failures for TestStreamingBadRecords in MAPREDUCE-3427. Change MiniYarnCluster to escape special chars in testname -- Key: MAPREDUCE-3775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor When using MiniYarnCluster with the testname set to a nested classname, the $ within the class name creates issues with the container launch scripts as they try to expand the $... within the paths/variables in use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3775) Change MiniYarnCluster to escape special chars in testname
Change MiniYarnCluster to escape special chars in testname -- Key: MAPREDUCE-3775 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3775 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor When using MiniYarnCluster with the testname set to a nested classname, the $ within the class name creates issues with the container launch scripts as they try to expand the $... within the paths/variables in use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated MAPREDUCE-3427:
---
Attachment: MR-3427.1.patch
Patch to comment out testulimit and teststreamingbadrecords tests.
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1, 0.24.0
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hitesh Shah updated MAPREDUCE-3427:
---
Status: Patch Available (was: Open)
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1, 0.24.0
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197334#comment-13197334
]
Hadoop QA commented on MAPREDUCE-3427:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12512679/MR-3427.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1730//console
This message is automatically generated.
streaming tests fail with MR2
-
Key: MAPREDUCE-3427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Alejandro Abdelnur
Assignee: Hitesh Shah
Priority: Blocker
Fix For: 0.23.1, 0.24.0
Attachments: MR-3427.1.patch
After Mavenizing streaming and getting its testcases to use the MiniMRCluster
wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have
been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of
them).
*TestStreamingCombiner*, fails because returned counters don't match
assertion. However, counters printed in the test output indicate values that
would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
passing back counter information to the client API.
*TestStreamingBadRecords*, the job is failing with the following exception
{code}
Application application_1321575850006_0001 failed 1 times due to AM Container
for
appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
.Failing this attempt.. Failing the application.
{code}
Difficult to troubleshoot because there are not task logs from Mini MR/YARN
run.
*TestStreamingStatus* fails in validateTaskStatus() in the following assertion
{code}
expected:[before consuming input sort] but was:[SUCCEEDED]
{code}
*TestUlimit* fails with
{code}
org.junit.ComparisonFailure: output is wrong expected:[786432] but
was:[unlimited]
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3499) New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197358#comment-13197358 ] Mahadev konar commented on MAPREDUCE-3499: -- @Alejandro, Any pointers to re create the counters issue? Trying to fix it but would help if I can test the fix easily via oozie. New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work -- Key: MAPREDUCE-3499 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3499 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1, 0.24.0 Reporter: Alejandro Abdelnur Assignee: John George Priority: Blocker Labels: bigtop Fix For: 0.23.1, 0.24.0 Attachments: MR-3499.patch, MR-3499.patch, MR-3499.patch, MR-3499.patch, TestForMAPREDUCE-3499.patch The new MiniMR implementation is not taking proxyuser settings. Because of this, testcases using/testing doAs functionality fail. This affects all Oozie testcases that use MiniMR. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3776) org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator fails intermittently
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator fails intermittently
Key: MAPREDUCE-3776
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3776
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.24.0
Reporter: Robert Joseph Evans
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator fails intermittently
{noformat}
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.267 sec
FAILURE!
testReportedAppProgressWithOnlyMaps(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
Time elapsed: 11.908 sec FAILURE!
junit.framework.AssertionFailedError: Task state is not correct (timedout)
expected:RUNNING but was:SCHEDULED
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:283)
at junit.framework.Assert.assertEquals(Assert.java:64)
at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:236)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgressWithOnlyMaps(TestRMContainerAllocator.java:564)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3776) org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3776:
---
Attachment:
TEST-org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.xml
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.txt
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator-output.txt
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator fails
intermittently
Key: MAPREDUCE-3776
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3776
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.24.0
Reporter: Robert Joseph Evans
Attachments:
TEST-org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.xml,
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator-output.txt,
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.txt
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator fails
intermittently
{noformat}
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.267 sec
FAILURE!
testReportedAppProgressWithOnlyMaps(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
Time elapsed: 11.908 sec FAILURE!
junit.framework.AssertionFailedError: Task state is not correct (timedout)
expected:RUNNING but was:SCHEDULED
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:283)
at junit.framework.Assert.assertEquals(Assert.java:64)
at
org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:236)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testReportedAppProgressWithOnlyMaps(TestRMContainerAllocator.java:564)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3774) yarn-default.xml should be moved to hadoop-yarn-common.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197363#comment-13197363 ] Hadoop QA commented on MAPREDUCE-3774: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12512666/MAPREDUCE-3774.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1729//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1729//console This message is automatically generated. yarn-default.xml should be moved to hadoop-yarn-common. --- Key: MAPREDUCE-3774 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3774 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Mahadev konar Assignee: Mahadev konar Fix For: 0.23.1 Attachments: MAPREDUCE-3774.patch yarn-default.xml right now resides in hadoop-yarn-server-common jars which is not the right thing to do since this jar might not be needed in some cases when depending upon yarn. We should move it to hadoop-yarn-common which is a required dependency for all the yarn components (client/server). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197378#comment-13197378 ] John George commented on MAPREDUCE-3696: Mahadev - verified that mapreduce.job.user.classpath.first gets propagated to both the AppM and the tasks by looking at the launch_container.sh script. MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3696) MR job via oozie does not work on hadoop 23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197380#comment-13197380 ] John George commented on MAPREDUCE-3696: Mahadev - verified that the classpath on both the AM and tasks containers are the same by looking at the launch_container.sh CLASSPATH after passing in the mapreduce.job.user.classpath.first=true. MR job via oozie does not work on hadoop 23 --- Key: MAPREDUCE-3696 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3696 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: John George Priority: Blocker Attachments: MR-3696.patch, MR-3696.patch, MR-3696.patch NM throws an error on submitting an MR job via oozie on the latest Hadoop 23. *Courtesy: Mona Chitnis (ooize) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3583) ProcfsBasedProcessTree#constructProcessInfo() may throw NumberFormatException
[
https://issues.apache.org/jira/browse/MAPREDUCE-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evan Pollan updated MAPREDUCE-3583:
---
Priority: Critical (was: Major)
This became a critical blocking issue for me today. This is preventing distcp
commands from completing successfully on two different CDH3 update 2
environment's I'm using, meaning I cannot do any offline log
processing/analytics.
I think the above analysis of the failure is a bit off -- it's not actually the
pid that's blowing up the number parsing: it's one of the (presumed) longs.
The code is extracting capture groups 7, 8, 10, and 11, parsing them as signed
64-bit longs, and interpreting them as utime, stime, vsize, and rss,
respectively.
Here's an example of the contents of a /proc/X/stat file on one of my affected
systems, listed in conjunction with how the man page describes each field
| pid | 1686 |
| comm | (ssh) |
| state | S |
| ppid | 1685 |
| pgrp | 1672 |
| session | 1415 |
| tty_nr | 34816 |
| tpgid | 4884 |
| flags | 4202496 |
| minflt | 1922 |
| cminflt | 0 |
| majflt | 3 |
| cmajflt | 0 |
| utime | 67 |
| stime | 82 |
| cutime | 0 |
| cstime | 0 |
| priority | 20 |
| nice | 0 |
| num_threads | 1 |
| itrealvalue | 0 |
| starttime | 144184 |
| vsize | 62341120 |
| rss | 1120 |
| rsslim | 18,446,744,073,709,500,000 |
| startcode | 139,935,780,638,720 |
| endcode | 139,935,781,007,452 |
| startstack | 140,735,070,560,080 |
| kstkesp | 140,735,070,553,640 |
| kstkeip | 139,935,743,316,835 |
| signal | 0 |
| blocked | 0 |
| sigignore | 4102 |
| sigcatch | 134234113 |
| wchan | 18,446,744,071,579,900,000 |
| nswap | 0 |
| cnswap | 0 |
| exit_signal | 17 |
| processor | 0 |
| rt_priority | 0 |
| policy | 0 |
| delayacct_blkio_ticks | 2 |
| guest_time | 0 |
| cguest_time | 0 |
As I said, I'm using cloudera CDH3U2, and the relevant regexp pattern used to
capture /proc/X/stat fields is:
{code}
private static final Pattern PROCFS_STAT_FILE_FORMAT = Pattern
.compile(^([0-9-]+)\\s([^\\s]+)\\s[^\\s]\\s([0-9-]+)\\s([0-9-]+)\\s([0-9-]+)\\s([0-9-]+\\s){16}([0-9]+)(\\s[0-9-]+){16});
{code}
The parsing code is:
{code}
// Set ( name ) ( ppid ) ( pgrpId ) (session ) (vsize )
pinfo.updateProcessInfo(m.group(2), Integer.parseInt(m.group(3)),
Integer
.parseInt(m.group(4)), Integer.parseInt(m.group(5)), Long
.parseLong(m.group(7)));
{code}
The thing that's baffling me is that the field the Long.parseLong is choking on
is nowhere to be found in the contents of any /proc/X/stat file that exists
while the job is running. E.g., :
{code}
2/01/31 23:31:03 INFO tools.DistCp: sourcePathsCount=1
12/01/31 23:31:03 INFO tools.DistCp: filesToCopyCount=1
12/01/31 23:31:03 INFO tools.DistCp: bytesToCopyCount=122.0k
12/01/31 23:31:03 INFO mapred.JobClient: Running job: job_201201312321_0002
12/01/31 23:31:04 INFO mapred.JobClient: map 0% reduce 0%
12/01/31 23:31:08 INFO mapred.JobClient: Task Id :
attempt_201201312321_0002_m_02_0, Status : FAILED
java.lang.NumberFormatException: For input string: 18446744073709551532
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Long.parseLong(Long.java:422)
at java.lang.Long.parseLong(Long.java:468)
at
org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:413)
at
org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:148)
at
org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
at org.apache.hadoop.mapred.Task.initialize(Task.java:532)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:306)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
{code}
Here's what the entire set of /proc/X/stat files look like while this job is
running (I'm looking at the /proc file system on the only task tracker/data
node in the cluster) -- if Long.parseLong was going to fail, I assume it would
choke on '18446744073709551615'.:
{code}
10 (async/mgr) S 2 0 0 0 -1 2149613632 0 0 0 0 0 0 0 0 20 0 1 0 93 0 0
18446744073709551615 0 0 0 0 0 0 0 2147483647 0 18446744073709551615 0 0 17 0 0
0 0 0 0
11 (xenwatch) S 2 0 0 0 -1 2149613888 0 0 0 0 1 0 0 0 20 0 1 0 93 0 0
18446744073709551615 0 0 0 0 0 0 0 2147483647 0 18446744073709551615 0 0 17 0 0
0 0 0 0
120 (upstart-udev-br) S 1 119 119 0 -1 4202560 215 0 0 0 6 0 0 0 20 0 1 0 234
17444864 239 18446744073709551615 140724289748992 140724289787412 0 0 0 0 0
4097 81920 18446744073709551615 0 0 17 1 0
[jira] [Created] (MAPREDUCE-3777) used mem and util have negative values after a queue addition
used mem and util have negative values after a queue addition - Key: MAPREDUCE-3777 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3777 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy Fix For: 0.23.1 After a queue addition to capacity scheduler and submission of an application, root queue utilization and used memory have negative values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197441#comment-13197441 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3754: bq. For consistency I would prefer to see something more like the history server The problem with doing that at the top level list is that once the page refreshes, the searched list (search on state) is gone. We can fix it by sending the search to the server, but don't think that's what we started with given we wanted all search on the client side. Though I agree that it is still useful for the individual pages after filtering out based on state. Let's open a separate JIRA. I am pushing this in. RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3778) Per-state RM app-pages should have search ala JHS pages
Per-state RM app-pages should have search ala JHS pages --- Key: MAPREDUCE-3778 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3778 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3754: --- Resolution: Fixed Release Note: Modified RM UI to filter applications based on state of the applications. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk and branch-0.23. RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3754) RM webapp should have pages filtered based on App-state
[ https://issues.apache.org/jira/browse/MAPREDUCE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13197454#comment-13197454 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3754: Thanks for the review, Robert! RM webapp should have pages filtered based on App-state --- Key: MAPREDUCE-3754 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3754 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3754-20120129.txt, MAPREDUCE-3754-20120130.txt, MAPREDUCE-3754-20120131.txt Helps a lot when we have lot of apps. Already having difficulties with gridmix with a single big list of apps of all states. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
