[jira] [Commented] (YARN-3272) Surface container locality info in RM web UI
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347035#comment-14347035 ] Hudson commented on YARN-3272: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2072 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2072/]) YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) (wangda: rev e17e5ba9d7e2bd45ba6884f59f8045817594b284) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Surface container locality info in RM web UI Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Fix For: 2.7.0 Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info in RM web UI
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346708#comment-14346708 ] Hudson commented on YARN-3272: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #122 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/122/]) YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) (wangda: rev e17e5ba9d7e2bd45ba6884f59f8045817594b284) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java Surface container locality info in RM web UI Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Fix For: 2.7.0 Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info in RM web UI
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346718#comment-14346718 ] Hudson commented on YARN-3272: -- FAILURE: Integrated in Hadoop-Yarn-trunk #856 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/856/]) YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) (wangda: rev e17e5ba9d7e2bd45ba6884f59f8045817594b284) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java Surface container locality info in RM web UI Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Fix For: 2.7.0 Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info in RM web UI
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346923#comment-14346923 ] Hudson commented on YARN-3272: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2054 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2054/]) YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) (wangda: rev e17e5ba9d7e2bd45ba6884f59f8045817594b284) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java Surface container locality info in RM web UI Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Fix For: 2.7.0 Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info in RM web UI
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346932#comment-14346932 ] Hudson commented on YARN-3272: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #113 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/113/]) YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) (wangda: rev e17e5ba9d7e2bd45ba6884f59f8045817594b284) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Surface container locality info in RM web UI Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Fix For: 2.7.0 Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info in RM web UI
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346998#comment-14346998 ] Hudson commented on YARN-3272: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #122 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/122/]) YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) (wangda: rev e17e5ba9d7e2bd45ba6884f59f8045817594b284) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/CHANGES.txt Surface container locality info in RM web UI Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Fix For: 2.7.0 Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345424#comment-14345424 ] Wangda Tan commented on YARN-3272: -- LGTM, +1 will commit today if no opposite opinions. Surface container locality info Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info in RM web UI
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345737#comment-14345737 ] Hudson commented on YARN-3272: -- FAILURE: Integrated in Hadoop-trunk-Commit #7246 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7246/]) YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) (wangda: rev e17e5ba9d7e2bd45ba6884f59f8045817594b284) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/NodeType.java * hadoop-yarn-project/hadoop-yarn/dev-support/findbugs-exclude.xml * hadoop-yarn-project/CHANGES.txt Surface container locality info in RM web UI Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Fix For: 2.7.0 Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343975#comment-14343975
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701987/YARN-3272.4.patch
against trunk revision ca1c00b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6808//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6808//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6808//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch,
YARN-3272.4.patch, container locality table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343788#comment-14343788 ] Wangda Tan commented on YARN-3272: -- Patch LGTM overall, but could you take a look at findbugs warning? Surface container locality info Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344155#comment-14344155
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702020/YARN-3272.5.patch
against trunk revision c5eac9c.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6810//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch,
YARN-3272.4.patch, YARN-3272.5.patch, container locality table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344322#comment-14344322
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702040/YARN-3272.5.patch
against trunk revision 431e7d8.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6813//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch,
YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, container locality
table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344587#comment-14344587 ] Jian He commented on YARN-3272: --- thanks for your comments, Xuan. I think regular array is more lightweight than that. suppressed the warnings. Surface container locality info Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344644#comment-14344644
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702085/YARN-3272.6.patch
against trunk revision d1c6acc.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6816//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6816//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6816//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch,
YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch,
YARN-3272.6.patch, container locality table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344549#comment-14344549 ] Xuan Gong commented on YARN-3272: - [~jianhe] Could you try AtomicIntegerArray to see if it can fix the findbug issue. Surface container locality info Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14344499#comment-14344499
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12702056/YARN-3272.6.patch
against trunk revision b18d383.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6815//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6815//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6815//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch,
YARN-3272.4.patch, YARN-3272.5.patch, YARN-3272.5.patch, YARN-3272.6.patch,
container locality table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343710#comment-14343710
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701946/YARN-3272.3.patch
against trunk revision ca1c00b.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6803//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6803//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6803//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch,
container locality table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343625#comment-14343625 ] Jian He commented on YARN-3272: --- fixed the test failures. Surface container locality info Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Attachments: YARN-3272.1.patch, YARN-3272.2.patch, YARN-3272.3.patch, container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341215#comment-14341215
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701500/YARN-3272.1.patch
against trunk revision edceced.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 8 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestAppPage
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6791//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6791//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6791//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, container locality
table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341267#comment-14341267
]
Hadoop QA commented on YARN-3272:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12701515/YARN-3272.2.patch
against trunk revision edceced.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 8 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestAppPage
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6793//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6793//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6793//console
This message is automatically generated.
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, YARN-3272.2.patch, container locality
table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[ https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341078#comment-14341078 ] Jian He commented on YARN-3272: --- attached a screens shot to demonstrate how the locality is represented. The table indicates the number of certain type of containers(i.e. node_local, rack_local, off_switch) satisfied by certain type of resource requests(i.e. node_local, rack_local, off_switch) Surface container locality info Key: YARN-3272 URL: https://issues.apache.org/jira/browse/YARN-3272 Project: Hadoop YARN Issue Type: Improvement Reporter: Jian He Assignee: Jian He Attachments: container locality table.png We can surface the container locality info on the web UI. This is useful to debug why my applications are progressing slow, especially when locality is bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3272) Surface container locality info
[
https://issues.apache.org/jira/browse/YARN-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341139#comment-14341139
]
Wangda Tan commented on YARN-3272:
--
Hi [~jianhe],
This is very useful! Just took a look, some comments:
1)
LeafQueue.assignNodeLocalContainers and 2 others, can you use MutableObject
instead of the array passed in?
2)
Logics in LeafQueue.assignContainersOnNode could be simplified, you can save
the resource-request.NodeType by a reference, which can avoid 3 if statements
like:
{code}
1313 // update locality statistics
1314 if (!allocatedContainers.isEmpty()) {
1315if (nodeLocalResourceRequest == null
1316 rackLocalResourceRequest == null) {
1317 application.incNumAllocatedContainers(NodeType.OFF_SWITCH,
1318NodeType.OFF_SWITCH);
1319} else if (nodeLocalResourceRequest == null
1320 rackLocalResourceRequest != null) {
1321 application.incNumAllocatedContainers(NodeType.OFF_SWITCH,
1322NodeType.RACK_LOCAL);
1323} else if (nodeLocalResourceRequest != null
1324 rackLocalResourceRequest != null) {
1325 application.incNumAllocatedContainers(NodeType.OFF_SWITCH,
1326NodeType.NODE_LOCAL);
1327}
1328LOG.info(allocatedContainers + allocatedContainers.get(0));
1329
1330 }
{code}
3)
This line in AppBlock can be simplified to:
{code}
235 .td(i == 0 ? : (i == 1 ? :
236
String.valueOf(attemptMetrics.getLocalityStatistics()[i][2])))._();
{code}
To be {{.td(i = 1 ? : : ...}}?
Surface container locality info
Key: YARN-3272
URL: https://issues.apache.org/jira/browse/YARN-3272
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-3272.1.patch, container locality table.png
We can surface the container locality info on the web UI. This is useful to
debug why my applications are progressing slow, especially when locality is
bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
