[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372830#comment-14372830 ] Hudson commented on YARN-3350: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #139 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/139/]) YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * hadoop-yarn-project/CHANGES.txt YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Fix For: 2.8.0 Attachments: YARN-3350.2.patch, YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372743#comment-14372743 ] Hudson commented on YARN-3350: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2089 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2089/]) YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Fix For: 2.8.0 Attachments: YARN-3350.2.patch, YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372769#comment-14372769 ] Hudson commented on YARN-3350: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2071 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2071/]) YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * hadoop-yarn-project/CHANGES.txt YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Fix For: 2.8.0 Attachments: YARN-3350.2.patch, YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372795#comment-14372795 ] Hudson commented on YARN-3350: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #130 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/130/]) YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Fix For: 2.8.0 Attachments: YARN-3350.2.patch, YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372664#comment-14372664 ] Hudson commented on YARN-3350: -- FAILURE: Integrated in Hadoop-Yarn-trunk #873 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/873/]) YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * hadoop-yarn-project/CHANGES.txt YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Fix For: 2.8.0 Attachments: YARN-3350.2.patch, YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372431#comment-14372431 ] Hudson commented on YARN-3350: -- FAILURE: Integrated in Hadoop-trunk-Commit #7391 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7391/]) YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg (junping_du: rev 7f1e2f996995e1883d9336f720c27621cf1b73b6) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Fix For: 2.8.0 Attachments: YARN-3350.2.patch, YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363267#comment-14363267 ] Junping Du commented on YARN-3350: -- Thanks [~wilfreds] for reporting the issue and delivering a quick patch to fix it. A nit comment for the patch: LOG.debug() can be wrap inside of if (LOG.isDebugEnabled()) which could be faster is LOG is not enabling debug level. I just pinged [~sandyr] on SPARK-5393 as I am curious that why we didn't fix it in YARN side earlier. Let's wait for his feedback or we may miss something here. YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Attachments: YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363269#comment-14363269 ] Junping Du commented on YARN-3350: -- Sorry for typo: is LOG is not enabling debug level = if LOG is not enabling debug level YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Attachments: YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363456#comment-14363456 ] Junping Du commented on YARN-3350: -- Just get confirmation from Sandy at SPARK-5393. [~wilfreds], mind go ahead to deliver an updated patch here? Thanks! YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Attachments: YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363713#comment-14363713 ] Junping Du commented on YARN-3350: -- Thanks [~wilfreds] for updating the patch. +1 on latest patch. Will commit it tomorrow if no further comments from others. YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Attachments: YARN-3350.2.patch, YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[
https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363738#comment-14363738
]
Hadoop QA commented on YARN-3350:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704841/YARN-3350.2.patch
against trunk revision ed4e72a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6977//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6977//console
This message is automatically generated.
YARN RackResolver spams logs with messages at info level
Key: YARN-3350
URL: https://issues.apache.org/jira/browse/YARN-3350
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Wilfred Spiegelenburg
Assignee: Wilfred Spiegelenburg
Attachments: YARN-3350.2.patch, YARN-3350.patch,
yarn-RackResolver-log.txt
When you run an application the container logs shows a lot of messages for
the RackResolver:
2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator]
org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to
/rack15
A real world example for a large job was generating 20+ messages in 2
milliseconds during a sustained period of time flooding the logs causing the
node to run out of disk space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[
https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362718#comment-14362718
]
Hadoop QA commented on YARN-3350:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704712/YARN-3350.patch
against trunk revision 3ff1ba2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6970//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6970//console
This message is automatically generated.
YARN RackResolver spams logs with messages at info level
Key: YARN-3350
URL: https://issues.apache.org/jira/browse/YARN-3350
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Wilfred Spiegelenburg
Assignee: Wilfred Spiegelenburg
Attachments: YARN-3350.patch, yarn-RackResolver-log.txt
When you run an application the container logs shows a lot of messages for
the RackResolver:
2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator]
org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to
/rack15
A real world example for a large job was generating 20+ messages in 2
milliseconds during a sustained period of time flooding the logs causing the
node to run out of disk space.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3350) YARN RackResolver spams logs with messages at info level
[ https://issues.apache.org/jira/browse/YARN-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362771#comment-14362771 ] Wilfred Spiegelenburg commented on YARN-3350: - No tests for this change: it is just a simple log level change without any further code changes. BTW: this same issue was encountered by Spark as an application using the RackResolver and they changed the log level from their side to prevent log flooding via SPARK-5393 YARN RackResolver spams logs with messages at info level Key: YARN-3350 URL: https://issues.apache.org/jira/browse/YARN-3350 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.6.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Attachments: YARN-3350.patch, yarn-RackResolver-log.txt When you run an application the container logs shows a lot of messages for the RackResolver: 2015-03-10 00:58:30,483 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved node175.example.com to /rack15 A real world example for a large job was generating 20+ messages in 2 milliseconds during a sustained period of time flooding the logs causing the node to run out of disk space. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
