[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052666#comment-14052666 ] Wangda Tan commented on YARN-2144: -- Hi [~sunilg], Thanks for your review, but this patch is contained by YARN-2181. And YARN-2181 now puts log-printing logic in RMAppAttempt, it's similar using RMContainer#isAMContainer or comparing containerId with AM containerid. I'll close this JIRA to prevent misleading people. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052610#comment-14052610
]
Sunil G commented on YARN-2144:
---
Hi [~leftnoteasy] A quick nit!
RMContainer has isAMContainer interface now, could that be used instead of
{code}
if (attempt.getMasterContainer() != null
&& attempt.getMasterContainer().getId().equals(containerId))
{code}
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041543#comment-14041543 ] Wangda Tan commented on YARN-2144: -- Thanks Jian for review! > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041172#comment-14041172
]
Hadoop QA commented on YARN-2144:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652015/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4051//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4051//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040436#comment-14040436
]
Hadoop QA commented on YARN-2144:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651924/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4043//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4043//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040421#comment-14040421
]
Hadoop QA commented on YARN-2144:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651920/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4042//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4042//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4042//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040397#comment-14040397 ] Jian He commented on YARN-2144: --- bq. Yes, it should be never happen, but it's better to handle NPE here If it can never happen, generally we should not add null checks. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040361#comment-14040361
]
Hadoop QA commented on YARN-2144:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651911/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4041//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4041//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4041//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040343#comment-14040343 ] Wangda Tan commented on YARN-2144: -- Thanks [~jianhe] for review. bq. The first check can never happen? Yes, it should be never happen, but it's better to handle NPE here bq. Please add logs also if app/attempt equals to null. +1 Uploaded a new patch > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040328#comment-14040328
]
Jian He commented on YARN-2144:
---
The first check can never happen?
{code}
if (null == attemptId) {
return;
}
{code}
Please add logs also if app/attempt equals to null.
{code}
if (null == app) {
return;
}
if (null == attempt) {
return;
}
{code}
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040286#comment-14040286 ] Wangda Tan commented on YARN-2144: -- Hi [~curino], thanks for your comment. I think it's very important to make some logs parseable. Maybe putting it to system log is not a good solution, system log should be dedicated for human to read, we may need a separated module to do parseable logging, leveraging project like Apache BookKeeper. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040213#comment-14040213 ] Carlo Curino commented on YARN-2144: >From a quick skim, the patch seems fine. As a broader comment (not just about >these logs), I think it would be worth looking into more easily parseable >logs. We use a rather free form of logs, a bit more structure could go a long >way to facilitate debugging. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040140#comment-14040140 ] Wangda Tan commented on YARN-2144: -- The findbug warning is caused by un-synchronized method accesses rmContext. The method used in my latest patch is RMContext#getRMApps, it returns a ConcurrentMap, no need to access it in a synchronized block. I'd prefer not to fix this findbugs warning. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, > YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040128#comment-14040128
]
Hadoop QA commented on YARN-2144:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651866/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4039//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4039//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4039//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch,
> YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040115#comment-14040115 ] Wangda Tan commented on YARN-2144: -- Attached a new patch only changed CapacityScheduler.killContainer method to log needed information. [~jianhe], could you take a look at it again? Thanks. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040010#comment-14040010 ] Wangda Tan commented on YARN-2144: -- Since suggested by [~jianhe], to propose of this JIRA, I will simply add logs to CapacityScheduler.killContainer(). [~curino], thanks for your comment, since I may not need change event objects as suggested by Jian, I will do as your suggested when working on other items like YARN-2181 > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039934#comment-14039934 ] Carlo Curino commented on YARN-2144: I only skimmed the patch quickly, but I see several places where you are changing method signatures by adding booleans to communicate the container was preempted. Would it be possibly to use/extend some of the container state / event objects that are already passed around? It might be less intrusive, and if we ever get to different levels of preemption or anything like that, will also be more flexible of a mechanism. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039580#comment-14039580 ] Jian He commented on YARN-2144: --- "appId=appattempt_1402886643897_0002_01", appId can be changed to attemptId > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039434#comment-14039434 ] Jian He commented on YARN-2144: --- For the purpose of this jira, can we just log whatever Preemption information we need in CapacityScheduler#killContainer method ? > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038185#comment-14038185
]
Hadoop QA commented on YARN-2144:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651559/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4028//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4028//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch, YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037115#comment-14037115 ] Wangda Tan commented on YARN-2144: -- Attached a patch only contains preemption log related changes, [~jianhe], [~tassapola], do you have any comments? Thanks, > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036845#comment-14036845 ] Wangda Tan commented on YARN-2144: -- [~jianhe], I created a JIRA YARN-2181 to track web UI related changes, and leave this JIRA focus on logs. Will split and upload patch only for logs later. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036396#comment-14036396 ] Jian He commented on YARN-2144: --- Wangda, can you separate the patch ? let's have this jira for logging support for preemption and a separate jira for web-ui support for preemptions. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034725#comment-14034725
]
Hadoop QA commented on YARN-2144:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650937/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4018//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4018//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch, YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> 1. For debugging purpose, RM should log this.
> 2. For administrative purpose, RM webpage should have a page to show recent
> preemption events.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034624#comment-14034624 ] Jian He commented on YARN-2144: --- the patch needs rebase, can you update please? thx > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033326#comment-14033326 ] Wangda Tan commented on YARN-2144: -- bq. Haven't looked at the patch. YARN-1809 is adding the attempt UI, Maybe the app UI should show the total preempted containers info, and attempt UI should show each attempt's preempted containers info? Thanks for pointing the attempt UI JIRA. I think RM will cleanup application's resource usage at the beginning of each attempt start, so it should make sense to show latest attempt's preempted containers info on app UI. And after we can persist preemption info across RM restart and YARN-1807 committed, we can show each attempt's preempted containers info on attempts UI. Do you agree? > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033306#comment-14033306 ] Wangda Tan commented on YARN-2144: -- Hi [~tassapola], bq. In AM page, Does "Resource Preempted from Current Attempt" mean "Total Resource Preempted from Latest AM attempt"? Can it show only data point from current (is it latest?) attempt? Yes, Yes, it can only show data point from current(latest) attempt. bq. Can you change "#Container Preempted from Current Attempt:" to "Number of Containers Preempted from Current(Latest) Attempt"? # syntax maybe hard to comprehend for wider group of user. Thanks for this comment, I agree with you. I'll address this comment later. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033190#comment-14033190 ] Jian He commented on YARN-2144: --- Haven't looked at the patch. YARN-1809 is adding the attempt UI, Maybe the app UI should show the total preempted containers info, and attempt UI should show each attempt's preempted containers info ? > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032611#comment-14032611
]
Hadoop QA commented on YARN-2144:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650581/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3998//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3998//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch, YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> 1. For debugging purpose, RM should log this.
> 2. For administrative purpose, RM webpage should have a page to show recent
> preemption events.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032603#comment-14032603 ] Tassapol Athiapinya commented on YARN-2144: --- [~leftnoteasy] Can you please clarify me on these points? - In AM page, Does "Resource Preempted from Current Attempt" mean "Total Resource Preempted from Latest AM attempt"? Can it show only data point from current (is it latest?) attempt? - Can you change "#Container Preempted from Current Attempt:" to "Number of Containers Preempted from Current(Latest) Attempt"? # syntax maybe hard to comprehend for wider group of user. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > Attachments: AM-page-preemption-info.png, YARN-2144.patch, > YARN-2144.patch, YARN-2144.patch > > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032333#comment-14032333
]
Hadoop QA commented on YARN-2144:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650541/YARN-2144.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestChildQueueOrder
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3996//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3996//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch,
> YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> 1. For debugging purpose, RM should log this.
> 2. For administrative purpose, RM webpage should have a page to show recent
> preemption events.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[
https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032287#comment-14032287
]
Hadoop QA commented on YARN-2144:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650540/AM-page-preemption-info.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3995//console
This message is automatically generated.
> Add logs when preemption occurs
> ---
>
> Key: YARN-2144
> URL: https://issues.apache.org/jira/browse/YARN-2144
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.5.0
>Reporter: Tassapol Athiapinya
>Assignee: Wangda Tan
> Attachments: AM-page-preemption-info.png, YARN-2144.patch
>
>
> There should be easy-to-read logs when preemption does occur.
> 1. For debugging purpose, RM should log this.
> 2. For administrative purpose, RM webpage should have a page to show recent
> preemption events.
> RM logs should have following properties:
> * Logs are retrievable when an application is still running and often flushed.
> * Can distinguish between AM container preemption and task container
> preemption with container ID shown.
> * Should be INFO level log.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2144) Add logs when preemption occurs
[ https://issues.apache.org/jira/browse/YARN-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030331#comment-14030331 ] Wangda Tan commented on YARN-2144: -- Start working on this, assigned it to me. > Add logs when preemption occurs > --- > > Key: YARN-2144 > URL: https://issues.apache.org/jira/browse/YARN-2144 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.5.0 >Reporter: Tassapol Athiapinya >Assignee: Wangda Tan > > There should be easy-to-read logs when preemption does occur. > 1. For debugging purpose, RM should log this. > 2. For administrative purpose, RM webpage should have a page to show recent > preemption events. > RM logs should have following properties: > * Logs are retrievable when an application is still running and often flushed. > * Can distinguish between AM container preemption and task container > preemption with container ID shown. > * Should be INFO level log. -- This message was sent by Atlassian JIRA (v6.2#6252)
