[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[ https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368482#comment-14368482 ] Rohith commented on YARN-3284: -- Thanks [~xgong] for working this jira.. [~leftnoteasy] for review.. Since patch size is huge, I think this task can be logically divide into 3 sub tasks which would help for reviewer for granular review and for implementer to rebase the code. # API changes includes proto's # Web UI includes updating metrics # Application CLI Any thoughts ? > Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN > command > - > > Key: YARN-3284 > URL: https://issues.apache.org/jira/browse/YARN-3284 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch, > YARN-3284.3.patch, YARN-3284.3.rebase.patch, YARN-3284.4.patch > > > Current, we have some extra metrics about the application and current attempt > in RM Web UI. We should expose that information through YARN Command, too. > 1. Preemption metrics > 2. application outstanding resource requests > 3. container locality info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[
https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368208#comment-14368208
]
Hadoop QA commented on YARN-3284:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705440/YARN-3284.4.patch
against trunk revision c239b6d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.v2.TestUberAM
The test build failed in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7017//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7017//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7017//console
This message is automatically generated.
> Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN
> command
> -
>
> Key: YARN-3284
> URL: https://issues.apache.org/jira/browse/YARN-3284
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch,
> YARN-3284.3.patch, YARN-3284.3.rebase.patch, YARN-3284.4.patch
>
>
> Current, we have some extra metrics about the application and current attempt
> in RM Web UI. We should expose that information through YARN Command, too.
> 1. Preemption metrics
> 2. application outstanding resource requests
> 3. container locality info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[ https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368070#comment-14368070 ] Wangda Tan commented on YARN-3284: -- Hi [~xgong], Thanks for working on this, It's very helpful! I reviewed part of the patch, majorly for API definition and RM internal changes: 1) RMAppAttemptMetrics.convertToApplicationAttemptMetrics(..): readLock needed? 2) Comment of ApplicationAttemptMetrics#getResourceRequests: "a list of ...this attempt has made" -> "a list of outstanding ResourceRequest of this attempt". 3) ApplicationAttemptMetrics#getLocalityStatistics seems not very proper to me, transfering 2d int array to string discards information. Instead, I suggest to keep such information and format it in CLI only, my proposal here is: - Add a LocalityType PB enum message, which contains "node-local", "rack-local", "offswitch". - Add a PB message named "LocalityStatisticProto", contains (LocalityType request, LocalityType container, int numberOfContainers) - repeat LocalityStatisticProto is a part of ApplicationAttemptMetrics 4) headroom of attempt metrics: I noticed FiCaSchedulerApp and schedulers will set the headroom when getAllocation/assignContainers are invoked. I suggest you can use SchedulerApplicationAttempt#getHeadroom to get up-to-date headroom of an app. Will include CLI changes in next round. > Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN > command > - > > Key: YARN-3284 > URL: https://issues.apache.org/jira/browse/YARN-3284 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch, > YARN-3284.3.patch, YARN-3284.3.rebase.patch, YARN-3284.4.patch > > > Current, we have some extra metrics about the application and current attempt > in RM Web UI. We should expose that information through YARN Command, too. > 1. Preemption metrics > 2. application outstanding resource requests > 3. container locality info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[ https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363317#comment-14363317 ] Rohith commented on YARN-3284: -- Thank you:-) > Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN > command > - > > Key: YARN-3284 > URL: https://issues.apache.org/jira/browse/YARN-3284 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch, > YARN-3284.3.patch > > > Current, we have some extra metrics about the application and current attempt > in RM Web UI. We should expose that information through YARN Command, too. > 1. Preemption metrics > 2. application outstanding resource requests > 3. container locality info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[
https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362207#comment-14362207
]
Hadoop QA commented on YARN-3284:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704628/YARN-3284.3.patch
against trunk revision bd0a9ba.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 8 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6963//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6963//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6963//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6963//console
This message is automatically generated.
> Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN
> command
> -
>
> Key: YARN-3284
> URL: https://issues.apache.org/jira/browse/YARN-3284
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch,
> YARN-3284.3.patch
>
>
> Current, we have some extra metrics about the application and current attempt
> in RM Web UI. We should expose that information through YARN Command, too.
> 1. Preemption metrics
> 2. application outstanding resource requests
> 3. container locality info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[ https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362134#comment-14362134 ] Xuan Gong commented on YARN-3284: - bq. For displaying application headroom for running applications( one of the point in YARN-3273), it is required to expose applicationHeadroom field in the ApplicationAttemptMetrics.java. Would you please mind adding this field in your patch that helps retrieve headroom? Added. But use the default value: Resource.newInstance(0,0). We could replace it with the real value in YARN-3273 > Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN > command > - > > Key: YARN-3284 > URL: https://issues.apache.org/jira/browse/YARN-3284 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch, > YARN-3284.3.patch > > > Current, we have some extra metrics about the application and current attempt > in RM Web UI. We should expose that information through YARN Command, too. > 1. Preemption metrics > 2. application outstanding resource requests > 3. container locality info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[ https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361607#comment-14361607 ] Rohith commented on YARN-3284: -- Hi [~xgong], thanks for working on this Jira. For displaying application headroom for running applications( one of the point in YARN-3273), it is required to expose applicationHeadroom field in the ApplicationAttemptMetrics.java. Would you please mind adding this field in your patch that helps retrieve headroom? > Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN > command > - > > Key: YARN-3284 > URL: https://issues.apache.org/jira/browse/YARN-3284 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch > > > Current, we have some extra metrics about the application and current attempt > in RM Web UI. We should expose that information through YARN Command, too. > 1. Preemption metrics > 2. application outstanding resource requests > 3. container locality info -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[
https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361572#comment-14361572
]
Hadoop QA commented on YARN-3284:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704544/YARN-3284.2.patch
against trunk revision 6fdef76.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1153 javac
compiler warnings (more than the trunk's current 1152 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6960//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6960//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6960//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6960//console
This message is automatically generated.
> Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN
> command
> -
>
> Key: YARN-3284
> URL: https://issues.apache.org/jira/browse/YARN-3284
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-3284.1.patch, YARN-3284.1.patch, YARN-3284.2.patch
>
>
> Current, we have some extra metrics about the application and current attempt
> in RM Web UI. We should expose that information through YARN Command, too.
> 1. Preemption metrics
> 2. application outstanding resource requests
> 3. container locality info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3284) Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN command
[
https://issues.apache.org/jira/browse/YARN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359969#comment-14359969
]
Hadoop QA commented on YARN-3284:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704326/YARN-3284.1.patch
against trunk revision 8212877.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 9 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
org.apache.hadoop.yarn.server.resourcemanager.TestRM
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6949//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6949//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6949//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/6949//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6949//console
This message is automatically generated.
> Expose more ApplicationMetrics and ApplicationAttemptMetrics through YARN
> command
> -
>
> Key: YARN-3284
> URL: https://issues.apache.org/jira/browse/YARN-3284
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-3284.1.patch, YARN-3284.1.patch
>
>
> Current, we have some extra metrics about the application and current attempt
> in RM Web UI. We should expose that information through YARN Command, too.
> 1. Preemption metrics
> 2. application outstanding resource requests
> 3. container locality info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
