[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769277#comment-13769277 ] Zhijie Shen commented on YARN-1001: --- Vinod, thanks for committing the patch. In addition to the new feature we added in this jira, some additional thought came into my mind: 1. RMContext store all the applications in one map. This data structure is not friendly to GetApps and GetAppStatistics, which need to go through the whole collection. Given some regular query criteria, such as states and applicationTypes, it would be good if we can somehow index the apps according to these criteria. Therefore, we can speed up the two APIs, maybe also some other operations on the app collection of RMContext. 2. Now web API ignores the case when matching the applicationType. IMHO, ApplicationClientProtocol should have the same behavior. Moreover, I think it's good to define criteria of naming an applicationType, for example, letter and number only, case insensitive and etc. Thoughts? YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769427#comment-13769427 ] Hudson commented on YARN-1001: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #335 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/335/]) YARN-1001. Added a web-service to get statistics about per application-type per state for consumption by downstream projects. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1523855) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ApplicationStatisticsInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/StatisticsItemInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769450#comment-13769450 ] Hudson commented on YARN-1001: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1525 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1525/]) YARN-1001. Added a web-service to get statistics about per application-type per state for consumption by downstream projects. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1523855) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ApplicationStatisticsInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/StatisticsItemInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769513#comment-13769513 ] Hudson commented on YARN-1001: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1551 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1551/]) YARN-1001. Added a web-service to get statistics about per application-type per state for consumption by downstream projects. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1523855) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ApplicationStatisticsInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/StatisticsItemInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769645#comment-13769645 ] Xuan Gong commented on YARN-1001: - bq. RMContext store all the applications in one map. This data structure is not friendly to GetApps and GetAppStatistics, which need to go through the whole collection. Given some regular query criteria, such as states and applicationTypes, it would be good if we can somehow index the apps according to these criteria. Therefore, we can speed up the two APIs, maybe also some other operations on the app collection of RMContext. I think that we are fine to use ConcurrentMap. Because most of time, we are just using add/remove. The situation that need to go through the whole collection to find a specific value mostly exist in webService and CLIs, and those webService request and client requests are very high frequency (at least not as frequent as we are using add/remove). So, we are fine to just use ConcurrentMap data structure. bq. Now web API ignores the case when matching the applicationType. IMHO, ApplicationClientProtocol should have the same behavior. Moreover, I think it's good to define criteria of naming an applicationType, for example, letter and number only, case insensitive and etc. About the case insensitive, I think that happens mostly when the users do the input. Since we have already take care of those case insensitive in ApplicationCLI methods or WebService methods, I do not think ApplicationClientProtocol (ClientRMService) need to consider this. So, for ApplicationClientProtocol, everything can be capital, for the users input, they can be either capital or lower case. When we do actually comparison(in applicationCLI methods or WebService methods), we can convert them to either capital or lower case. That is what we are doing currently, and I think it is enough. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769703#comment-13769703 ] Vinod Kumar Vavilapalli commented on YARN-1001: --- Re indices. I think its a fine tradeoff. Like Xuan mentioned, the non UI code path is more critical. Re the app-Type. In the minimum, we should document in the submitApplication() call that it is case insensitive. I did observe this, but it went over my mind. Please file a ticket to fix documentation. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768549#comment-13768549
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603372/YARN-1001.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1938//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1938//console
This message is automatically generated.
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch,
YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768464#comment-13768464 ] Xuan Gong commented on YARN-1001: - May need to update the patch with the latest trunk. RMServerUtils. createApplicationState() has been changed YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13768948#comment-13768948
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603470/YARN-1001.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1942//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1942//console
This message is automatically generated.
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch,
YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769001#comment-13769001 ] Vinod Kumar Vavilapalli commented on YARN-1001: --- +1, looks good. Checking this in. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769030#comment-13769030 ] Hudson commented on YARN-1001: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4425 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4425/]) YARN-1001. Added a web-service to get statistics about per application-type per state for consumption by downstream projects. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1523855) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ApplicationStatisticsInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/StatisticsItemInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Fix For: 2.1.1-beta Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch, YARN-1001.6.patch, YARN-1001.7.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766878#comment-13766878
]
Srimanth Gunturi commented on YARN-1001:
What we need is a call like {{/appStates?type=mapreduce}}, and that would give
per-state counts of the various MR apps. It should also include a total count
of MR apps.
Something like
{noformat}
{
total: 10,
submitted: 10,
running: 3,
pending: 4,
completed: 2,
killed: 1,
failed: 1
}
{noformat}
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766950#comment-13766950
]
Zhijie Shen commented on YARN-1001:
---
Having talked to [~srimanth.gunturi] offline. Here's more specifications for
the API:
1. The API takes exact one applicationType now. If we check there's no or more
than one applicationType, we throw exception. We may work only multiple
applicationTypes in the future.
2. The API takes zero to many states. If no state is specified, we enumerate
all states of RMApp, and return the count of each state. If the states are
specified, we just return the counts of these states.
3. We output the results as following:
{code}
appStatInfo
statItem
statesubmitted/state
typemapreduce/type
count10/count
/statItem
statItem
staterunning/state
typemapreduce/type
count3/count
/statItem
/appStatInfo
{code}
We don't particularly list the total count, as it can be concluded by summing
up all the counts, and it is not fit for the schema.
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767159#comment-13767159
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603145/YARN-1001.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1925//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1925//console
This message is automatically generated.
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch,
YARN-1001.4.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767162#comment-13767162 ] Xuan Gong commented on YARN-1001: - One quick comment: Do we suppose to use YarnApplicationState instead of exposing the real RMApp state ? It is true that most of them are one-to-one matching. Also ApplicationCLI uses YarnApplicationState, do they need to be consistent ? YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767183#comment-13767183 ] Xuan Gong commented on YARN-1001: - And maybe let appState accepts ALL type ? Looks good for other parts YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767284#comment-13767284 ] Xuan Gong commented on YARN-1001: - +1 LGTM YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Blocker Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch, YARN-1001.4.patch, YARN-1001.5.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764107#comment-13764107
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602544/YARN-1001.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1895//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1895//console
This message is automatically generated.
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Priority: Blocker
Attachments: YARN-1001.1.patch, YARN-1001.2.patch, YARN-1001.3.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763634#comment-13763634 ] Xuan Gong commented on YARN-1001: - +1 Looks good YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Priority: Critical Attachments: YARN-1001.1.patch, YARN-1001.2.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760507#comment-13760507
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12601872/YARN-1001.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1864//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1864//console
This message is automatically generated.
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Attachments: YARN-1001.1.patch, YARN-1001.2.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13760460#comment-13760460 ] Zhijie Shen commented on YARN-1001: --- bq. we are expecting /ws/v1/cluster/appscount to provide all app-types/state-counts in 1 call To prevent RM from being overwhelmed, when no params is specified, the API returns a empty response. Users must supply the states and the types (at least one) to get the counts. bq. Apart from that, we need /ws/v1/cluster/appscount information pushed to Ganglia. Though some discussion, we'd like to exclude this requirement from the scope of this jira due to the performance concern. We can open a ticket to trace it separately. Another issue we need to clarify is that the counts depend on the current apps in RMContext. Old finished apps may be removed from the context if the total app number reaches the limit. Therefore, the count of completed apps may only reflect the count of those in RMContext. Should emphasize the apps in RMContext in the document. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Attachments: YARN-1001.1.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757597#comment-13757597 ] Steve Loughran commented on YARN-1001: -- As long as everyone is happy with the URL structure, yes. If it were just one app type at a time I'd push for having the app in the URL. One thing I'd like to see is tests to verify that it works for app types with the following characters in them: space, ?, ,, , %. \. There's no restrictions on the type of an app (unless someone wants to add those restrictions right now, *which may not be a bad idea*), so you need to make sure the unusual characters make it all the way to this API and back YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Attachments: YARN-1001.1.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757925#comment-13757925
]
Srimanth Gunturi commented on YARN-1001:
Ambari's primary use-case is to show in YARN UI the app-type/state distribution
as a time-series graph. For this we would make periodic calls to
{{/ws/v1/cluster/appscount}}. Apart from that we need similar information for
MR2 UI, where a call to {{/ws/v1/cluster/appscount?types=mapreduce}} would be
made.
For now having these calls should suffice.
I am hoping that these calls include both the current/real-time app-type
counts, as well as historical information (atleast until last RM restart)?
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Attachments: YARN-1001.1.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757973#comment-13757973 ] Zhijie Shen commented on YARN-1001: --- [~srimanth.gunturi], for /ws/v1/cluster/appscount, does Ambari want to specify multiple states and multiple application-types in the params, and get the per application-type and state count for every combination of one application-type and one state? Or Ambari is fine to make multiple calls, each of which just specify one (or zero) application-type and state? The two cases make a big difference to the results. The former one needs to return a table which contains the counts of all application-type and state combinations, while the latter one is so simple to return one number. bq. I am hoping that these calls include both the current/real-time app-type counts, as well as historical information (atleast until last RM restart)? A count every constant time interval? Would you please specify more about the requirement? bq. There's no restrictions on the type of an app (unless someone wants to add those restrictions right now, which may not be a bad idea), so you need to make sure the unusual characters make it all the way to this API and back Sounds a good idea YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Attachments: YARN-1001.1.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13758182#comment-13758182
]
Srimanth Gunturi commented on YARN-1001:
[~zjshen], we are expecting {{/ws/v1/cluster/appscount}} to provide all
app-types/state-counts in 1 call.
We are expecting {{/ws/v1/cluster/appscount?types=mapreduce}} to provide all
mapreduce state-counts in 1 call.
Apart from that, we need {{/ws/v1/cluster/appscount}} information pushed to
Ganglia. Or else Ambari will not be able to show various graphs which are
important. We currently populate
{{/etc/hadoop/conf/hadoop-metrics2.properties}} file telling RM to push to
Ganglia (resourcemanager.sink.ganglia.servers).
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Attachments: YARN-1001.1.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13757378#comment-13757378 ] Srimanth Gunturi commented on YARN-1001: If the work is done, can this patch be committed for the next release? Ambari has functionality dependent on this. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi Assignee: Zhijie Shen Attachments: YARN-1001.1.patch In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752203#comment-13752203
]
Hadoop QA commented on YARN-1001:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600347/YARN-1001.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1780//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1780//console
This message is automatically generated.
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Attachments: YARN-1001.1.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[
https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752612#comment-13752612
]
Steve Loughran commented on YARN-1001:
--
I'd recommend using {{toLowerCase(EN_US)}} in case conversion so it works
consistently everywhere
YARN should provide per application-type and state statistics
-
Key: YARN-1001
URL: https://issues.apache.org/jira/browse/YARN-1001
Project: Hadoop YARN
Issue Type: Task
Components: api
Affects Versions: 2.1.0-beta
Reporter: Srimanth Gunturi
Assignee: Zhijie Shen
Attachments: YARN-1001.1.patch
In Ambari we plan to show for MR2 the number of applications finished,
running, waiting, etc. It would be efficient if YARN could provide per
application-type and state aggregated counts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733219#comment-13733219 ] Zhijie Shen commented on YARN-1001: --- [~srimanth.gunturi], let me reword the requirement. Giving some application types and states, Ambari wants to categorize the applications into the buckets of all combinations of these application types and states, and count the number of the applications in each bucket. For example, users want to know the number of the applications of three application types: type1, type2, and type3, and two states: state1 and state2. Assume RM has 5 applications: app1(type1, state1), app2(type2, state1), app3(type2, state1), app4(type2, state2), app5(type3, state1). The users will get the following statistics: [type1, state1]: 1 [type1, state2]: 0 [type2, state1]: 2 [type2, state2]: 1 [type3, state1]: 1 [type3, state2]: 0 Is this exactly what Ambari wants? YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733588#comment-13733588 ] Srimanth Gunturi commented on YARN-1001: [~zjshen], yes. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13733688#comment-13733688 ] Zhijie Shen commented on YARN-1001: --- Then, the limitation of this requirement is that the dimension of the params to categorize the applications is not free to scale. If you want add one more para, the number of buckets will be increased exponentially. Therefore, I'm afraid this proposed API cannot support as many params as getApps() can do. @Srimanth Gunturi, do you think application type and state are enough for Ambari? YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725740#comment-13725740 ] Zhijie Shen commented on YARN-1001: --- [~srimanth.gunturi], I'm not fully sure it will meet Ambari's requirement, but it is worth mentioning that actually getApplication() now can get the applications of a certain type by supplying the type name. Then the count can be easily concluded from the response. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1001) YARN should provide per application-type and state statistics
[ https://issues.apache.org/jira/browse/YARN-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13725751#comment-13725751 ] Srimanth Gunturi commented on YARN-1001: [~zjshen], yes we could filter by app-type + state and count. But this is very inefficient as the number of applications could be large with potentially paging involved. We dont want to read the huge outputs just to get the counts. It would be helpful if state-counts per app-type be provided. YARN should provide per application-type and state statistics - Key: YARN-1001 URL: https://issues.apache.org/jira/browse/YARN-1001 Project: Hadoop YARN Issue Type: Task Components: api Affects Versions: 2.1.0-beta Reporter: Srimanth Gunturi In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
