[jira] [Commented] (YARN-925) Augment HistoryStorage Reader Interface to Support Filters When Getting Applications
[ https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094763#comment-14094763 ] Karthik Kambatla commented on YARN-925: --- Folks, I just realized a part of this went into 2.5.0 and the remaining is yet to be committed. Can we please open a follow-up JIRA to finish this, so we don't have the same JIRA against multiple releases. If people working on this are okay with that, please mark this as resolved with fixversion 2.5.0. Thanks. Augment HistoryStorage Reader Interface to Support Filters When Getting Applications Key: YARN-925 URL: https://issues.apache.org/jira/browse/YARN-925 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Shinichi Yamashita Fix For: YARN-321 Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch, YARN-925-4.patch, YARN-925-5.patch, YARN-925-6.patch, YARN-925-7.patch, YARN-925-8.patch, YARN-925-9.patch We need to allow filter parameters for getApplications, pushing filtering to the implementations of the interface. The implementations should know the best about optimizing filtering. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-925) Augment HistoryStorage Reader Interface to Support Filters When Getting Applications
[ https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094825#comment-14094825 ] Zhijie Shen commented on YARN-925: -- bq. Folks, I just realized a part of this went into 2.5.0 and the remaining is yet to be committed. AFAIK, YARN-925 is part of 2.4. I reopened the ticket as we may want to improve the reader interface. Apparently, it was not able to be done within 2.4 window. bq. If people working on this are okay with that, please mark this as resolved with fixversion 2.5.0. Let's mark it be fixed for 2.4. And move the following discussion to a followup Jira. Augment HistoryStorage Reader Interface to Support Filters When Getting Applications Key: YARN-925 URL: https://issues.apache.org/jira/browse/YARN-925 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Shinichi Yamashita Fix For: YARN-321 Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch, YARN-925-4.patch, YARN-925-5.patch, YARN-925-6.patch, YARN-925-7.patch, YARN-925-8.patch, YARN-925-9.patch We need to allow filter parameters for getApplications, pushing filtering to the implementations of the interface. The implementations should know the best about optimizing filtering. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-925) Augment HistoryStorage Reader Interface to Support Filters When Getting Applications
[
https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13909294#comment-13909294
]
Hadoop QA commented on YARN-925:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630465/YARN-925-9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3155//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3155//console
This message is automatically generated.
Augment HistoryStorage Reader Interface to Support Filters When Getting
Applications
Key: YARN-925
URL: https://issues.apache.org/jira/browse/YARN-925
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Shinichi Yamashita
Fix For: YARN-321
Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch,
YARN-925-4.patch, YARN-925-5.patch, YARN-925-6.patch, YARN-925-7.patch,
YARN-925-8.patch, YARN-925-9.patch
We need to allow filter parameters for getApplications, pushing filtering to
the implementations of the interface. The implementations should know the
best about optimizing filtering.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (YARN-925) Augment HistoryStorage Reader Interface to Support Filters When Getting Applications
[ https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13885152#comment-13885152 ] Shinichi Yamashita commented on YARN-925: - Thank you for the check of the patch. As you say, the current patch is not good for a histroy of huge application. I thought about a plan to add filter information to a history file name and a plan to add find command to HDFS. However, I thought that your idea was simple and better than mine. I think about a better method. Augment HistoryStorage Reader Interface to Support Filters When Getting Applications Key: YARN-925 URL: https://issues.apache.org/jira/browse/YARN-925 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Shinichi Yamashita Fix For: YARN-321 Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch, YARN-925-4.patch, YARN-925-5.patch, YARN-925-6.patch, YARN-925-7.patch, YARN-925-8.patch We need to allow filter parameters for getApplications, pushing filtering to the implementations of the interface. The implementations should know the best about optimizing filtering. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-925) Augment HistoryStorage Reader Interface to Support Filters When Getting Applications
[ https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13883828#comment-13883828 ] Zhijie Shen commented on YARN-925: -- [~sinchii], thanks for taking care of the filters. I had quick look at the patch. IMO, it's on the right track. However, the major task of this issue is to optimize the filtering in the implementation of application history store, in particular FileSystemApplicationHistoryStore. The current patch still reads each individual history file and loads the full historical information of an application, followed by a number of filtering conditions. It doesn't make the difference with doing this filtering in ApplicationHistoryManager. Given a million history files, it will be a disaster to read all of them. By pushing the filters back to the implementation of application history store, I suppose that the implementation knows best about how the historic data is stored, and we can do optimization here. In the FS implementation, ideally, we should build an index in some way, and only read the historical files that hit the filters. Augment HistoryStorage Reader Interface to Support Filters When Getting Applications Key: YARN-925 URL: https://issues.apache.org/jira/browse/YARN-925 Project: Hadoop YARN Issue Type: Sub-task Reporter: Mayank Bansal Assignee: Mayank Bansal Fix For: YARN-321 Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch, YARN-925-4.patch, YARN-925-5.patch, YARN-925-6.patch, YARN-925-7.patch, YARN-925-8.patch We need to allow filter parameters for getApplications, pushing filtering to the implementations of the interface. The implementations should know the best about optimizing filtering. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (YARN-925) Augment HistoryStorage Reader Interface to Support Filters When Getting Applications
[
https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13878365#comment-13878365
]
Hadoop QA commented on YARN-925:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12624288/YARN-925-8.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/2925//console
This message is automatically generated.
Augment HistoryStorage Reader Interface to Support Filters When Getting
Applications
Key: YARN-925
URL: https://issues.apache.org/jira/browse/YARN-925
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Fix For: YARN-321
Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch,
YARN-925-4.patch, YARN-925-5.patch, YARN-925-6.patch, YARN-925-7.patch,
YARN-925-8.patch
We need to allow filter parameters for getApplications, pushing filtering to
the implementations of the interface. The implementations should know the
best about optimizing filtering.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
