[ https://issues.apache.org/jira/browse/YARN-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800433#comment-13800433 ]
Zhijie Shen commented on YARN-925: ---------------------------------- bq. I have two methods in the interface one with the filters and other without the filters. It's not about one or two methods. In fact, if we don't supply any filters in one method, it actually degrades to the other without param. Essentially, we just need the one with the parameters. No matter what kind of filters we can provide, one query can still return a huge number of applications. For example, if we have 1 Million Mapreduce applications in store, and we filter with applicationType = mapreduce. What I mean about *two*, is one for small dataset and the other for large dataset: * Map<> getAllApplications(...) * Iterator<> getAllApplications(...) We need set the upper bound of the number of applications for the first one. On the other hand, the second one is prepared for iteratively retrieve the applications one-by-one (maybe not one, but the user defined acceptable number). Therefore, we don't worry about exploding the memory. The server needs to implement the detail of how to respond the users one-by-one. Probably it's not just the scope of HistoryStore, but the end-to-end feature. bq. Implementation details of these apis, we can discuss in the corrsponding JIRAS. I'd like to confirm what filters we want to push back to HistoryStore first, instead of aggressively adding all. If the implementation of the interface has no choice but enumerating each applications and then check, the filter is worthless of being pushed back. > HistoryStorage Reader Interface for Application History Server > -------------------------------------------------------------- > > Key: YARN-925 > URL: https://issues.apache.org/jira/browse/YARN-925 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Mayank Bansal > Assignee: Mayank Bansal > Fix For: YARN-321 > > Attachments: YARN-925-1.patch, YARN-925-2.patch, YARN-925-3.patch, > YARN-925-4.patch, YARN-925-5.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)