[
https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15550218#comment-15550218
]
Sangjin Lee commented on YARN-5585:
-----------------------------------
Thanks [~rohithsharma] for contributing the initial patch for this! I have some
high level comments on this and a couple of specific ones.
(1)
I think the patch does this, but it would be great to leave this prefix
generic. For Tez, I'm assuming the (inverted) created time would be it. For
others, it might be something different (something that can be provided
easily). I think it is useful to have that flexibility. More importantly, it
should be *optional*. Any framework (and the YARN-generic ones) should be able
to skip the prefix and expect things to be sorted by the entity id order. I
think the patch reflects both, but wanted to clarify.
(2)
We also need to be *crystal clear* that timeline clients *must* provide the
same prefix for all subsequent updates of the same entity. I cannot stress that
point enough. Rohith, could you confirm that it is not an issue with Tez to
provide the created time for any subsequent updates for Tez entities?
(3)
I'm also realizing that we might have a bug in how we deal with entity id's. I
would have thought that we store the entities in the *reverse* entity id order,
but it appears that the entity id is encoded into the row key as is
({{EntityRowKey}}). Am I reading that right? If so, this is a bug to fix.
(4)
I agree with Varun that users should provide already inverted values. Users can
call {{LongConverter.invertLong(createdTime)}} to give us inverted values. We
also need to make this explicit in the javadoc.
(5)
I also agree with Varun that we need not store the prefix (again) as a column.
It would be part of the row key, and as such we should have no problem reading
it, right?
(6)
One other thing to deal with is the query by id. There, we need to be able to
distinguish the case where the data do not have the prefix to begin with and
that where data do. Ideally we would simply use the row key explicitly in the
case of data that don't have the prefix to begin with. For those that do have
the prefix, we cannot use the row key to fetch the row so we need to do
something different. I don't think this was done in the current patch, but this
is TBD.
(7)
Since this is a subtask for YARN-5355, can we base the patch on that feature
branch? Thanks!
> [Atsv2] Add a new filter fromId in REST endpoints
> -------------------------------------------------
>
> Key: YARN-5585
> URL: https://issues.apache.org/jira/browse/YARN-5585
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
> Reporter: Rohith Sharma K S
> Assignee: Rohith Sharma K S
> Priority: Critical
> Attachments: 0001-YARN-5585.patch, YARN-5585-workaround.patch,
> YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the
> applications. Along with those, it would be good to add new filter i.e fromId
> so that entities can be retrieved after the fromId.
> Current Behavior : Default limit is set to 100. If there are 1000 entities
> then REST call gives first/last 100 entities. How to retrieve next set of 100
> entities i.e 101 to 200 OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is
> no way to achieve this.
> So proposal is to have fromId in the filter like
> *getApps?limit=5&&fromId=app-5* which gives list of apps from app-6 to
> app-10.
> Since ATS is targeting large number of entities storage, it is very common
> use case to get next set of entities using fromId rather than querying all
> the entites. This is very useful for pagination in web UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
