[
https://issues.apache.org/jira/browse/YARN-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419666#comment-16419666
]
Haibo Chen commented on YARN-6936:
----------------------------------
I checked the entity publisher in TimelineV2ClientImpl.java
{code:java}
private void publishWithoutBlockingOnQueue(
EntitiesHolder entitiesHolder) {
if (entitiesHolder.isSync()) {
entitiesHolder.run();
return;
}
int count = 1;
while (true) {
// loop till we find a sync put Entities or there is nothing
// to take
EntitiesHolder nextEntityInTheQueue = timelineEntityQueue.poll();
if (nextEntityInTheQueue == null) {
// Nothing in the queue just publish and get back to the
// blocked wait state
entitiesHolder.run();
break;
} else if (nextEntityInTheQueue.isSync()) {
// flush all the prev async entities first
entitiesHolder.run();
// and then flush the sync entity
nextEntityInTheQueue.run();
break;
} else {
// append all async entities together and then flush
entitiesHolder.getEntities().addEntities(
nextEntityInTheQueue.getEntities().getEntities());
count++;
if (count == numberOfAsyncsToMerge) {
// Flush the entities if the number of the async
// putEntites merged reaches the desired limit. To avoid
// collecting multiple entities and delaying for a long
// time.
entitiesHolder.run();
break;
}
}
}{code}
IIUC, it coalesces/merges multiple TimelineEntities into one putEntities() REST
call to TimelineCollector. Given that we won't change TimelineWriter API, I
think it is safe to not have another entity publisher for subApplication
entities.
Say there are two regular entities and one subApplication entity on the queue,
what the above does is that it will post all entities in one call. On the
HBaseTimelineWriter side, because we check the Entity Type for each entity, we
will handle all three entities correctly. Hence, I think we don't need to
create another publisher just for SubApplication Entities.
> [Atsv2] Retrospect storing entities into sub application table from client
> perspective
> --------------------------------------------------------------------------------------
>
> Key: YARN-6936
> URL: https://issues.apache.org/jira/browse/YARN-6936
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Rohith Sharma K S
> Assignee: Rohith Sharma K S
> Priority: Major
> Attachments: YARN-6936.000.patch, YARN-6936.001.patch
>
>
> Currently YARN-6734 stores entities into sub application table only if doAs
> user and submitted users are different. This holds good for Tez kind of use
> cases. But AM runs as same as submitted user like MR also need to store
> entities in sub application table so that it could read entities without
> application id.
> This would be a point of concern later stages when ATSv2 is deployed into
> production. This JIRA is to retrospect decision of storing entities into sub
> application table based on client side configuration driven rather than user
> driven.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
