[
https://issues.apache.org/jira/browse/YARN-11407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021660#comment-18021660
]
zeekling commented on YARN-11407:
---------------------------------
same as YARN-11644
> short-live job log aggregation blocked on LRA
> ----------------------------------------------
>
> Key: YARN-11407
> URL: https://issues.apache.org/jira/browse/YARN-11407
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Shuyi Zhang
> Priority: Minor
>
> Currently, log aggregation threads are scheduled based on applications start
> time. This means later applications that start later but finish earlier, will
> be blocked on log aggregation, until the previous running jobs finish and
> release a log aggregation thread for it to use. To resolve this bottleneck,
> we can decouple the initialization from the thread scheduling of the
> ApplogAggregator:
> Initializing an AppLogAggregator on application start
> But scheduling it in the thread pool on application finish
> This schedules log aggregation threads based on finish time instead
> application start time, so prevents having a pileup of jobs waiting for log
> aggregation, especially when waiting for long-running jobs to finish.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
