[
https://issues.apache.org/jira/browse/YARN-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351456#comment-17351456
]
Qi Zhu commented on YARN-10781:
-------------------------------
[~zhangxiping]
If you enabled rolling log aggregation for long running jobs?
It can ease your problem.
> The Thread of the NM aggregate log is exhausted and no other Application can
> aggregate the log
> ----------------------------------------------------------------------------------------------
>
> Key: YARN-10781
> URL: https://issues.apache.org/jira/browse/YARN-10781
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Affects Versions: 2.9.2, 3.3.0
> Reporter: Xiping Zhang
> Priority: Major
> Attachments: applications.png, containers.png, containers.png
>
>
> We observed more than 100 applications running on one NM.Most of these
> applications are SparkStreaming applications, but these applications do not
> have running Containers.When the offline application running on it finishes,
> the log cannot be reported to HDFS. When we killed a large number of
> SparkStreaming applications, we found that a large number of log files were
> being created on the NN side, causing the read and write performance on the
> NN side to degrade significantly.Causes the business application to time out。
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
