[
https://issues.apache.org/jira/browse/YARN-8617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714696#comment-16714696
]
Bibin A Chundatt commented on YARN-8617:
----------------------------------------
[~Prabhu Joseph]
Looked into the issue again ..YARN-2583 contains 2 parts
# limit number of files per node
public static final String NM_LOG_AGGREGATION_NUM_LOG_FILES_SIZE_PER_APP
= NM_PREFIX + "log-aggregation.num-log-files-per-app";
# Delete files old than expiry time.
{code}
if (appDir.isDirectory() &&
appDir.getModificationTime() < cutoffMillis) {
{code}
{quote}
The AggrgeatedLogDeletionService does deletion for Running Job based upon the
file modification time which always will be latest as the rolled logs are
getting updated into the node1 file regularly
{quote}
For long running service the *application folder* eg
:user/logs/application_1234 modification time gets updated on every upload
cycle.
This could cause nodefile to remain in hdfs if no new containers are allocated
to same node.
> Aggregated Application Logs accumulates for long running jobs
> -------------------------------------------------------------
>
> Key: YARN-8617
> URL: https://issues.apache.org/jira/browse/YARN-8617
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: log-aggregation
> Affects Versions: 2.7.4
> Reporter: Prabhu Joseph
> Priority: Major
>
> Currently AggregationDeletionService will delete older aggregated log files
> once when they are complete. This will cause logs to accumulate for Long
> Running Jobs like Llap, Spark Streaming.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
