[
https://issues.apache.org/jira/browse/YARN-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarun Parimi updated YARN-8279:
-------------------------------
Release Note:
This issue affects only when IndexedFormat is used for log-aggregation by
setting below properties:
yarn.log-aggregation.file-formats=IndexedFormat
yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
The fix is to set the below properties in yarn-site.xml and then restart
Mapreduce HistoryServer and YARN services.
yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix and
yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile" which
serves AggregationLogDeletionService
was:
This issue affects only when IndexedFormat is used for log-aggregation by
setting below properties:
yarn.log-aggregation.file-formats=IndexedFormat
yarn.log-aggregation.file-controller.IndexedFormat.class=org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController
The fix is to set yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix
and yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile" and
then restart Mapreduce HistoryServer which serves AggregationLogDeletionService
> AggregationLogDeletionService does not honor
> yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix
> ---------------------------------------------------------------------------------------------------------
>
> Key: YARN-8279
> URL: https://issues.apache.org/jira/browse/YARN-8279
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 2.9.1
> Reporter: Prabhu Joseph
> Assignee: Tarun Parimi
> Priority: Major
>
> AggregationLogDeletionService does not honor
> yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix.
> AggregationLogService writes the logs into /app-logs/<user-dir>/logs-ifile
> where as AggregationLogDeletion tries to delete from
> /app-logs/<user-dir>/logs.
> Workaround is to set
> yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix and
> yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile" and
> Restart HistoryServer which serves AggregationLogDeletionService
> AggregationLogDeletionService has to check the format and based upon that
> choose the suffix. Currently it only checks the older suffix
> yarn.nodemanager.remote-app-log-dir-suffix.
> AggregatedLogDeletionService tries to delete older suffix directory.
> {code}
> 2018-05-11 08:48:19,989 ERROR logaggregation.AggregatedLogDeletionService
> (AggregatedLogDeletionService.java:logIOException(182)) - Could not read the
> contents of hdfs://prabhucluster:8020/app-logs/hive/logs
> java.io.FileNotFoundException: File
> hdfs://prabhucluster:8020/app-logs/hive/logs does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:923)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:985)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:981)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:992)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.deleteOldLogDirsFrom(AggregatedLogDeletionService.java:98)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:85)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
