Prabhu Joseph created YARN-8279: ----------------------------------- Summary: AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix Key: YARN-8279 URL: https://issues.apache.org/jira/browse/YARN-8279 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 2.7.3 Reporter: Prabhu Joseph
AggregationLogDeletionService does not honor yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix. AggregationLogService writes the logs into /app-logs/<user-dir>/logs-ifile where as AggregationLogDeletion tries to delete from /app-logs/<user-dir>/logs. Workaround is to set yarn.log-aggregation.IndexedFormat.remote-app-log-dir-suffix and yarn.nodemanager.remote-app-log-dir-suffix to same value "logs-ifile" AggregationLogDeletionService has to check the format and based upon that choose the suffix. Currently it only checks the older suffix yarn.nodemanager.remote-app-log-dir-suffix. AggregatedLogDeletionService tries to delete older suffix directory. {code} 2018-05-11 08:48:19,989 ERROR logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:logIOException(182)) - Could not read the contents of hdfs://prabhucluster:8020/app-logs/hive/logs java.io.FileNotFoundException: File hdfs://prabhucluster:8020/app-logs/hive/logs does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:923) at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:985) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:981) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:992) at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.deleteOldLogDirsFrom(AggregatedLogDeletionService.java:98) at org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService$LogDeletionTask.run(AggregatedLogDeletionService.java:85) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org