[ https://issues.apache.org/jira/browse/YARN-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569873#comment-16569873 ]
Tarun Parimi commented on YARN-8627: ------------------------------------ On further analysis, I found that this error occurs for application directories which themselves have another application directory such as /ats/done/1500089190015/0000/017/application_1500089190015_17219/application_1500089190015_17219. In audit logs I see that ats tries to list the folder after deleting it, which causes the error. {code:java} 19:35:45,944 INFO FSNamesystem.audit: allowed=true ugi=yarn/rm-u...@example.com (auth:KERBEROS) ip=/x.x.x.x cmd=listStatus src=/ats/done/1500089190015/0000/017/application_1500089190015_17219 dst=null perm=null proto=rpc callerContext=yarn_ats_server_v1_5 19:35:45,945 INFO FSNamesystem.audit: allowed=true ugi=yarn/rm-u...@example.com (auth:KERBEROS) ip=/x.x.x.x cmd=listStatus src=/ats/done/1500089190015/0000/017/application_1500089190015_17219 dst=null perm=null proto=rpc callerContext=yarn_ats_server_v1_5 19:35:45,946 INFO FSNamesystem.audit: allowed=true ugi=yarn/rm-u...@example.com (auth:KERBEROS) ip=/x.x.x.x cmd=listStatus src=/ats/done/1500089190015/0000/017/application_1500089190015_17219/appattempt_1500089190015_17219_000001 dst=null perm=null proto=rpc callerContext=yarn_ats_server_v1_5 19:35:45,947 INFO FSNamesystem.audit: allowed=true ugi=yarn/rm-u...@example.com (auth:KERBEROS) ip=/x.x.x.x cmd=listStatus src=/ats/done/1500089190015/0000/017/application_1500089190015_17219/application_1500089190015_17219 dst=null perm=null proto=rpc callerContext=yarn_ats_server_v1_5 19:35:45,948 INFO FSNamesystem.audit: allowed=true ugi=yarn/rm-u...@example.com (auth:KERBEROS) ip=/x.x.x.x cmd=listStatus src=/ats/done/1500089190015/0000/017/application_1500089190015_17219/application_1500089190015_17219/appattempt_1500089190015_17219_000001 dst=null perm=null proto=rpc callerContext=yarn_ats_server_v1_5 19:35:45,952 INFO FSNamesystem.audit: allowed=true ugi=yarn/rm-u...@example.com (auth:KERBEROS) ip=/x.x.x.x cmd=delete src=/ats/done/1500089190015/0000/017/application_1500089190015_17219 dst=null perm=null proto=rpc callerContext=yarn_ats_server_v1_5 19:35:45,953 INFO FSNamesystem.audit: allowed=true ugi=yarn/rm-u...@example.com (auth:KERBEROS) ip=/x.x.x.x cmd=listStatus src=/ats/done/1500089190015/0000/017/application_1500089190015_17219 dst=null perm=null proto=rpc callerContext=yarn_ats_server_v1_5 {code} I am not sure how this directory structure got created in the first place. But the cleaner thread should not list a directory after deleting the same. The {{EntityGroupFSTimelineStore#cleanLogs}} method tries to delete the parent directory {{dirpath}}, while it is iterating over the same dirpath. It should only try to delete its children so as to avoid these issues. Testing a patch which does this in my environment and it seems to fix the issue. Will upload a patch soon after doing further tests. > EntityGroupFSTimelineStore hdfs done directory keeps on accumulating > -------------------------------------------------------------------- > > Key: YARN-8627 > URL: https://issues.apache.org/jira/browse/YARN-8627 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver > Affects Versions: 2.8.0 > Reporter: Tarun Parimi > Assignee: Tarun Parimi > Priority: Major > > The EntityLogCleaner threads exits with the following ERROR every time it > runs. > {code:java} > 2018-07-18 19:59:39,837 INFO timeline.EntityGroupFSTimelineStore > (EntityGroupFSTimelineStore.java:cleanLogs(462)) - Deleting > hdfs://namenode/ats/done/1499684568068/0000/018/application_1499684568068_18268 > 2018-07-18 19:59:39,844 INFO timeline.EntityGroupFSTimelineStore > (EntityGroupFSTimelineStore.java:cleanLogs(462)) - Deleting > hdfs://namenode/ats/done/1499684568068/0000/018/application_1499684568068_18270 > 2018-07-18 19:59:39,848 ERROR timeline.EntityGroupFSTimelineStore > (EntityGroupFSTimelineStore.java:run(899)) - Error cleaning files > java.io.FileNotFoundException: File > hdfs://namenode/ats/done/1499684568068/0000/018/application_1499684568068_18270 > does not exist. at > org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1062) > at > org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1069) > at > org.apache.hadoop.hdfs.DistributedFileSystem$DirListingIterator.<init>(DistributedFileSystem.java:1040) > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1019) > at > org.apache.hadoop.hdfs.DistributedFileSystem$23.doCall(DistributedFileSystem.java:1015) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.listStatusIterator(DistributedFileSystem.java:1015) > at > org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.shouldCleanAppLogDir(EntityGroupFSTimelineStore.java:480) > > {code} > > Each time the thread gets scheduled, it is a different folder encountering > the error. As a result, the thread is not able to clean all the old done > directories, since it stops after this error. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org