[ https://issues.apache.org/jira/browse/SPARK-30069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17172264#comment-17172264 ]
Apache Spark commented on SPARK-30069: -------------------------------------- User 'LantaoJin' has created a pull request for this issue: https://github.com/apache/spark/pull/29378 > Clean up non-shuffle disk block manager files following executor exists on > YARN > ------------------------------------------------------------------------------- > > Key: SPARK-30069 > URL: https://issues.apache.org/jira/browse/SPARK-30069 > Project: Spark > Issue Type: Bug > Components: Spark Core, YARN > Affects Versions: 3.0.0 > Reporter: Lantao Jin > Priority: Major > > Currently we only clean up the local directories on application removed. > However, when executors die and restart repeatedly, many temp files are left > untouched in the local directories, which is undesired behavior and could > cause disk space used up gradually. > SPARK-24340 had fixed this problem in standalone mode. But in YARN mode, this > issue still exists. Especially, in long running service like Spark > thrift-server with dynamic resource allocation disabled, it's very easy > causes local disk full. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org