[ https://issues.apache.org/jira/browse/SPARK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570339#comment-14570339 ]
Weizhong edited comment on SPARK-7917 at 6/3/15 6:20 AM: --------------------------------------------------------- On standalone mode, will kept one tmp dir on each executor as local root dir and reuse it each time we submit a application, the tmp dir will be deleted when Worker stopped. was (Author: sephiroth-lin): On standalone mode, will kept one tmp dir on each executor and reuse it each time we submit a application, the tmp dir will be deleted when Worker stopped. > Spark doesn't clean up Application Directories (local dirs) > ------------------------------------------------------------ > > Key: SPARK-7917 > URL: https://issues.apache.org/jira/browse/SPARK-7917 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.3.0 > Reporter: Zach Fry > > Similar to SPARK-4834. > Spark does clean up the cache and lock files in the local dirs, however, it > doesn't clean up the actual directories. > We have to write custom scripts to go back through the local dirs and find > directories that don't contain any files and clear those out. > Its a pretty simple repro: > Run a job that does some shuffling, wait for the shuffle files to get cleaned > up, go and look on disk at spark.local.dir and notice that the directory(s) > are still there, but there are no files in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org