[ https://issues.apache.org/jira/browse/SPARK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-7917. ------------------------------ Resolution: Duplicate A-ha, that's the ticket. I knew there was something like this already resolved. > Spark doesn't clean up Application Directories (local dirs) > ------------------------------------------------------------ > > Key: SPARK-7917 > URL: https://issues.apache.org/jira/browse/SPARK-7917 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.3.0 > Reporter: Zach Fry > Priority: Minor > > Similar to SPARK-4834. > Spark does clean up the cache and lock files in the local dirs, however, it > doesn't clean up the actual directories. > We have to write custom scripts to go back through the local dirs and find > directories that don't contain any files and clear those out. > Its a pretty simple repro: > Run a job that does some shuffling, wait for the shuffle files to get cleaned > up, go and look on disk at spark.local.dir and notice that the directory(s) > are still there, but there are no files in them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org