Zach Fry created SPARK-7917:
-------------------------------
Summary: Spark doesn't clean up Application Directories (local
dirs)
Key: SPARK-7917
URL: https://issues.apache.org/jira/browse/SPARK-7917
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.3.0
Reporter: Zach Fry
Similar to SPARK-4834.
Spark does clean up the cache and lock files in the local dirs, however, it
doesn't clean up the actual directories.
We have to write custom scripts to go back through the local dirs and find
directories that don't contain any files and clear those out.
Its a pretty simple repro:
Run a job that does some shuffling, wait for the shuffle files to get cleaned
up, go and look on disk at spark.local.dir and notice that the directory(s) are
still there, but there are no files in them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
