Jiang Xingbo created SPARK-24340:
------------------------------------
Summary: Clean up non-shuffle disk block manager files following
executor death
Key: SPARK-24340
URL: https://issues.apache.org/jira/browse/SPARK-24340
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Jiang Xingbo
Currently we only clean up local folders on application removed, and we don't
clean up non-shuffle files, such as temp. shuffle blocks, cached RDD/broadcast
blocks, spill files, etc. and this can cause disk space leaks when executors
periodically die and are replaced.
To avoid this source of disk space leak, we can clean up executor disk store
files except for shuffle index and data files on executor finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
