[ https://issues.apache.org/jira/browse/SPARK-24340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jiang Xingbo resolved SPARK-24340. ---------------------------------- Resolution: Fixed > Clean up non-shuffle disk block manager files following executor death > ---------------------------------------------------------------------- > > Key: SPARK-24340 > URL: https://issues.apache.org/jira/browse/SPARK-24340 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.4.0 > Reporter: Jiang Xingbo > Priority: Major > > Currently we only clean up local folders on application removed, and we don't > clean up non-shuffle files, such as temp. shuffle blocks, cached > RDD/broadcast blocks, spill files, etc. and this can cause disk space leaks > when executors periodically die and are replaced. > To avoid this source of disk space leak, we can clean up executor disk store > files except for shuffle index and data files on executor finished. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org