[
https://issues.apache.org/jira/browse/SPARK-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557149#comment-15557149
]
holdenk commented on SPARK-1865:
--------------------------------
So ALS specifically has a work around for this with cleaning up shuffle files
on checkpoint, but if this is impacting other workloads we could look at maybe
trying to make that solution more general.
> Improve behavior of cleanup of disk state
> -----------------------------------------
>
> Key: SPARK-1865
> URL: https://issues.apache.org/jira/browse/SPARK-1865
> Project: Spark
> Issue Type: Improvement
> Components: Deploy, Spark Core
> Reporter: Aaron Davidson
>
> Right now the behavior of disk cleanup is centered around the exit hook of
> the executor, which attempts to cleanup shuffle files and disk manager
> blocks, but may fail. We should make this behavior more predictable, perhaps
> by letting the Standalone Worker cleanup the disk state, and adding a flag to
> disable having the executor cleanup its own state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
