[
https://issues.apache.org/jira/browse/SPARK-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606682#comment-14606682
]
Andrew Or commented on SPARK-8716:
----------------------------------
Hm, I think you're right. Otherwise there's no way for external shuffle service
to work.
> Remove executor shared cache feature
> ------------------------------------
>
> Key: SPARK-8716
> URL: https://issues.apache.org/jira/browse/SPARK-8716
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Josh Rosen
>
> More specifically, this is the feature that is currently flagged by
> `spark.files.useFetchCache`. There are several reasons why we should remove
> it.
> (1) It doesn't even work. Recently, each executor gets its own unique temp
> directory for security reasons.
> (2) There is no way to fix it. The constraints in (1) are fundamentally
> opposed to sharing resources across executors.
> (3) It is very complex. The method Utils.fetchFile would be greatly
> simplified without this feature that is not even used.
> (4) There are no tests for it and it is difficult to test.
> Note that we can't just revert the respective patches because they were
> merged a long time ago.
> Related issues: SPARK-8130, SPARK-6313, SPARK-2713
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
