Andrew Or created SPARK-8716: -------------------------------- Summary: Remove executor shared cache feature Key: SPARK-8716 URL: https://issues.apache.org/jira/browse/SPARK-8716 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 1.2.0 Reporter: Andrew Or Assignee: Josh Rosen Priority: Minor
More specifically, this is the feature that is currently flagged by `spark.files.useFetchCache`. There are several reasons why we should remove it. (1) It doesn't even work. Recently, each executor gets its own unique temp directory for security reasons. (2) There is no way to fix it. The constraints in (1) are fundamentally opposed to sharing resources across executors. (3) It is very complex. The method Utils.fetchFile would be greatly simplified without this feature that is not even used. (4) There are no tests for it and it is difficult to test. Note that we can't just revert the respective patches because they were merged a long time ago. Related issues: SPARK-8130, SPARK-6313, SPARK-2713 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org