[ https://issues.apache.org/jira/browse/SPARK-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or updated SPARK-8716: ----------------------------- Summary: Write tests for executor shared cache feature (was: Remove executor shared cache feature) > Write tests for executor shared cache feature > --------------------------------------------- > > Key: SPARK-8716 > URL: https://issues.apache.org/jira/browse/SPARK-8716 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.2.0 > Reporter: Andrew Or > Assignee: Josh Rosen > > More specifically, this is the feature that is currently flagged by > `spark.files.useFetchCache`. There are several reasons why we should remove > it. > (1) It doesn't even work. Recently, each executor gets its own unique temp > directory for security reasons. > (2) There is no way to fix it. The constraints in (1) are fundamentally > opposed to sharing resources across executors. > (3) It is very complex. The method Utils.fetchFile would be greatly > simplified without this feature that is not even used. > (4) There are no tests for it and it is difficult to test. > Note that we can't just revert the respective patches because they were > merged a long time ago. > Related issues: SPARK-8130, SPARK-6313, SPARK-2713 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org