[ https://issues.apache.org/jira/browse/SPARK-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725657#comment-14725657 ]
Iulian Dragos commented on SPARK-9708: -------------------------------------- I'm not sure if this is the entire story. Remember that shuffle files need to survive the executor when dynamic allocation is enabled. So, with this proposed change, if the scheduler decides to kill an executor its shuffle files will be gone and the external shuffle server won't be able to find them anymore. At least shuffle files need to go on another directory, not under the sandbox. Also, Spark allows one to configure `spark.local.dir`, and that should take precedence. In the Hadoop world, this can be used to specify several directories on different physical disks (to allow fast parallel writes). > Spark should create local temporary directories in Mesos sandbox when > launched with Mesos > ----------------------------------------------------------------------------------------- > > Key: SPARK-9708 > URL: https://issues.apache.org/jira/browse/SPARK-9708 > Project: Spark > Issue Type: Bug > Components: Mesos > Reporter: Timothy Chen > > Currently Spark creates temporary directories with > Utils.getConfiguredLocalDirs, and it writes to YARN directories if YARN is > detected, otherwise just writes in a temporary directory in the host. > However, Mesos does create a directory per task and ideally Spark should use > that directory to create its local temporary directories since it then can be > cleaned up when the task is gone and not left on the host or cleaned until > reboot. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org