[ https://issues.apache.org/jira/browse/SPARK-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728864#comment-14728864 ]
Chris Bannister commented on SPARK-9708: ---------------------------------------- When the executor is killed by Mesos it does not immediately clean up the sandbox dir, it waits for a GC time period based on total disk usage in the work_dir, im not entirely sure what will happen if the executor is stopped, if the data is still readable by external applications. Regarding spark.local.dir, as far as I understand it when running in YARN this is overridden by the YARN config, I intended to do something similar here. Would it be better to add a config option to explicitly enable this behaviour? > Spark should create local temporary directories in Mesos sandbox when > launched with Mesos > ----------------------------------------------------------------------------------------- > > Key: SPARK-9708 > URL: https://issues.apache.org/jira/browse/SPARK-9708 > Project: Spark > Issue Type: Bug > Components: Mesos > Reporter: Timothy Chen > > Currently Spark creates temporary directories with > Utils.getConfiguredLocalDirs, and it writes to YARN directories if YARN is > detected, otherwise just writes in a temporary directory in the host. > However, Mesos does create a directory per task and ideally Spark should use > that directory to create its local temporary directories since it then can be > cleaned up when the task is gone and not left on the host or cleaned until > reboot. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org