[ https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512649#comment-14512649 ]
Marcelo Vanzin commented on SPARK-7108: --------------------------------------- The way I read the documentation, {{spark.local.dir}} should only ever work on the driver, never on executors, since as the documentation says, that is managed by the cluster manager (regardless of whether you set SPARK_LOCAL_DIRS for your app or not - as the doc says, the cluster manager sets that!). If {{spark.local.dir}} is not working for the driver, then there's a potential bug here. > spark.local.dir is no longer honored in Standalone mode > ------------------------------------------------------- > > Key: SPARK-7108 > URL: https://issues.apache.org/jira/browse/SPARK-7108 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.2.1, 1.3.0 > Reporter: Josh Rosen > Priority: Critical > > Prior to SPARK-4834, configuring spark.local.dir in the driver would affect > the local directories created on the executor. After this patch, executors > will always ignore this setting in favor of directories read from > {{SPARK_LOCAL_DIRS}}, which is set by the standalone worker based on the > worker's own configuration and not the application configuration. > This change impacts users who configured {{spark.local.dir}} only in their > driver and not via their cluster's {{spark-defaults.conf}} or > {{spark-env.sh}} files. This is an atypical use-case, since the available > local directories / disks are a property of the cluster and not the > application, which probably explains why this issue has not been reported > previously. > The correct fix might be comment + documentation improvements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org