[
https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512280#comment-14512280
]
Patrick Wendell edited comment on SPARK-7108 at 4/25/15 6:02 AM:
-----------------------------------------------------------------
Hey I think [~joshrosen] actually worded this a bit confusingly. The issue is
that even if SPARK_LOCAL_DIRS is not set at all by the user, the setting of
"spark.local.dir" is not used from the application. This regresses from earlier
versions of spark which (as the documentation implies) would respect
spark.local.dir if set.
This isn't great because Spark will silently start using a different local
directory when upgraded. In our case it caused us to run out of disk space
because /tmp was used instead of a directory we'd explicitly set.
was (Author: pwendell):
Hey I think [~joshrosen] actually worded this a bit confusingly. The issue is
that even if SPARK_LOCAL_DIRS is not set at all by the user, the setting of
"spark.local.dir" is not used from the application. This regresses from earlier
versions of spark which (as the documentation implies) would respect
spark.local.dir if set.
> Setting spark.local.dir in driver no longer overrides the standalone worker's
> local directory setting
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-7108
> URL: https://issues.apache.org/jira/browse/SPARK-7108
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.1, 1.3.0
> Reporter: Josh Rosen
> Priority: Critical
>
> Prior to SPARK-4834, configuring spark.local.dir in the driver would affect
> the local directories created on the executor. After this patch, executors
> will always ignore this setting in favor of directories read from
> {{SPARK_LOCAL_DIRS}}, which is set by the standalone worker based on the
> worker's own configuration and not the application configuration.
> This change impacts users who configured {{spark.local.dir}} only in their
> driver and not via their cluster's {{spark-defaults.conf}} or
> {{spark-env.sh}} files. This is an atypical use-case, since the available
> local directories / disks are a property of the cluster and not the
> application, which probably explains why this issue has not been reported
> previously.
> The correct fix might be comment + documentation improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
