[
https://issues.apache.org/jira/browse/SPARK-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514822#comment-14514822
]
Patrick Wendell edited comment on SPARK-7108 at 4/27/15 7:50 PM:
-----------------------------------------------------------------
Yes - the case I'm referring to is the following:
1. The standalone cluster does not set SPARK_LOCAL_DIRS.
2. The user sets spark.local.dir to some value.
The expected behavior is that "spark.local.dir" is respected for that
application. I don't consider it a "bug" that we respect "spark.local.dir" if
SPARK_LOCAL_DIRS is not set. The reasons why I don't consider it a bug are:
1. Prior to Spark 1.0 this was the only way for users to set the local dir, so
there is reasonable user expectation that this will still work.
2. The patch specifically intended to let it still work, for compatibility
reasons (https://github.com/apache/spark/pull/299).
3. The documentation does not imply that it won't work. The documentation just
implies that if both are present that SPARK_LOCAL_DIRS will take precedence.
Prior to 1.3.0, this continued to work in accordance with all previous versions
of Spark. The expected behavior is that if you set spark.local.dir it will
honor it.
In my view it is not user error, it's clearly a regression in user facing
behavior.
It might be that we just document as such and move on, but I want to clarify
that I think things like this are clearly regressions. If we don't intend this
to work then why does "spark.local.dir" even exist as a documented
configuration?
was (Author: pwendell):
Yes - the case I'm referring to is the following:
1. The standalone cluster does not set SPARK_LOCAL_DIRS.
2. The user sets spark.local.dir to some value.
The expected behavior is that "spark.local.dir" is respected for that
application. I don't consider it a "bug" that we respect "spark.local.dir" if
SPARK_LOCAL_DIRS is not set. The reasons why I don't consider it a bug are:
1. Prior to Spark 1.0 this was the only way for users to set the local dir, so
there is reasonable user expectation that this will still work.
2. The patch specifically intended to let it still work, for compatibility
reasons (https://github.com/apache/spark/pull/299).
3. The documentation does not imply that it won't work. The documentation just
implies that if both are present that SPARK_LOCAL_DIRS will take precedence.
Prior to 1.3.0, this continued to work in accordance with all previous versions
of Spark. The expected behavior is that if you set spark.local.dir it will
honor it.
In my view it is not user error, it's clearly a regression in user facing
behavior.
It might be that we just document as such and move on, but I want to clarify
that I think things like this are clearly regressions.
> spark.local.dir is no longer honored in Standalone mode
> -------------------------------------------------------
>
> Key: SPARK-7108
> URL: https://issues.apache.org/jira/browse/SPARK-7108
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.1, 1.3.0
> Reporter: Josh Rosen
> Priority: Critical
>
> Prior to SPARK-4834, configuring spark.local.dir in the driver would affect
> the local directories created on the executor. After this patch, executors
> will always ignore this setting in favor of directories read from
> {{SPARK_LOCAL_DIRS}}, which is set by the standalone worker based on the
> worker's own configuration and not the application configuration.
> This change impacts users who configured {{spark.local.dir}} only in their
> driver and not via their cluster's {{spark-defaults.conf}} or
> {{spark-env.sh}} files. This is an atypical use-case, since the available
> local directories / disks are a property of the cluster and not the
> application, which probably explains why this issue has not been reported
> previously.
> The correct fix might be comment + documentation improvements.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
