Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1969#issuecomment-52348177 This is an updated JIRA for the same issue [SPARK-2290](https://issues.apache.org/jira/browse/SPARK-2290). We established that, for standalone mode, we don't need to ship the driver's spark home to the executors, which may not use the same spark home. Instead, we should just use the `Worker`'s current working directory. However, I am not familiar enough with Mesos to comment on the need of shipping `SPARK_HOME` there. @iven There are many other places where we export `SPARK_HOME` in addition to these two. From a quick grep, I found the following: ``` bin/pyspark:export SPARK_HOME="$FWDIR" bin/run-example:export SPARK_HOME="$FWDIR" bin/spark-class:export SPARK_HOME="$FWDIR" bin/spark-submit:export SPARK_HOME="$(cd `dirname $0`/..; pwd)" sbin/spark-config.sh:export SPARK_HOME=${SPARK_PREFIX} ``` We need to do the same for all of these places in order for your original intended behavior to take effect. In the longer run, however, we should just clean up our usages of `SPARK_HOME`, since in many places we don't actually have any need to export it (or even use the variable `SPARK_HOME`).
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org