[ https://issues.apache.org/jira/browse/SPARK-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-13258. ------------------------------- Resolution: Not A Problem Great, thanks for the research and follow up. > --conf properties not honored in Mesos cluster mode > --------------------------------------------------- > > Key: SPARK-13258 > URL: https://issues.apache.org/jira/browse/SPARK-13258 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 1.6.0 > Reporter: Michael Gummelt > > Spark properties set on {{spark-submit}} via the deprecated > {{SPARK_JAVA_OPTS}} are passed along to the driver, but those set via the > preferred {{--conf}} are not. > For example, this results in the URI being fetched in the executor: > {{SPARK_JAVA_OPTS="-Dspark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md > -Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" > ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 > --class org.apache.spark.examples.SparkPi > http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}} > This does not: > {{SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=mesosphere/spark:1.6.0" > ./bin/spark-submit --deploy-mode cluster --master mesos://10.0.78.140:7077 > --conf > spark.mesos.uris=https://raw.githubusercontent.com/mesosphere/spark/master/README.md > --class org.apache.spark.examples.SparkPi > http://downloads.mesosphere.com.s3.amazonaws.com/assets/spark/spark-examples_2.10-1.5.0.jar}} > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L369 > In the above line of code, you can see that SPARK_JAVA_OPTS is passed along > to the driver, so those properties take effect. > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L373 > Whereas in this line of code, you see that {{--conf}} variables are set on > {{SPARK_EXECUTOR_OPTS}}, which AFAICT has absolutely no effect because this > env var is being set on the driver, not the executor. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org