[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059937#comment-15059937 ]
Saisai Shao commented on SPARK-12345: ------------------------------------- I think by default Spark Mesos implementation will ship all the environment variables to the remote nodes, which includes {{SPARK_HOME}}, and Mesos itself will invoke the spark application through scripts, and inside the scripts we honor if {{SPARK_HOME}} is already set, so that's the problem. Basically, I think there're two sides we could fix: 1. We should not expose {{SPARK_HOME}} to the environment if it is not set specifically. Otherwise cases like here will potentially have problem. 2. Spark on Mesos should not blindly ship all the environment variables to the remote side. The best way for Spark on Mesos is to invoke the Java program like what YARN did currently, not rely on scripts. > Mesos cluster mode is broken > ---------------------------- > > Key: SPARK-12345 > URL: https://issues.apache.org/jira/browse/SPARK-12345 > Project: Spark > Issue Type: Bug > Components: Mesos > Affects Versions: 1.6.0 > Reporter: Andrew Or > Assignee: Apache Spark > Priority: Critical > > The same setup worked in 1.5.2 but is now failing for 1.6.0-RC2. > The driver is confused about where SPARK_HOME is. It resolves > `mesos.executor.uri` or `spark.mesos.executor.home` relative to the > filesystem where the driver runs, which is wrong. > {code} > I1215 15:00:39.411212 28032 exec.cpp:134] Version: 0.25.0 > I1215 15:00:39.413512 28037 exec.cpp:208] Executor registered on slave > 130bdc39-44e7-4256-8c22-602040d337f1-S1 > bin/spark-submit: line 27: > /Users/dragos/workspace/Spark/dev/rc-tests/spark-1.6.0-bin-hadoop2.6/bin/spark-class: > No such file or directory > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org