Hi Timothy,
Thanks for your feedback, I logged
https://issues.apache.org/jira/browse/SPARK-11143 to trace this issue.
If any more suggestions, please let me know :).
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://www.cguru.net
Subject: Re: How to enable Spark mesos docker executor?
From: t...@mesosphere.io
Date: Fri, 16 Oct 2015 10:11:36 +0800
CC: user@spark.apache.org
To: kl...@cguru.net
Hi Klaus,
Sorry not next to a computer but it could possibily be a bug that it doesn't
take SPARK_HOME as the base path. Currently the spark image seems to set the
working directory so that it works.
I'll look at the code to verify but seems like it could be the case. If it's
true feel free to create a JIRA and/or provide a fix.
Tim
On Oct 16, 2015, at 9:28 AM, Klaus Ma <kl...@cguru.net> wrote:
Hi team,
I'm working on integration between Mesos & Spark. For now, I can start
SlaveMesosDispatcher in a docker; and I like to also run Spark executor in
Mesos docker. I do the following configuration for it, but I got an error; any
suggestion?Configuration:Spark:
conf/spark-defaults.confspark.mesos.executor.docker.imageubuntu
spark.mesos.executor.docker.volumes
/usr/bin:/usr/bin,/usr/local/lib:/usr/local/lib,/usr/lib:/usr/lib,/lib:/lib,/home/test/workshop/spark:/root/spark
spark.mesos.executor.home/root/spark
#spark.executorEnv.SPARK_HOME /root/spark
spark.executorEnv.MESOS_NATIVE_LIBRARY /usr/local/lib
NOTE: The spark are installed in /home/test/workshop/spark, and all
dependencies are installed.After submit SparkPi to the dispatcher, the driver
job is started but failed. The error messes is:I1015 11:10:29.488456 18697
exec.cpp:134] Version: 0.26.0
I1015 11:10:29.506619 18699 exec.cpp:208] Executor registered on slave
b7e24114-7585-40bc-879b-6a1188cb65b6-S1
WARNING: Your kernel does not support swap limit capabilities, memory limited
without swap.
/bin/sh: 1: ./bin/spark-submit: not found
Does any know how to map/set spark home in docker for this case?
Da (Klaus), Ma (马达) | PMP® | Advisory Software Engineer
Platform Symphony/DCOS Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://www.cguru.net