For an answer to my question see this: http://stackoverflow.com/a/35660466?noredirect=1.
But for your problem did you define the Spark.mesos.docker. home or something like that property? Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana <eran.chinth...@gmail.com> a écrit : > Hi > > I'm also having this issue and can not get the tasks to work inside mesos. > > In my case, the spark-submit command is the following. > > $SPARK_HOME/bin/spark-submit \ > --class com.mycompany.SparkStarter \ > --master mesos://mesos-dispatcher:7077 \ --name SparkStarterJob \ > --driver-memory 1G \ > --executor-memory 4G \ > --deploy-mode cluster \ > --total-executor-cores 1 \ > --conf > spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \ > http://abc.com/spark-starter.jar > > > And the error I'm getting is the following. > > I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1 > I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave > 20160223-000314-3439362570-5050-631-S0 > sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found > > > (Looked into Spark JIRA and I found that > https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed > since https://issues.apache.org/jira/browse/SPARK-12345 is marked as > resolved) > > Really appreciate if I can get some help here. > > Thanks, > Eran Chinthaka Withana > > On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bonte...@gmail.com < > g.eynard.bonte...@gmail.com> wrote: > >> Hi everybody, >> >> I am testing the use of Docker for executing Spark algorithms on MESOS. I >> managed to execute Spark in client mode with executors inside Docker, but >> I >> wanted to go further and have also my Driver running into a Docker >> Container. Here I ran into a behavior that I'm not sure is normal, let me >> try to explain. >> >> I submit my spark application through MesosClusterDispatcher using a >> command >> like: >> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master >> mesos://spark-master-1:7077 --deploy-mode cluster --conf >> spark.mesos.executor.docker.image=myuser/myimage:0.0.2 >> >> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar >> 10 >> >> My driver is running fine, inside its docker container, but the executors >> fail: >> "sh: /some/spark/home/bin/spark-class: No such file or directory" >> >> Looking on MESOS slaves log, I think that the executors do not run inside >> docker: "docker.cpp:775] No container info found, skipping launch". As my >> Mesos slaves do not have spark installed, it fails. >> >> *It seems that the spark conf that I gave in the first spark-submit is not >> transmitted to the Driver submitted conf*, when launched in the docker >> container. The only workaround I found is to modify my Docker image in >> order >> to define inside its spark conf the spark.mesos.executor.docker.image >> property. This way, my executors get the conf well and are launched inside >> docker on Mesos. This seems a little complicated to me, and I feel the >> configuration passed to the early spark-submit should be transmitted to >> the >> Driver submit... >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
