Here is an example dockerfile, although it's a bit dated now if you build it today it should still work:
https://github.com/tnachen/spark/tree/dockerfile/mesos_docker Tim On Thu, Mar 10, 2016 at 8:06 AM, Ashish Soni <asoni.le...@gmail.com> wrote: > Hi Tim , > > Can you please share your dockerfiles and configuration as it will help a > lot , I am planing to publish a blog post on the same . > > Ashish > > On Thu, Mar 10, 2016 at 10:34 AM, Timothy Chen <t...@mesosphere.io> wrote: > >> No you don't need to install spark on each slave, we have been running >> this setup in Mesosphere without any problem at this point, I think most >> likely configuration problem and perhaps a chance something is missing in >> the code to handle some cases. >> >> What version of spark are you guys running? And also did you set the >> working dir in your image to be spark home? >> >> Tim >> >> >> On Mar 10, 2016, at 3:11 AM, Ashish Soni <asoni.le...@gmail.com> wrote: >> >> You need to install spark on each mesos slave and then while starting >> container make a workdir to your spark home so that it can find the spark >> class. >> >> Ashish >> >> On Mar 10, 2016, at 5:22 AM, Guillaume Eynard Bontemps < >> g.eynard.bonte...@gmail.com> wrote: >> >> For an answer to my question see this: >> http://stackoverflow.com/a/35660466?noredirect=1. >> >> But for your problem did you define the Spark.mesos.docker. home or >> something like that property? >> >> Le jeu. 10 mars 2016 04:26, Eran Chinthaka Withana < >> eran.chinth...@gmail.com> a écrit : >> >>> Hi >>> >>> I'm also having this issue and can not get the tasks to work inside >>> mesos. >>> >>> In my case, the spark-submit command is the following. >>> >>> $SPARK_HOME/bin/spark-submit \ >>> --class com.mycompany.SparkStarter \ >>> --master mesos://mesos-dispatcher:7077 \ --name SparkStarterJob \ >>> --driver-memory 1G \ >>> --executor-memory 4G \ >>> --deploy-mode cluster \ >>> --total-executor-cores 1 \ >>> --conf >>> spark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6 \ >>> http://abc.com/spark-starter.jar >>> >>> >>> And the error I'm getting is the following. >>> >>> I0310 03:13:11.417009 131594 exec.cpp:132] Version: 0.23.1 >>> I0310 03:13:11.419452 131601 exec.cpp:206] Executor registered on slave >>> 20160223-000314-3439362570-5050-631-S0 >>> sh: 1: /usr/spark-1.6.0-bin-hadoop2.6/bin/spark-class: not found >>> >>> >>> (Looked into Spark JIRA and I found that >>> https://issues.apache.org/jira/browse/SPARK-11759 is marked as closed >>> since https://issues.apache.org/jira/browse/SPARK-12345 is marked as >>> resolved) >>> >>> Really appreciate if I can get some help here. >>> >>> Thanks, >>> Eran Chinthaka Withana >>> >>> On Wed, Feb 17, 2016 at 2:00 PM, g.eynard.bonte...@gmail.com < >>> g.eynard.bonte...@gmail.com> wrote: >>> >>>> Hi everybody, >>>> >>>> I am testing the use of Docker for executing Spark algorithms on MESOS. >>>> I >>>> managed to execute Spark in client mode with executors inside Docker, >>>> but I >>>> wanted to go further and have also my Driver running into a Docker >>>> Container. Here I ran into a behavior that I'm not sure is normal, let >>>> me >>>> try to explain. >>>> >>>> I submit my spark application through MesosClusterDispatcher using a >>>> command >>>> like: >>>> $ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master >>>> mesos://spark-master-1:7077 --deploy-mode cluster --conf >>>> spark.mesos.executor.docker.image=myuser/myimage:0.0.2 >>>> >>>> https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar >>>> 10 >>>> >>>> My driver is running fine, inside its docker container, but the >>>> executors >>>> fail: >>>> "sh: /some/spark/home/bin/spark-class: No such file or directory" >>>> >>>> Looking on MESOS slaves log, I think that the executors do not run >>>> inside >>>> docker: "docker.cpp:775] No container info found, skipping launch". As >>>> my >>>> Mesos slaves do not have spark installed, it fails. >>>> >>>> *It seems that the spark conf that I gave in the first spark-submit is >>>> not >>>> transmitted to the Driver submitted conf*, when launched in the docker >>>> container. The only workaround I found is to modify my Docker image in >>>> order >>>> to define inside its spark conf the spark.mesos.executor.docker.image >>>> property. This way, my executors get the conf well and are launched >>>> inside >>>> docker on Mesos. This seems a little complicated to me, and I feel the >>>> configuration passed to the early spark-submit should be transmitted to >>>> the >>>> Driver submit... >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com >>>> <http://nabble.com>. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >
