Hi Pradeep, Yes we still have a pending PR that will start propagating these settings down to the executors, right now it's only applied on the driver. As a work around you can download or set spark.mesos.executor.docker.image in the spark-default.conf file in the docker image you use to launch the driver and it should automatically get this setting when the driver is launched.
Tim On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri < pradeep.chhetr...@gmail.com> wrote: > Hello Timothy, > > I am setting spark.mesos.executor.docker.image. In my case, the driver is > actually started as a docker container (SparkPi in screenshot) but the > tasks which are spawned by driver are not starting as containers but plain > java processes. Is this expected ? > > Thanks > > On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <t...@mesosphere.io> wrote: > >> You can launch the driver and executor in docker containers as well by >> setting spark.mesos.executor.docker.image to the image you want to use to >> launch them. >> >> Tim >> >> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com> >> wrote: >> >> Pradeep, >> >> You can mount a spark directory as a volume. This means you have to have >> spark deployed on every agent. >> >> Another thing you can do, place spark in hdfs, assuming that you have >> hdfs available but that too will download a copy to the sandbox. >> >> I'd prefer the former. >> >> Sent from Outlook Mobile <https://aka.ms/qtex0l> >> >> _____________________________ >> From: Pradeep Chhetri <pradeep.chhetr...@gmail.com> >> Sent: Tuesday, March 15, 2016 4:41 pm >> Subject: Apache Spark Over Mesos >> To: <user@mesos.apache.org> >> >> >> Hello, >> >> I am able to run Apache Spark over Mesos. Its quite simple to run Spark >> Dispatcher over marathon and ask it to run Spark Executor (I guess also can >> be called as Spark Driver) as docker container. >> >> I have a query regarding this: >> >> All spark tasks are spawned directly by first downloading the spark >> artifacts. I was thinking if there is some way I can start them too as >> docker containers. This will save the time for downloading the spark >> artifacts. I am running spark in fine-grained mode. >> >> I have attached a screenshot of a sample job >> >> <Screen Shot 2016-03-15 at 15.15.06.png> >> >> Thanks, >> >> -- >> Pradeep Chhetri >> >> >> > > > -- > Pradeep Chhetri >
