TIm, sorry I am wrong above. The above config is from spark dispatcher container and those configuration is not being propagated to the driver.
I will try the workaround you told and let you know how it goes. Thanks On Tue, Mar 15, 2016 at 4:42 PM, Pradeep Chhetri < pradeep.chhetr...@gmail.com> wrote: > Hello Tim, > > Here is my conf/spark-defaults.conf which is inside the docker image: > > $ cat conf/spark-defaults.conf > > spark.mesos.coarse: false > spark.mesos.executor.docker.image: docker-registry/mesos-spark:master-12 > spark.mesos.mesosExecutor.cores: 0.25 > spark.mesos.executor.home: /opt/spark > spark.mesos.uris: file:///etc/docker.tar.gz > > I am already setting it inside the docker image. > > Am I missing something ? > > Regards, > > On Tue, Mar 15, 2016 at 4:37 PM, Tim Chen <t...@mesosphere.io> wrote: > >> Hi Pradeep, >> >> Yes we still have a pending PR that will start propagating these settings >> down to the executors, right now it's only applied on the driver. As a work >> around you can download or set spark.mesos.executor.docker.image in the >> spark-default.conf file in the docker image you use to launch the driver >> and it should automatically get this setting when the driver is launched. >> >> Tim >> >> On Tue, Mar 15, 2016 at 9:26 AM, Pradeep Chhetri < >> pradeep.chhetr...@gmail.com> wrote: >> >>> Hello Timothy, >>> >>> I am setting spark.mesos.executor.docker.image. In my case, the driver >>> is actually started as a docker container (SparkPi in screenshot) but the >>> tasks which are spawned by driver are not starting as containers but plain >>> java processes. Is this expected ? >>> >>> Thanks >>> >>> On Tue, Mar 15, 2016 at 4:19 PM, Timothy Chen <t...@mesosphere.io> wrote: >>> >>>> You can launch the driver and executor in docker containers as well by >>>> setting spark.mesos.executor.docker.image to the image you want to use to >>>> launch them. >>>> >>>> Tim >>>> >>>> On Mar 15, 2016, at 8:49 AM, Radoslaw Gruchalski <ra...@gruchalski.com> >>>> wrote: >>>> >>>> Pradeep, >>>> >>>> You can mount a spark directory as a volume. This means you have to >>>> have spark deployed on every agent. >>>> >>>> Another thing you can do, place spark in hdfs, assuming that you have >>>> hdfs available but that too will download a copy to the sandbox. >>>> >>>> I'd prefer the former. >>>> >>>> Sent from Outlook Mobile <https://aka.ms/qtex0l> >>>> >>>> _____________________________ >>>> From: Pradeep Chhetri <pradeep.chhetr...@gmail.com> >>>> Sent: Tuesday, March 15, 2016 4:41 pm >>>> Subject: Apache Spark Over Mesos >>>> To: <user@mesos.apache.org> >>>> >>>> >>>> Hello, >>>> >>>> I am able to run Apache Spark over Mesos. Its quite simple to run Spark >>>> Dispatcher over marathon and ask it to run Spark Executor (I guess also can >>>> be called as Spark Driver) as docker container. >>>> >>>> I have a query regarding this: >>>> >>>> All spark tasks are spawned directly by first downloading the spark >>>> artifacts. I was thinking if there is some way I can start them too as >>>> docker containers. This will save the time for downloading the spark >>>> artifacts. I am running spark in fine-grained mode. >>>> >>>> I have attached a screenshot of a sample job >>>> >>>> <Screen Shot 2016-03-15 at 15.15.06.png> >>>> >>>> Thanks, >>>> >>>> -- >>>> Pradeep Chhetri >>>> >>>> >>>> >>> >>> >>> -- >>> Pradeep Chhetri >>> >> >> > > > -- > Pradeep Chhetri > -- Pradeep Chhetri
