Re: Problem mixing MESOS Cluster Mode and Docker task execution

When you say driver running on mesos can you explain how are you doing that...?? > On Mar 10, 2016, at 4:44 PM, Eran Chinthaka Withana > wrote: > > Yanling I'm already running the driver on mesos (through docker). FYI, I'm > running this on cluster mode with

Re: Problem mixing MESOS Cluster Mode and Docker task execution

Yanling I'm already running the driver on mesos (through docker). FYI, I'm running this on cluster mode with MesosClusterDispatcher. Mac (client) > MesosClusterDispatcher > Driver running on Mesos --> Workers running on Mesos My next step is to run MesosClusterDispatcher in mesos through

Re: Problem mixing MESOS Cluster Mode and Docker task execution

Hi Everyone, a quick question with in this context. What is the underneath persistent storage that you guys are using? With regards to this containerized environment? Thanks On Thursday, March 10, 2016, yanlin wang wrote: > How you guys make driver docker within container to be

Re: Problem mixing MESOS Cluster Mode and Docker task execution

How you guys make driver docker within container to be reachable from spark worker ? Would you share your driver docker? i am trying to put only driver in docker and spark running with yarn outside of container and i don’t want to use —net=host Thx Yanlin > On Mar 10, 2016, at 11:06 AM,

Re: Problem mixing MESOS Cluster Mode and Docker task execution

Glad to hear it. Thanks all for sharing your solutions. Le jeu. 10 mars 2016 19:19, Eran Chinthaka Withana a écrit : > Phew, it worked. All I had to do was to add *export > SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6"

Re: Problem mixing MESOS Cluster Mode and Docker task execution

Phew, it worked. All I had to do was to add *export SPARK_JAVA_OPTS="-Dspark.mesos.executor.docker.image=echinthaka/mesos-spark:0.23.1-1.6.0-2.6" *before calling spark-submit. Guillaume, thanks for the pointer. Timothy, thanks for looking into this. Looking forward to see a fix soon. Thanks,

Re: Problem mixing MESOS Cluster Mode and Docker task execution

Hi Timothy What version of spark are you guys running? > I'm using Spark 1.6.0. You can see the Dockerfile I used here: https://github.com/echinthaka/spark-mesos-docker/blob/master/docker/mesos-spark/Dockerfile > And also did you set the working dir in your image to be spark home? > Yes I

Re: Problem mixing MESOS Cluster Mode and Docker task execution

Hi Eran, I need to investigate but perhaps that's true, we're using SPARK_JAVA_OPTS to pass all the options and not --conf. I'll take a look at the bug, but if you can try the workaround and see if that fixes your problem. Tim On Thu, Mar 10, 2016 at 10:08 AM, Eran Chinthaka Withana <

Re: Problem mixing MESOS Cluster Mode and Docker task execution

t; My driver is running fine, inside its docker container, but the >>>> executors >>>> fail: >>>> "sh: /some/spark/home/bin/spark-class: No such file or directory" >>>> >>>> Looking on MESOS slaves log, I think that the executors

Re: Problem mixing MESOS Cluster Mode and Docker task execution

o not have spark installed, it fails. >>> >>> *It seems that the spark conf that I gave in the first spark-submit is >>> not >>> transmitted to the Driver submitted conf*, when launched in the docker >>> container. The only workaround I found is

Re: Problem mixing MESOS Cluster Mode and Docker task execution

ot have spark installed, it fails. >>>> >>>> *It seems that the spark conf that I gave in the first spark-submit is not >>>> transmitted to the Driver submitted conf*, when launched in the docker >>>> container. The only workaround I found is to modify m

Re: Problem mixing MESOS Cluster Mode and Docker task execution

aunched in the docker >>> container. The only workaround I found is to modify my Docker image in order >>> to define inside its spark conf the spark.mesos.executor.docker.image >>> property. This way, my executors get the conf well and are launched inside &g

Re: Problem mixing MESOS Cluster Mode and Docker task execution

transmitted to the Driver submitted conf*, when launched in the docker >> container. The only workaround I found is to modify my Docker image in >> order >> to define inside its spark conf the spark.mesos.executor.docker.image >> property. This way, my executors get the conf well

Re: Problem mixing MESOS Cluster Mode and Docker task execution

ll and are launched inside > docker on Mesos. This seems a little complicated to me, and I feel the > configuration passed to the early spark-submit should be transmitted to the > Driver submit... > > > > -- > View this message in context: > http://apache-spark-user-list.10015

Problem mixing MESOS Cluster Mode and Docker task execution

passed to the early spark-submit should be transmitted to the Driver submit... -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-mixing-MESOS-Cluster-Mode-and-Docker-task-execution-tp26258.html Sent from the Apache Spark User List mailing list a