Re: Spark on Mesos with Jobs in Cluster Mode Documentation

That could be the behavior but spark.mesos.executor.home being unset still raises an exception inside the dispatcher preventing a docker from even being started. I can see if other properties are inherited from the default environment when that's set, if you'd like. I think the main problem is

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

You can still provide properties through the docker container by putting configuration in the conf directory, but we try to pass all properties submitted from the driver spark-submit through which I believe will override the defaults. This is not what you are seeing? Tim > On Sep 19, 2015,

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

I guess I need a bit more clarification, what kind of assumptions was the dispatcher making? Tim On Thu, Sep 17, 2015 at 10:18 PM, Alan Braithwaite wrote: > Hi Tim, > > Thanks for the follow up. It's not so much that I expect the executor to > inherit the configuration

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

One other piece of information: We're using zookeeper for persistence and when we brought the dispatcher back online, it crashed on the same exception after loading the config from zookeeper. Cheers, - Alan On Thu, Sep 17, 2015 at 12:29 PM, Alan Braithwaite wrote: > Hey

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

Hey All, To bump this thread once again, I'm having some trouble using the dispatcher as well. I'm using Mesos Cluster Manager with Docker Executors. I've deployed the dispatcher as Marathon job. When I submit a job using spark submit, the dispatcher writes back that the submission was

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

Small update: I found properties-file spark-submit parameter by reading the code and that seems to work, but appears to be undocumented in the submitting applications doc page. - Alan On Thu, Sep 17, 2015 at 12:39 PM, Alan Braithwaite wrote: > One other piece of

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

Hi Alan, If I understand correctly, you are setting executor home when you launch the dispatcher and not on the configuration when you submit job, and expect it to inherit that configuration? When I worked on the dispatcher I was assuming all configuration is passed to the dispatcher to

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

Hi Tim, Thanks for the follow up. It's not so much that I expect the executor to inherit the configuration of the dispatcher as I* don't *expect the dispatcher to make assumptions about the system environment of the executor (since it lives in a docker). I could potentially see a case where you

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

Yes you can create an issue, or actually contribute a patch to update it :) Sorry the docs is a bit light, I'm going to make it more complete along the way. Tim On Fri, Sep 11, 2015 at 11:11 AM, Tom Waterhouse (tomwater) < tomwa...@cisco.com> wrote: > Tim, > > Thank you for the explanation.

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

Tim, Thank you for the explanation. You are correct, my Mesos experience is very light, and I haven’t deployed anything via Marathon yet. What you have stated here makes sense, I will look into doing this. Adding this info to the docs would be great. Is the appropriate action to create an

Re: Spark on Mesos with Jobs in Cluster Mode Documentation

Hi Tom, Sorry the documentation isn't really rich, since it's probably assuming users understands how Mesos and framework works. First I need explain the rationale of why create the dispatcher. If you're not familiar with Mesos yet, each node in your datacenter is installed a Mesos slave where

Spark on Mesos with Jobs in Cluster Mode Documentation

After spending most of yesterday scouring the Internet for sources of documentation for submitting Spark jobs in cluster mode to a Spark cluster managed by Mesos I was able to do just that, but I am not convinced that how I have things setup is correct. I used the Mesos