Yes you can create an issue, or actually contribute a patch to update it :)
Sorry the docs is a bit light, I'm going to make it more complete along the way. Tim On Fri, Sep 11, 2015 at 11:11 AM, Tom Waterhouse (tomwater) < tomwa...@cisco.com> wrote: > Tim, > > Thank you for the explanation. You are correct, my Mesos experience is > very light, and I haven’t deployed anything via Marathon yet. What you > have stated here makes sense, I will look into doing this. > > Adding this info to the docs would be great. Is the appropriate action to > create an issue regarding improvement of the docs? For those of us who are > gaining the experience having such a pointer is very helpful. > > Tom > > From: Tim Chen <t...@mesosphere.io> > Date: Thursday, September 10, 2015 at 10:25 AM > To: Tom Waterhouse <tomwa...@cisco.com> > Cc: "user@spark.apache.org" <user@spark.apache.org> > Subject: Re: Spark on Mesos with Jobs in Cluster Mode Documentation > > Hi Tom, > > Sorry the documentation isn't really rich, since it's probably assuming > users understands how Mesos and framework works. > > First I need explain the rationale of why create the dispatcher. If you're > not familiar with Mesos yet, each node in your datacenter is installed a > Mesos slave where it's responsible for publishing resources and > running/watching tasks, and Mesos master is responsible for taking the > aggregated resources and scheduling them among frameworks. > > Frameworks are not managed by Mesos, as Mesos master/slave doesn't launch > and maintain framework but assume they're launched and kept running on its > own. All the existing frameworks in the ecosystem therefore all have their > own ways to deploy, HA and persist state (e.g: Aurora, Marathon, etc). > > Therefore, to introduce cluster mode with Mesos, we must create a > framework that is long running that can be running in your datacenter, and > can handle launching spark drivers on demand and handle HA, etc. This is > what the dispatcher is all about. > > So the idea is that you should launch the dispatcher not on the client, > but on a machine in your datacenter. In Mesosphere's DCOS we launch all > frameworks and long running services with Marathon, and you can use > Marathon to launch the Spark dispatcher. > > Then all clients instead of specifying the Mesos master URL (e.g: > mesos://mesos.master:2181), then just talks to the dispatcher only > (mesos://spark-dispatcher.mesos:7077), and the dispatcher will then start > and watch the driver for you. > > Tim > > > > On Thu, Sep 10, 2015 at 10:13 AM, Tom Waterhouse (tomwater) < > tomwa...@cisco.com> wrote: > >> After spending most of yesterday scouring the Internet for sources of >> documentation for submitting Spark jobs in cluster mode to a Spark cluster >> managed by Mesos I was able to do just that, but I am not convinced that >> how I have things setup is correct. >> >> I used the Mesos published >> <https://open.mesosphere.com/getting-started/datacenter/install/> >> instructions for setting up my Mesos cluster. I have three Zookeeper >> instances, three Mesos master instances, and three Mesos slave instances. >> This is all running in Openstack. >> >> The documentation on the Spark documentation site states that “To use >> cluster mode, you must start the MesosClusterDispatcher in your cluster via >> the sbin/start-mesos-dispatcher.sh script, passing in the Mesos master >> url (e.g: mesos://host:5050).” That is it, no more information than >> that. So that is what I did: I have one machine that I use as the Spark >> client for submitting jobs. I started the Mesos dispatcher with script as >> described, and using the client machine’s IP address and port as the target >> for the job submitted the job. >> >> The job is currently running in Mesos as expected. This is not however >> how I would have expected to configure the system. As running there is one >> instance of the Spark Mesos dispatcher running outside of Mesos, so not a >> part of the sphere of Mesos resource management. >> >> I used the following Stack Overflow posts as guidelines: >> http://stackoverflow.com/questions/31164725/spark-mesos-dispatcher >> http://stackoverflow.com/questions/31294515/start-spark-via-mesos >> >> There must be better documentation on how to deploy Spark in Mesos with >> jobs able to be deployed in cluster mode. >> >> I can follow up with more specific information regarding my deployment >> if necessary. >> >> Tom >> > >
