In cluster mode, you need to first run *MesosClusterDispatcher* application on marathon (Read more about that here: http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode)
In both client and cluster mode, you need to specify --master flag while submitting job, the only difference is that you will specifying the value as the URL of dispatcher in cluster mode (mesos://<dispatcher_address>:<dispatcher_port>) while in client mode, you will be specifying URL of mesos-master (mesos://<mesos_master>:<mesos_master_port>) On Wed, Apr 13, 2016 at 3:24 PM, June Taylor <j...@umn.edu> wrote: > I'm interested in what the "best practice" is for running pyspark jobs > against a mesos cluster. > > Right now, we're simply passing the --master mesos://host:5050 flag, which > appears to register a framework properly. > > However, I was told this isn't "cluster mode" - and I'm a bit confused. > What is the recommended method of doing this? > > Thanks, > June Taylor > System Administrator, Minnesota Population Center > University of Minnesota > -- Regards, Pradeep Chhetri
