How can I configure Mesos allocation policy to share resources between all
current Spark applications? I can't seem to find it in the architecture
docs.
*Romi Kuntsman*, *Big Data Engineer*
http://www.totango.com
On Tue, Nov 4, 2014 at 9:11 AM, Akhil Das ak...@sigmoidanalytics.com
wrote:
Yes.
You can look at different modes over here
http://docs.sigmoidanalytics.com/index.php/Spark_On_Mesos#Mesos_Run_Modes
These people has very good tutorial to get you started
http://mesosphere.com/docs/tutorials/run-spark-on-mesos/#overview
Thanks
Best Regards
On Tue, Nov 4, 2014 at 1:44 PM, Romi
I have a single Spark cluster, not multiple frameworks and not multiple
versions. Is it relevant for my use-case?
Where can I find information about exactly how to make Mesos tell Spark how
many resources of the cluster to use? (instead of the default take-all)
*Romi Kuntsman*, *Big Data
You need to install mesos on your cluster. Then you will run your spark
applications by specifying mesos master (mesos://) instead of (spark://).
Spark can run over Mesos in two modes: “*fine-grained*” (default) and “
*coarse-grained*”.
In “*fine-grained*” mode (default), each Spark task runs as
Let's say that I run Spark on Mesos in fine-grained mode, and I have 12
cores and 64GB memory.
I run application A on Spark, and some time after that (but before A
finished) application B.
How many CPUs will each of them get?
*Romi Kuntsman*, *Big Data Engineer*
http://www.totango.com
On Tue,
Have a look at scheduling pools
https://spark.apache.org/docs/latest/job-scheduling.html. If you want
more sophisticated resource allocation, then you are better of to use
cluster managers like mesos or yarn
Thanks
Best Regards
On Mon, Nov 3, 2014 at 9:10 PM, Romi Kuntsman r...@totango.com
So, as said there, static partitioning is used in Spark’s standalone and
YARN modes, as well as the coarse-grained Mesos mode.
That leaves us only with Mesos, where there is *dynamic sharing* of CPU
cores.
It says when the application is not running tasks on a machine, other
applications may run
Yes. i believe Mesos is the right choice for you.
http://mesos.apache.org/documentation/latest/mesos-architecture/
Thanks
Best Regards
On Mon, Nov 3, 2014 at 9:33 PM, Romi Kuntsman r...@totango.com wrote:
So, as said there, static partitioning is used in Spark’s standalone and
YARN modes, as
