Hi Josh, If you want to cap the amount of memory per executor in Coarse grain mode, then yes you only get 240GB of memory as you mentioned. What's the reason you don't want to raise the capacity of memory you use per executor?
In coarse grain mode the Spark executor is long living and it internally will get tasks distributed by Spark internal Coarse grained scheduler. I think the assumption is that it already allocated the maximum available on that slave and don't really assume we need another one. I think it's worth considering having a configuration of number of cores per executor, especially when Mesos have inverse offers and optimistic offers so we can choose to launch more executors when resources becomes available even in coarse grain mode and then support giving the executors back but more higher priority tasks arrive. For fine grain mode, the spark executors are started by Mesos executors that is configured from Mesos scheduler backend. I believe the RDD is cached as long as the Mesos executor is running as the BlockManager is created on executor registration. Let me know if you need any more info. Tim > ---------- Forwarded message ---------- > From: Josh Devins <j...@soundcloud.com> > Date: 22 December 2014 at 17:23 > Subject: Mesos resource allocation > To: user@spark.apache.org > > > We are experimenting with running Spark on Mesos after running > successfully in Standalone mode for a few months. With the Standalone > resource manager (as well as YARN), you have the option to define the > number of cores, number of executors and memory per executor. In > Mesos, however, it appears as though you cannot specify the number of > executors, even in coarse-grained mode. If this is the case, how do > you define the number of executors to run with? > > Here's an example of why this matters (to us). Let's say we have the > following cluster: > > num nodes: 8 > num cores: 256 (32 per node) > total memory: 512GB (64GB per node) > > If I set my job to require 256 cores and per-executor-memory to 30GB, > then Mesos will schedule a single executor per machine (8 executors > total) and each executor will get 32 cores to work with. This means > that we have 8 executors * 32GB each for a total of 240G of cluster > memory in use — less than half of what is available. If you want > actually 16 executors in order to increase the amount of memory in use > across the cluster, how can you do this with Mesos? It seems that a > parameter is missing (or I haven't found it yet) which lets me tune > this for Mesos: > * number of executors per n-cores OR > * number of executors total > > Furthermore, in fine-grained mode in Mesos, how are the executors > started/allocated? That is, since Spark tasks map to Mesos tasks, when > and how are executors started? If they are transient and an executor > per task is created, does this mean we cannot have cached RDDs? > > Thanks for any advice or pointers, > > Josh >
