Hi Xuelin, Spark 1.2 includes a "dynamic allocation" feature that allows Spark on YARN to modulate its YARN resource consumption as the demands of the application grow and shrink. This is somewhat coarser than what you call task-level resource management. Elasticity comes through allocating and releasing executors, not through requesting resources from YARN for individual tasks. It would be good to add finer-grained task-level elasticity as well, but this will rely on some YARN work (YARN-1197) for changing the resource allocation of a running container.
Mesos has a "fine-grained" mode similar to what you're wondering about. It's documented here: https://spark.apache.org/docs/latest/running-on-mesos.html#mesos-run-modes. -Sandy On Wed, Jan 7, 2015 at 10:55 PM, Xuelin Cao <xuelincao2...@gmail.com> wrote: > > Hi, > > Currently, we are building up a middle scale spark cluster (100 > nodes) in our company. One thing bothering us is, the how spark manages the > resource (CPU, memory). > > I know there are 3 resource management modes: stand-along, Mesos, Yarn > > In the stand along mode, the cluster master simply allocates the > resource when the application is launched. In this mode, suppose an > engineer launches a spark-shell, claiming 100 CPU cores and 100G memory, > but doing nothing. But the cluster master simply allocates the resource to > this app even if the spark-shell does nothing. This is definitely not what > we want. > > What we want is, the resource is allocated when the actual task is > about to run. For example, in the map stage, the app may need 100 cores > because the RDD has 100 partitions, while in the reduce stage, only 20 > cores is needed because the RDD is shuffled into 20 partitions. > > I'm not very clear about the granularity of the spark resource > management. In the stand-along mode, the resource is allocated when the app > is launched. What about Mesos and Yarn? Can they support task level > resource management? > > And, what is the recommended mode for resource management? (Mesos? > Yarn?) > > Thanks > > >
