In coarse grain mode, the spark executors are launched and kept running while the scheduler is running. So if you have a spark shell launched and remained open, the executors are running and won't finish until the shell is exited.
In fine grain mode, the overhead time mostly comes from downloading the spark tar (if it's not already deployed in the slaves) and launching the spark executor. I suggest you try it out and look at the latency to see if it fits your use case or not. Tim On Wed, Jan 7, 2015 at 11:19 PM, Xuelin Cao <xuelincao2...@gmail.com> wrote: > > Hi, > > Thanks for the information. > > One more thing I want to clarify, when does Mesos or Yarn allocate > and release the resource? Aka, what is the resource life time? > > For example, in the stand-along mode, the resource is allocated when > the application is launched, resource released when the application > finishes. > > Then, it looks like, in the Mesos fine-grain mode, the resource is > allocated when the task is about to run; and released when the task > finishes. > > How about Mesos coarse-grain mode and Yarn mode? Is the resource > managed on the Job level? Aka, the resource life time equals the job life > time? Or on the stage level? > > One more question for the Mesos fine-grain mode. How is the overhead > of resource allocation and release? In MapReduce, a noticeable time is > spend on waiting the resource allocation. What is Mesos fine-grain mode? > > > > On Thu, Jan 8, 2015 at 3:07 PM, Tim Chen <t...@mesosphere.io> wrote: > >> Hi Xuelin, >> >> I can only speak about Mesos mode. There are two modes of management in >> Spark's Mesos scheduler, which are fine-grain mode and coarse-grain mode. >> >> In fine grain mode, each spark task launches one or more spark executors >> that only live through the life time of the task. So it's comparable to >> what you spoke about. >> >> In coarse grain mode it's going to support dynamic allocation of >> executors but that's being at a higher level than tasks. >> >> As for resource management recommendation, I think it's important to see >> what other applications you want to be running besides Spark in the same >> cluster and also your use cases, to see what resource management fits your >> need. >> >> Tim >> >> >> On Wed, Jan 7, 2015 at 10:55 PM, Xuelin Cao <xuelincao2...@gmail.com> >> wrote: >> >>> >>> Hi, >>> >>> Currently, we are building up a middle scale spark cluster (100 >>> nodes) in our company. One thing bothering us is, the how spark manages the >>> resource (CPU, memory). >>> >>> I know there are 3 resource management modes: stand-along, Mesos, >>> Yarn >>> >>> In the stand along mode, the cluster master simply allocates the >>> resource when the application is launched. In this mode, suppose an >>> engineer launches a spark-shell, claiming 100 CPU cores and 100G memory, >>> but doing nothing. But the cluster master simply allocates the resource to >>> this app even if the spark-shell does nothing. This is definitely not what >>> we want. >>> >>> What we want is, the resource is allocated when the actual task is >>> about to run. For example, in the map stage, the app may need 100 cores >>> because the RDD has 100 partitions, while in the reduce stage, only 20 >>> cores is needed because the RDD is shuffled into 20 partitions. >>> >>> I'm not very clear about the granularity of the spark resource >>> management. In the stand-along mode, the resource is allocated when the app >>> is launched. What about Mesos and Yarn? Can they support task level >>> resource management? >>> >>> And, what is the recommended mode for resource management? (Mesos? >>> Yarn?) >>> >>> Thanks >>> >>> >>> >> >
