That makes sense. From the documentation it looks like the executors are not supposed to terminate: http://spark.apache.org/docs/latest/running-on-mesos.html#fine-grained-deprecated
> Note that while Spark tasks in fine-grained will relinquish cores as they > terminate, they will not relinquish memory, as the JVM does not give memory > back to the Operating System. Neither will executors terminate when they’re > idle. I suppose your task to executor CPU ratio is low enough that it looks like most of the resources are not being reclaimed. If your tasks were using significantly more CPU the amortized cost of the idle executors would not be such a big deal. — *Joris Van Remoortere* Mesosphere On Mon, Dec 19, 2016 at 11:26 AM, Timothy Chen <tnac...@gmail.com> wrote: > Hi Chawla, > > One possible reason is that Mesos fine grain mode also takes up cores > to run the executor per host, so if you have 20 agents running Fine > grained executor it will take up 20 cores while it's still running. > > Tim > > On Fri, Dec 16, 2016 at 8:41 AM, Chawla,Sumit <sumitkcha...@gmail.com> > wrote: > > Hi > > > > I am using Spark 1.6. I have one query about Fine Grained model in Spark. > > I have a simple Spark application which transforms A -> B. Its a single > > stage application. To begin the program, It starts with 48 partitions. > > When the program starts running, in mesos UI it shows 48 tasks and 48 > CPUs > > allocated to job. Now as the tasks get done, the number of active tasks > > number starts decreasing. How ever, the number of CPUs does not decrease > > propotionally. When the job was about to finish, there was a single > > remaininig task, however CPU count was still 20. > > > > My questions, is why there is no one to one mapping between tasks and > cpus > > in Fine grained? How can these CPUs be released when the job is done, so > > that other jobs can start. > > > > > > Regards > > Sumit Chawla >
