Yea, the idea is to use dynamic allocation. I can't speak to how well it works with Mesos, though.
On Mon, Dec 19, 2016 at 11:01 AM, Mehdi Meziane <mehdi.mezi...@ldmobile.net> wrote: > I think that what you are looking for is Dynamic resource allocation: > http://spark.apache.org/docs/latest/job-scheduling.html# > dynamic-resource-allocation > > Spark provides a mechanism to dynamically adjust the resources your > application occupies based on the workload. This means that your > application may give resources back to the cluster if they are no longer > used and request them again later when there is demand. This feature is > particularly useful if multiple applications share resources in your Spark > cluster. > > ----- Mail Original ----- > De: "Sumit Chawla" <sumitkcha...@gmail.com> > À: "Michael Gummelt" <mgumm...@mesosphere.io> > Cc: u...@mesos.apache.org, "Dev" <d...@mesos.apache.org>, "User" < > u...@spark.apache.org>, "dev" <dev@spark.apache.org> > Envoyé: Lundi 19 Décembre 2016 19h35:51 GMT +01:00 Amsterdam / Berlin / > Berne / Rome / Stockholm / Vienne > Objet: Re: Mesos Spark Fine Grained Execution - CPU count > > > But coarse grained does the exact same thing which i am trying to avert > here. At the cost of lower startup, it keeps the resources reserved till > the entire duration of the job. > > Regards > Sumit Chawla > > > On Mon, Dec 19, 2016 at 10:06 AM, Michael Gummelt <mgumm...@mesosphere.io> > wrote: > >> Hi >> >> I don't have a lot of experience with the fine-grained scheduler. It's >> deprecated and fairly old now. CPUs should be relinquished as tasks >> complete, so I'm not sure why you're seeing what you're seeing. There have >> been a few discussions on the spark list regarding deprecating the >> fine-grained scheduler, and no one seemed too dead-set on keeping it. I'd >> recommend you move over to coarse-grained. >> >> On Fri, Dec 16, 2016 at 8:41 AM, Chawla,Sumit <sumitkcha...@gmail.com> >> wrote: >> >>> Hi >>> >>> I am using Spark 1.6. I have one query about Fine Grained model in >>> Spark. I have a simple Spark application which transforms A -> B. Its a >>> single stage application. To begin the program, It starts with 48 >>> partitions. When the program starts running, in mesos UI it shows 48 tasks >>> and 48 CPUs allocated to job. Now as the tasks get done, the number of >>> active tasks number starts decreasing. How ever, the number of CPUs does >>> not decrease propotionally. When the job was about to finish, there was a >>> single remaininig task, however CPU count was still 20. >>> >>> My questions, is why there is no one to one mapping between tasks and >>> cpus in Fine grained? How can these CPUs be released when the job is done, >>> so that other jobs can start. >>> >>> >>> Regards >>> Sumit Chawla >>> >>> >> >> >> -- >> Michael Gummelt >> Software Engineer >> Mesosphere >> > > -- Michael Gummelt Software Engineer Mesosphere
