Thanks for echo on this. Possible to adjust resource based on container numbers? e.g to allocate more container when driver need more resources and return some resource by delete some container when parts of container already have enough cores/memory Best Regards Jun Feng Liu IBM China Systems & Technology Laboratory in Beijing
Phone: 86-10-82452683 E-mail: liuj...@cn.ibm.com BLD 28,ZGC Software Park No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 China Patrick Wendell <pwend...@gmail.com> 2014/08/08 13:10 To Jun Feng Liu/China/IBM@IBMCN, cc "dev@spark.apache.org" <dev@spark.apache.org> Subject Re: Fine-Grained Scheduler on Yarn Hey sorry about that - what I said was the opposite of what is true. The current YARN mode is equivalent to "coarse grained" mesos. There is no fine-grained scheduling on YARN at the moment. I'm not sure YARN supports scheduling in units other than containers. Fine-grained scheduling requires scheduling at the granularity of individual cores. On Thu, Aug 7, 2014 at 9:43 PM, Patrick Wendell <pwend...@gmail.com> wrote: The current YARN is equivalent to what is called "fine grained" mode in Mesos. The scheduling of tasks happens totally inside of the Spark driver. On Thu, Aug 7, 2014 at 7:50 PM, Jun Feng Liu <liuj...@cn.ibm.com> wrote: Any one know the answer? Best Regards Jun Feng Liu IBM China Systems & Technology Laboratory in Beijing Phone: 86-10-82452683 E-mail: liuj...@cn.ibm.com BLD 28,ZGC Software Park No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 China Jun Feng Liu/China/IBM 2014/08/07 15:37 To dev@spark.apache.org, cc Subject Fine-Grained Scheduler on Yarn Hi, there Just aware right now Spark only support fine grained scheduler on Mesos with MesosSchedulerBackend. The Yarn schedule sounds like only works on coarse-grained model. Is there any plan to implement fine-grained scheduler for YARN? Or there is any technical issue block us to do that. Best Regards Jun Feng Liu IBM China Systems & Technology Laboratory in Beijing Phone: 86-10-82452683 E-mail: liuj...@cn.ibm.com BLD 28,ZGC Software Park No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193 China
