The other way to do it is to build a custom version of Spark where you have changed the value of DEFAULT_SCHEDULING_MODE -- and if you were paying close attention, I accidentally let it slip that that is what I've done. I previously wrote "schedulingMode = DEFAULT_SCHEDULING_MODE -- i.e. SchedulingMode.FAIR", but that should actually be SchedulingMode.FIFO if you haven't changed the code: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L65
On Tue, Jan 5, 2016 at 5:29 PM, Jeff Zhang <zjf...@gmail.com> wrote: > Right, I can override the root pool in configuration file, Thanks Mark. > > On Wed, Jan 6, 2016 at 8:45 AM, Mark Hamstra <m...@clearstorydata.com> > wrote: > >> Just configure <pool name="default"> with >> <schedulingMode>FAIR</schedulingMode> in fairscheduler.xml (or >> in spark.scheduler.allocation.file if you have over-riden the default name >> for the config file.) `buildDefaultPool()` will only build the pool named >> "default" with the default properties (such as schedulingMode = >> DEFAULT_SCHEDULING_MODE -- i.e. SchedulingMode.FAIR) if that pool name is >> not already built ( >> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L90 >> ). >> >> >> On Tue, Jan 5, 2016 at 4:15 PM, Jeff Zhang <zjf...@gmail.com> wrote: >> >>> Sorry, I don't make it clearly. What I want is the default pool is fair >>> scheduling. But seems if I want to use fair scheduling now, I have to set >>> spark.scheduler.pool explicitly. >>> >>> On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra <m...@clearstorydata.com> >>> wrote: >>> >>>> I don't understand. If you're using fair scheduling and don't set a >>>> pool, the default pool will be used. >>>> >>>> On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang <zjf...@gmail.com> wrote: >>>> >>>>> >>>>> It seems currently spark.scheduler.pool must be set as localProperties >>>>> (associate with thread). Any reason why spark.scheduler.pool can not be >>>>> used globally. My scenario is that I want my thriftserver started with >>>>> fair scheduler as the default pool without using set command to set the >>>>> pool. Is there anyway to do that ? Or do I miss anything here ? >>>>> >>>>> -- >>>>> Best Regards >>>>> >>>>> Jeff Zhang >>>>> >>>> >>>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >> >> > > > -- > Best Regards > > Jeff Zhang >
