Can spark.scheduler.pool be applied globally ?

[Jeff Zhang]

It seems currently spark.scheduler.pool must be set as localProperties
(associate with thread). Any reason why spark.scheduler.pool can not be
used globally.  My scenario is that I want my thriftserver started with
fair scheduler as the default pool without using set command to set the
pool. Is there anyway to do that ? Or do I miss anything here ?

-- 
Best Regards

Jeff Zhang

Re: Can spark.scheduler.pool be applied globally ?

[Mark Hamstra]

I don't understand.  If you're using fair scheduling and don't set a pool,
the default pool will be used.

On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang  wrote:

>
> It seems currently spark.scheduler.pool must be set as localProperties
> (associate with thread). Any reason why spark.scheduler.pool can not be
> used globally.  My scenario is that I want my thriftserver started with
> fair scheduler as the default pool without using set command to set the
> pool. Is there anyway to do that ? Or do I miss anything here ?
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Can spark.scheduler.pool be applied globally ?

[Mark Hamstra]

The other way to do it is to build a custom version of Spark where you have
changed the value of DEFAULT_SCHEDULING_MODE -- and if you were paying
close attention, I accidentally let it slip that that is what I've done.  I
previously wrote "schedulingMode = DEFAULT_SCHEDULING_MODE -- i.e.
SchedulingMode.FAIR", but that should actually be SchedulingMode.FIFO if
you haven't changed the code:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L65

On Tue, Jan 5, 2016 at 5:29 PM, Jeff Zhang  wrote:

> Right, I can override the root pool in configuration file, Thanks Mark.
>
> On Wed, Jan 6, 2016 at 8:45 AM, Mark Hamstra 
> wrote:
>
>> Just configure  with
>> FAIR in fairscheduler.xml (or
>> in spark.scheduler.allocation.file if you have over-riden the default name
>> for the config file.)  `buildDefaultPool()` will only build the pool named
>> "default" with the default properties (such as schedulingMode =
>> DEFAULT_SCHEDULING_MODE -- i.e. SchedulingMode.FAIR) if that pool name is
>> not already built (
>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L90
>> ).
>>
>>
>> On Tue, Jan 5, 2016 at 4:15 PM, Jeff Zhang  wrote:
>>
>>> Sorry, I don't make it clearly. What I want is the default pool is fair
>>> scheduling. But seems if I want to use fair scheduling now, I have to set
>>> spark.scheduler.pool explicitly.
>>>
>>> On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra 
>>> wrote:
>>>
 I don't understand.  If you're using fair scheduling and don't set a
 pool, the default pool will be used.

 On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang  wrote:

>
> It seems currently spark.scheduler.pool must be set as localProperties
> (associate with thread). Any reason why spark.scheduler.pool can not be
> used globally.  My scenario is that I want my thriftserver started with
> fair scheduler as the default pool without using set command to set the
> pool. Is there anyway to do that ? Or do I miss anything here ?
>
> --
> Best Regards
>
> Jeff Zhang
>


>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Can spark.scheduler.pool be applied globally ?

[Jeff Zhang]

Thanks Mark, custom configuration file would be better for me. Changing
code will make it affect all the applications, this is too risky for me.



On Wed, Jan 6, 2016 at 10:50 AM, Mark Hamstra 
wrote:

> The other way to do it is to build a custom version of Spark where you
> have changed the value of DEFAULT_SCHEDULING_MODE -- and if you were
> paying close attention, I accidentally let it slip that that is what I've
> done.  I previously wrote "schedulingMode = DEFAULT_SCHEDULING_MODE --
> i.e. SchedulingMode.FAIR", but that should actually be SchedulingMode.FIFO
> if you haven't changed the code:
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L65
>
> On Tue, Jan 5, 2016 at 5:29 PM, Jeff Zhang  wrote:
>
>> Right, I can override the root pool in configuration file, Thanks Mark.
>>
>> On Wed, Jan 6, 2016 at 8:45 AM, Mark Hamstra 
>> wrote:
>>
>>> Just configure  with
>>> FAIR in fairscheduler.xml (or
>>> in spark.scheduler.allocation.file if you have over-riden the default name
>>> for the config file.)  `buildDefaultPool()` will only build the pool named
>>> "default" with the default properties (such as schedulingMode =
>>> DEFAULT_SCHEDULING_MODE -- i.e. SchedulingMode.FAIR) if that pool name is
>>> not already built (
>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L90
>>> ).
>>>
>>>
>>> On Tue, Jan 5, 2016 at 4:15 PM, Jeff Zhang  wrote:
>>>
 Sorry, I don't make it clearly. What I want is the default pool is fair
 scheduling. But seems if I want to use fair scheduling now, I have to set
 spark.scheduler.pool explicitly.

 On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra 
 wrote:

> I don't understand.  If you're using fair scheduling and don't set a
> pool, the default pool will be used.
>
> On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang  wrote:
>
>>
>> It seems currently spark.scheduler.pool must be set as
>> localProperties (associate with thread). Any reason why
>> spark.scheduler.pool can not be used globally.  My scenario is that I 
>> want
>> my thriftserver started with fair scheduler as the default pool without
>> using set command to set the pool. Is there anyway to do that ? Or do I
>> miss anything here ?
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>


 --
 Best Regards

 Jeff Zhang

>>>
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>


-- 
Best Regards

Jeff Zhang

Re: Can spark.scheduler.pool be applied globally ?

[Jeff Zhang]

Right, I can override the root pool in configuration file, Thanks Mark.

On Wed, Jan 6, 2016 at 8:45 AM, Mark Hamstra 
wrote:

> Just configure  with
> FAIR in fairscheduler.xml (or
> in spark.scheduler.allocation.file if you have over-riden the default name
> for the config file.)  `buildDefaultPool()` will only build the pool named
> "default" with the default properties (such as schedulingMode =
> DEFAULT_SCHEDULING_MODE -- i.e. SchedulingMode.FAIR) if that pool name is
> not already built (
> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L90
> ).
>
>
> On Tue, Jan 5, 2016 at 4:15 PM, Jeff Zhang  wrote:
>
>> Sorry, I don't make it clearly. What I want is the default pool is fair
>> scheduling. But seems if I want to use fair scheduling now, I have to set
>> spark.scheduler.pool explicitly.
>>
>> On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra 
>> wrote:
>>
>>> I don't understand.  If you're using fair scheduling and don't set a
>>> pool, the default pool will be used.
>>>
>>> On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang  wrote:
>>>

 It seems currently spark.scheduler.pool must be set as localProperties
 (associate with thread). Any reason why spark.scheduler.pool can not be
 used globally.  My scenario is that I want my thriftserver started with
 fair scheduler as the default pool without using set command to set the
 pool. Is there anyway to do that ? Or do I miss anything here ?

 --
 Best Regards

 Jeff Zhang

>>>
>>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>


-- 
Best Regards

Jeff Zhang

Re: Can spark.scheduler.pool be applied globally ?

[Jeff Zhang]

Sorry, I don't make it clearly. What I want is the default pool is fair
scheduling. But seems if I want to use fair scheduling now, I have to set
spark.scheduler.pool explicitly.

On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra 
wrote:

> I don't understand.  If you're using fair scheduling and don't set a pool,
> the default pool will be used.
>
> On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang  wrote:
>
>>
>> It seems currently spark.scheduler.pool must be set as localProperties
>> (associate with thread). Any reason why spark.scheduler.pool can not be
>> used globally.  My scenario is that I want my thriftserver started with
>> fair scheduler as the default pool without using set command to set the
>> pool. Is there anyway to do that ? Or do I miss anything here ?
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>


-- 
Best Regards

Jeff Zhang

Re: Can spark.scheduler.pool be applied globally ?

[Mark Hamstra]

Just configure  with
FAIR in fairscheduler.xml (or
in spark.scheduler.allocation.file if you have over-riden the default name
for the config file.)  `buildDefaultPool()` will only build the pool named
"default" with the default properties (such as schedulingMode =
DEFAULT_SCHEDULING_MODE -- i.e. SchedulingMode.FAIR) if that pool name is
not already built (
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L90
).


On Tue, Jan 5, 2016 at 4:15 PM, Jeff Zhang  wrote:

> Sorry, I don't make it clearly. What I want is the default pool is fair
> scheduling. But seems if I want to use fair scheduling now, I have to set
> spark.scheduler.pool explicitly.
>
> On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra 
> wrote:
>
>> I don't understand.  If you're using fair scheduling and don't set a
>> pool, the default pool will be used.
>>
>> On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang  wrote:
>>
>>>
>>> It seems currently spark.scheduler.pool must be set as localProperties
>>> (associate with thread). Any reason why spark.scheduler.pool can not be
>>> used globally.  My scenario is that I want my thriftserver started with
>>> fair scheduler as the default pool without using set command to set the
>>> pool. Is there anyway to do that ? Or do I miss anything here ?
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>