I filed a JIRA for this task for future reference.
https://issues.apache.org/jira/browse/SPARK-2780
On Thu, Jul 31, 2014 at 5:37 PM, Tathagata Das
wrote:
> Whoa! That worked! I was half afraid it wont, since I hadnt tried it myself.
>
> TD
>
> On Wed, Jul 30, 2014 at 8:32 PM, liuwei wrote:
>> H
Whoa! That worked! I was half afraid it wont, since I hadnt tried it myself.
TD
On Wed, Jul 30, 2014 at 8:32 PM, liuwei wrote:
> Hi, Tathagata Das:
>
> I followed your advice and solved this problem, thank you very much!
>
>
> 在 2014年7月31日，上午3:07，Tathagata Das 写道：
>
>> This is because set
Hi, Tathagata Das:
I followed your advice and solved this problem, thank you very much!
在 2014年7月31日，上午3:07，Tathagata Das 写道：
> This is because setLocalProperty makes all Spark jobs submitted using
> the current thread belong to the set pool. However, in Spark
> Streaming, all the jobs a
This is because setLocalProperty makes all Spark jobs submitted using
the current thread belong to the set pool. However, in Spark
Streaming, all the jobs are actually launched in the background from a
different thread. So this setting does not work. However, there is a
work around. If you are doi
In my spark streaming program, I set scheduler pool, just as follows:
val myFairSchedulerFile = “xxx.xml”
val myStreamingPool = “xxx”
System.setProperty(“spark.scheduler.allocation.file”, myFairSchedulerFile)
val conf = new SparkConf()
val ssc = new StreamingContext(conf, batchInterval)
ssc.spark
