Fwd: Assign Custom receiver to a scheduler pool
HI All, I have a spark streaming job, which is running fine. And in that job I am using FAIR scheduler with multiple pools. One pool for main streaming job. And another for some parallel batch processing . All is working as expected, except .. "The custom receiver that I used is not assigned to the pool used for streaming." I am using stream.foreachRDD.. { rdd => rdd.sparkContext.setLocalProperty("spark.scheduler.pool","streaming") } What I can understand is , I am setting the Thread local after Spark Custom Receiver is initialized. Receiver is running on default pool. Setting Receiver using : ssc.receiverStream(MyCustomReceiver()) Any Help ? Regards, Rabin Banerjee
Fwd: Assign Custom receiver to a scheduler pool
HI All, I have a spark streaming job, which is running fine. And in that job I am using FAIR scheduler with multiple pools. One pool for main streaming job. And another for some parallel batch processing . All is working as expected, except .. "The custom receiver that I used is not assigned to the pool used for streaming." I am using stream.foreachRDD.. { rdd => rdd.sparkContext.setLocalProperty("spark.scheduler.pool","streaming") } What I can understand is , I am setting the Thread local after Spark Custom Receiver is initialized. Receiver is running on default pool. Setting Receiver using : ssc.receiverStream(MyCustomReceiver()) Any Help ? Regards, Rabin Banerjee