In my spark streaming program, I set scheduler pool, just as follows: val myFairSchedulerFile = “xxx.xml” val myStreamingPool = “xxx”
System.setProperty(“spark.scheduler.allocation.file”, myFairSchedulerFile) val conf = new SparkConf() val ssc = new StreamingContext(conf, batchInterval) ssc.sparkContext.setLocalProperty(“spark.scheduler.pool”, myStreamingPool) …. ssc.start() ssc.awaitTermination() I submit my spark streaming job in my spark cluster, and I found stage’s pool name is “default”, it seem ssc.sparkContext.setLocalProperty(“spark.scheduler.pool”, myStreamingPool) not work.
