The default pool (`<pool name = "default">`) can be configured like any other pool: https://spark.apache.org/docs/latest/job-scheduling.html#configuring-pool-properties
On Thu, Sep 1, 2016 at 11:11 AM, enrico d'urso <e.du...@live.com> wrote: > Is there a way to force scheduling to be fair *inside* the default pool? > I mean, round robin for the jobs that belong to the default pool. > > Cheers, > ------------------------------ > *From:* Mark Hamstra <m...@clearstorydata.com> > *Sent:* Thursday, September 1, 2016 7:24:54 PM > *To:* enrico d'urso > *Cc:* user@spark.apache.org > *Subject:* Re: Spark scheduling mode > > Just because you've flipped spark.scheduler.mode to FAIR, that doesn't > mean that Spark can magically configure and start multiple scheduling pools > for you, nor can it know to which pools you want jobs assigned. Without > doing any setup of additional scheduling pools or assigning of jobs to > pools, you're just dumping all of your jobs into the one available default > pool (which is now being fair scheduled with an empty set of other pools) > and the scheduling of jobs within that pool is still the default intra-pool > scheduling, FIFO -- i.e., you've effectively accomplished nothing by only > flipping spark.scheduler.mode to FAIR. > > On Thu, Sep 1, 2016 at 7:10 AM, enrico d'urso <e.du...@live.com> wrote: > >> I am building a Spark App, in which I submit several jobs (pyspark). I am >> using threads to run them in parallel, and also I am setting: >> conf.set("spark.scheduler.mode", "FAIR") Still, I see the jobs run >> serially in FIFO way. Am I missing something? >> >> Cheers, >> >> >> Enrico >> > >