Just because you've flipped spark.scheduler.mode to FAIR, that doesn't mean
that Spark can magically configure and start multiple scheduling pools for
you, nor can it know to which pools you want jobs assigned. Without doing
any setup of additional scheduling pools or assigning of jobs to pools,
you're just dumping all of your jobs into the one available default pool
(which is now being fair scheduled with an empty set of other pools) and
the scheduling of jobs within that pool is still the default intra-pool
scheduling, FIFO -- i.e., you've effectively accomplished nothing by only
flipping spark.scheduler.mode to FAIR.
On Thu, Sep 1, 2016 at 7:10 AM, enrico d'urso <e.du...@live.com> wrote:
> I am building a Spark App, in which I submit several jobs (pyspark). I am
> using threads to run them in parallel, and also I am setting:
> conf.set("spark.scheduler.mode", "FAIR") Still, I see the jobs run
> serially in FIFO way. Am I missing something?
>
> Cheers,
>
>
> Enrico
>
