The fair scheduler doesn't have anything to do with reallocating resource across Applications.
https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-across-applications https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application On Thu, Jul 20, 2017 at 2:02 PM, Gokula Krishnan D <email2...@gmail.com> wrote: > Mark, Thanks for the response. > > Let me rephrase my statements. > > "I am submitting a Spark application(*Application*#A) with scheduler.mode > as FAIR and dynamicallocation=true and it got all the available executors. > > In the meantime, submitting another Spark Application (*Application* # B) > with the scheduler.mode as FAIR and dynamicallocation=true but it got only > one executor. " > > Thanks & Regards, > Gokula Krishnan* (Gokul)* > > On Thu, Jul 20, 2017 at 4:56 PM, Mark Hamstra <m...@clearstorydata.com> > wrote: > >> First, Executors are not allocated to Jobs, but rather to Applications. >> If you run multiple Jobs within a single Application, then each of the >> Tasks associated with Stages of those Jobs has the potential to run on any >> of the Application's Executors. Second, once a Task starts running on an >> Executor, it has to complete before another Task can be scheduled using the >> prior Task's resources -- the fair scheduler is not preemptive of running >> Tasks. >> >> On Thu, Jul 20, 2017 at 1:45 PM, Gokula Krishnan D <email2...@gmail.com> >> wrote: >> >>> Hello All, >>> >>> We are having cluster with 50 Executors each with 4 Cores so can avail >>> max. 200 Executors. >>> >>> I am submitting a Spark application(JOB A) with scheduler.mode as FAIR >>> and dynamicallocation=true and it got all the available executors. >>> >>> In the meantime, submitting another Spark Application (JOB B) with the >>> scheduler.mode as FAIR and dynamicallocation=true but it got only one >>> executor. >>> >>> Normally this situation occurs when any of the JOB runs with the >>> Scheduler.mode= FIFO. >>> >>> 1) Have your ever faced this issue if so how to overcome this?. >>> >>> I was in the impression that as soon as I submit the JOB B the Spark >>> Scheduler should distribute/release few resources from the JOB A and share >>> it with the JOB A in the Round Robin fashion?. >>> >>> Appreciate your response !!!. >>> >>> >>> Thanks & Regards, >>> Gokula Krishnan* (Gokul)* >>> >> >> >
