I tried to run some test on EMR on yarn cluster mode. I have a cluster with 16 cores(8 processors with 2 threads each). If i run one job(use 5 core) takes 90 seconds, if i run 2 jobs simultaneous, both finished in 170 seconds. If i run 3 jobs simultaneous, all three finished in 240 seconds.
If i run 6 jobs, i expect to first 3 jobs to finish simultaneous in 240 seconds, and next 3 jobs finish in 480 seconds from cluster start time. But that doesn’t happened. My firs job finished after 120 second, second finished after 180 seconds, third finished after 240 second, the fourth and the fifth finished simultaneous after 360 seconds, and the last finished after 400 seconds. I expected to run in a FIFO mode, but that doesn’t happened. Seems to be a combination of FIFO and FAIR. Is this the correct behavior of spark? Thank you! 2017-02-08 9:29 GMT+02:00 Gourav Sengupta <gourav.sengu...@gmail.com>: > Hi, > > Michael's answer will solve the problem in case you using only SQL based > solution. > > Otherwise please refer to the wonderful details mentioned here > https://spark.apache.org/docs/latest/job-scheduling.html. With EMR 5.3.0 > released SPARK 2.1.0 is available in AWS. > > (note that there is an issue with using zeppelin in it and I have raised > it as an issue to AWS and they are looking into it now) > > Regards, > Gourav Sengupta > > On Tue, Feb 7, 2017 at 10:37 PM, Michael Segel <msegel_had...@hotmail.com> > wrote: > >> Why couldn’t you use the spark thrift server? >> >> >> On Feb 7, 2017, at 1:28 PM, Cosmin Posteuca <cosmin.poste...@gmail.com> >> wrote: >> >> answer for Gourav Sengupta >> >> I want to use same spark application because i want to work as a FIFO >> scheduler. My problem is that i have many jobs(not so big) and if i run an >> application for every job my cluster will split resources as a FAIR >> scheduler(it's what i observe, maybe i'm wrong) and exist the possibility >> to create bottleneck effect. The start time isn't a problem for me, because >> it isn't a real-time application. >> >> I need a business solution, that's the reason why i can't use code from >> github. >> >> Thanks! >> >> 2017-02-07 19:55 GMT+02:00 Gourav Sengupta <gourav.sengu...@gmail.com>: >> >>> Hi, >>> >>> May I ask the reason for using the same spark application? Is it because >>> of the time it takes in order to start a spark context? >>> >>> On another note you may want to look at the number of contributors in a >>> github repo before choosing a solution. >>> >>> >>> Regards, >>> Gourav >>> >>> On Tue, Feb 7, 2017 at 5:26 PM, vincent gromakowski < >>> vincent.gromakow...@gmail.com> wrote: >>> >>>> Spark jobserver or Livy server are the best options for pure technical >>>> API. >>>> If you want to publish business API you will probably have to build you >>>> own app like the one I wrote a year ago https://github.com/elppc/akka- >>>> spark-experiments >>>> It combines Akka actors and a shared Spark context to serve concurrent >>>> subsecond jobs >>>> >>>> >>>> 2017-02-07 15:28 GMT+01:00 ayan guha <guha.a...@gmail.com>: >>>> >>>>> I think you are loking for livy or spark jobserver >>>>> >>>>> On Wed, 8 Feb 2017 at 12:37 am, Cosmin Posteuca < >>>>> cosmin.poste...@gmail.com> wrote: >>>>> >>>>>> I want to run different jobs on demand with same spark context, but i >>>>>> don't know how exactly i can do this. >>>>>> >>>>>> I try to get current context, but seems it create a new spark >>>>>> context(with new executors). >>>>>> >>>>>> I call spark-submit to add new jobs. >>>>>> >>>>>> I run code on Amazon EMR(3 instances, 4 core & 16GB ram / instance), >>>>>> with yarn as resource manager. >>>>>> >>>>>> My code: >>>>>> >>>>>> val sparkContext = SparkContext.getOrCreate() >>>>>> val content = 1 to 40000 >>>>>> val result = sparkContext.parallelize(content, 5) >>>>>> result.map(value => value.toString).foreach(loop) >>>>>> >>>>>> def loop(x: String): Unit = { >>>>>> for (a <- 1 to 30000000) { >>>>>> >>>>>> } >>>>>> } >>>>>> >>>>>> spark-submit: >>>>>> >>>>>> spark-submit --executor-cores 1 \ >>>>>> --executor-memory 1g \ >>>>>> --driver-memory 1g \ >>>>>> --master yarn \ >>>>>> --deploy-mode cluster \ >>>>>> --conf spark.dynamicAllocation.enabled=true \ >>>>>> --conf spark.shuffle.service.enabled=true \ >>>>>> --conf spark.dynamicAllocation.minExecutors=1 \ >>>>>> --conf spark.dynamicAllocation.maxExecutors=3 \ >>>>>> --conf spark.dynamicAllocation.initialExecutors=3 \ >>>>>> --conf spark.executor.instances=3 \ >>>>>> >>>>>> If i run twice spark-submit it create 6 executors, but i want to run >>>>>> all this jobs on same spark application. >>>>>> >>>>>> How can achieve adding jobs to an existing spark application? >>>>>> >>>>>> I don't understand why SparkContext.getOrCreate() don't get existing >>>>>> spark context. >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Cosmin P. >>>>>> >>>>> -- >>>>> Best Regards, >>>>> Ayan Guha >>>>> >>>> >>>> >>> >> >> >
