Hi, I am not quite sure of your used case here, but I would use spark-submit and submit sequential jobs as steps to an EMR cluster.
Regards, Gourav On Wed, Feb 8, 2017 at 11:10 AM, Cosmin Posteuca <cosmin.poste...@gmail.com> wrote: > I tried to run some test on EMR on yarn cluster mode. > > I have a cluster with 16 cores(8 processors with 2 threads each). If i run > one job(use 5 core) takes 90 seconds, if i run 2 jobs simultaneous, both > finished in 170 seconds. If i run 3 jobs simultaneous, all three finished > in 240 seconds. > > If i run 6 jobs, i expect to first 3 jobs to finish simultaneous in 240 > seconds, and next 3 jobs finish in 480 seconds from cluster start time. But > that doesn’t happened. My firs job finished after 120 second, second > finished after 180 seconds, third finished after 240 second, the fourth and > the fifth finished simultaneous after 360 seconds, and the last finished > after 400 seconds. > > I expected to run in a FIFO mode, but that doesn’t happened. Seems to be a > combination of FIFO and FAIR. > > Is this the correct behavior of spark? > > Thank you! > > 2017-02-08 9:29 GMT+02:00 Gourav Sengupta <gourav.sengu...@gmail.com>: > >> Hi, >> >> Michael's answer will solve the problem in case you using only SQL based >> solution. >> >> Otherwise please refer to the wonderful details mentioned here >> https://spark.apache.org/docs/latest/job-scheduling.html. With EMR 5.3.0 >> released SPARK 2.1.0 is available in AWS. >> >> (note that there is an issue with using zeppelin in it and I have raised >> it as an issue to AWS and they are looking into it now) >> >> Regards, >> Gourav Sengupta >> >> On Tue, Feb 7, 2017 at 10:37 PM, Michael Segel <msegel_had...@hotmail.com >> > wrote: >> >>> Why couldn’t you use the spark thrift server? >>> >>> >>> On Feb 7, 2017, at 1:28 PM, Cosmin Posteuca <cosmin.poste...@gmail.com> >>> wrote: >>> >>> answer for Gourav Sengupta >>> >>> I want to use same spark application because i want to work as a FIFO >>> scheduler. My problem is that i have many jobs(not so big) and if i run an >>> application for every job my cluster will split resources as a FAIR >>> scheduler(it's what i observe, maybe i'm wrong) and exist the possibility >>> to create bottleneck effect. The start time isn't a problem for me, because >>> it isn't a real-time application. >>> >>> I need a business solution, that's the reason why i can't use code from >>> github. >>> >>> Thanks! >>> >>> 2017-02-07 19:55 GMT+02:00 Gourav Sengupta <gourav.sengu...@gmail.com>: >>> >>>> Hi, >>>> >>>> May I ask the reason for using the same spark application? Is it >>>> because of the time it takes in order to start a spark context? >>>> >>>> On another note you may want to look at the number of contributors in a >>>> github repo before choosing a solution. >>>> >>>> >>>> Regards, >>>> Gourav >>>> >>>> On Tue, Feb 7, 2017 at 5:26 PM, vincent gromakowski < >>>> vincent.gromakow...@gmail.com> wrote: >>>> >>>>> Spark jobserver or Livy server are the best options for pure technical >>>>> API. >>>>> If you want to publish business API you will probably have to build >>>>> you own app like the one I wrote a year ago >>>>> https://github.com/elppc/akka-spark-experiments >>>>> It combines Akka actors and a shared Spark context to serve concurrent >>>>> subsecond jobs >>>>> >>>>> >>>>> 2017-02-07 15:28 GMT+01:00 ayan guha <guha.a...@gmail.com>: >>>>> >>>>>> I think you are loking for livy or spark jobserver >>>>>> >>>>>> On Wed, 8 Feb 2017 at 12:37 am, Cosmin Posteuca < >>>>>> cosmin.poste...@gmail.com> wrote: >>>>>> >>>>>>> I want to run different jobs on demand with same spark context, but >>>>>>> i don't know how exactly i can do this. >>>>>>> >>>>>>> I try to get current context, but seems it create a new spark >>>>>>> context(with new executors). >>>>>>> >>>>>>> I call spark-submit to add new jobs. >>>>>>> >>>>>>> I run code on Amazon EMR(3 instances, 4 core & 16GB ram / instance), >>>>>>> with yarn as resource manager. >>>>>>> >>>>>>> My code: >>>>>>> >>>>>>> val sparkContext = SparkContext.getOrCreate() >>>>>>> val content = 1 to 40000 >>>>>>> val result = sparkContext.parallelize(content, 5) >>>>>>> result.map(value => value.toString).foreach(loop) >>>>>>> >>>>>>> def loop(x: String): Unit = { >>>>>>> for (a <- 1 to 30000000) { >>>>>>> >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> spark-submit: >>>>>>> >>>>>>> spark-submit --executor-cores 1 \ >>>>>>> --executor-memory 1g \ >>>>>>> --driver-memory 1g \ >>>>>>> --master yarn \ >>>>>>> --deploy-mode cluster \ >>>>>>> --conf spark.dynamicAllocation.enabled=true \ >>>>>>> --conf spark.shuffle.service.enabled=true \ >>>>>>> --conf spark.dynamicAllocation.minExecutors=1 \ >>>>>>> --conf spark.dynamicAllocation.maxExecutors=3 \ >>>>>>> --conf spark.dynamicAllocation.initialExecutors=3 \ >>>>>>> --conf spark.executor.instances=3 \ >>>>>>> >>>>>>> If i run twice spark-submit it create 6 executors, but i want to run >>>>>>> all this jobs on same spark application. >>>>>>> >>>>>>> How can achieve adding jobs to an existing spark application? >>>>>>> >>>>>>> I don't understand why SparkContext.getOrCreate() don't get >>>>>>> existing spark context. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Cosmin P. >>>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Ayan Guha >>>>>> >>>>> >>>>> >>>> >>> >>> >> >
