Is there any reason you need a context on the application launching the jobs? You can use SparkLauncher in a normal app and just listen for state transitions
On Wed, 21 Dec 2016, 11:44 Naveen, <hadoopst...@gmail.com> wrote: > Hi Team, > > Thanks for your responses. > Let me give more details in a picture of how I am trying to launch jobs. > > Main spark job will launch other spark-job similar to calling multiple > spark-submit within a Spark driver program. > These spawned threads for new jobs will be totally different components, > so these cannot be implemented using spark actions. > > sample code: > > --------------------- > > Object Mainsparkjob { > > main(...){ > > val sc=new SparkContext(..) > > Fetch from hive..using hivecontext > Fetch from hbase > > //spawning multiple Futures.. > Val future1=Future{ > Val sparkjob= SparkLauncher(...).launch; spark.waitFor > } > > Similarly, future2 to futureN. > > future1.onComplete{...} > } > > }// end of mainsparkjob > ---------------------- > > > [image: Inline image 1] > > On Wed, Dec 21, 2016 at 3:13 PM, David Hodeffi < > david.hode...@niceactimize.com> wrote: > > I am not familiar of any problem with that. > > Anyway, If you run spark applicaction you would have multiple jobs, which > makes sense that it is not a problem. > > > > Thanks David. > > > > *From:* Naveen [mailto:hadoopst...@gmail.com] > *Sent:* Wednesday, December 21, 2016 9:18 AM > *To:* d...@spark.apache.org; user@spark.apache.org > *Subject:* Launching multiple spark jobs within a main spark job. > > > > Hi Team, > > > > Is it ok to spawn multiple spark jobs within a main spark job, my main > spark job's driver which was launched on yarn cluster, will do some > preprocessing and based on it, it needs to launch multilple spark jobs on > yarn cluster. Not sure if this right pattern. > > > > Please share your thoughts. > > Sample code i ve is as below for better understanding.. > > --------------------- > > > > Object Mainsparkjob { > > > > main(...){ > > > > val sc=new SparkContext(..) > > > > Fetch from hive..using hivecontext > > Fetch from hbase > > > > //spawning multiple Futures.. > > Val future1=Future{ > > Val sparkjob= SparkLauncher(...).launch; spark.waitFor > > } > > > > Similarly, future2 to futureN. > > > > future1.onComplete{...} > > } > > > > }// end of mainsparkjob > > ---------------------- > > > Confidentiality: This communication and any attachments are intended for > the above-named persons only and may be confidential and/or legally > privileged. Any opinions expressed in this communication are not > necessarily those of NICE Actimize. If this communication has come to you > in error you must take no action based on it, nor must you copy or show it > to anyone; please delete/destroy and inform the sender by e-mail > immediately. > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. > Viruses: Although we have taken steps toward ensuring that this e-mail and > attachments are free from any virus, we advise that in keeping with good > computing practice the recipient should ensure they are actually virus free. > > >