Hi Sebastian, Yes, for fetching the details from Hive and HBase, I would want to use Spark's HiveContext etc. However, based on your point, I might have to check if JDBC based driver connection could be used to do the same.
Main reason for this is to avoid a client-server architecture design. If we go by a normal scala app without creating a sparkcontext as per your suggestion, then 1. it turns out to be a client program on cluster on a single node, and for any multiple invocation through xyz scheduler , it will be invoked always from that same node 2. Having client program on a single data node might create a hotspot for that data node which might create a bottleneck as all invocations might create JVMs on that node itself. 3. With above, we will loose the Spark on YARN's feature of dynamically allocating a driver on any available data node through RM and NM co-ordination. With YARN and Cluster mode of invoking a spark-job, it will help distribute multiple application(main one) in cluster uniformly. Thanks and please let me know your views. On Wed, Dec 21, 2016 at 5:43 PM, Sebastian Piu <sebastian....@gmail.com> wrote: > Is there any reason you need a context on the application launching the > jobs? > You can use SparkLauncher in a normal app and just listen for state > transitions > > On Wed, 21 Dec 2016, 11:44 Naveen, <hadoopst...@gmail.com> wrote: > >> Hi Team, >> >> Thanks for your responses. >> Let me give more details in a picture of how I am trying to launch jobs. >> >> Main spark job will launch other spark-job similar to calling multiple >> spark-submit within a Spark driver program. >> These spawned threads for new jobs will be totally different components, >> so these cannot be implemented using spark actions. >> >> sample code: >> >> --------------------- >> >> Object Mainsparkjob { >> >> main(...){ >> >> val sc=new SparkContext(..) >> >> Fetch from hive..using hivecontext >> Fetch from hbase >> >> //spawning multiple Futures.. >> Val future1=Future{ >> Val sparkjob= SparkLauncher(...).launch; spark.waitFor >> } >> >> Similarly, future2 to futureN. >> >> future1.onComplete{...} >> } >> >> }// end of mainsparkjob >> ---------------------- >> >> >> [image: Inline image 1] >> >> On Wed, Dec 21, 2016 at 3:13 PM, David Hodeffi < >> david.hode...@niceactimize.com> wrote: >> >> I am not familiar of any problem with that. >> >> Anyway, If you run spark applicaction you would have multiple jobs, which >> makes sense that it is not a problem. >> >> >> >> Thanks David. >> >> >> >> *From:* Naveen [mailto:hadoopst...@gmail.com] >> *Sent:* Wednesday, December 21, 2016 9:18 AM >> *To:* d...@spark.apache.org; user@spark.apache.org >> *Subject:* Launching multiple spark jobs within a main spark job. >> >> >> >> Hi Team, >> >> >> >> Is it ok to spawn multiple spark jobs within a main spark job, my main >> spark job's driver which was launched on yarn cluster, will do some >> preprocessing and based on it, it needs to launch multilple spark jobs on >> yarn cluster. Not sure if this right pattern. >> >> >> >> Please share your thoughts. >> >> Sample code i ve is as below for better understanding.. >> >> --------------------- >> >> >> >> Object Mainsparkjob { >> >> >> >> main(...){ >> >> >> >> val sc=new SparkContext(..) >> >> >> >> Fetch from hive..using hivecontext >> >> Fetch from hbase >> >> >> >> //spawning multiple Futures.. >> >> Val future1=Future{ >> >> Val sparkjob= SparkLauncher(...).launch; spark.waitFor >> >> } >> >> >> >> Similarly, future2 to futureN. >> >> >> >> future1.onComplete{...} >> >> } >> >> >> >> }// end of mainsparkjob >> >> ---------------------- >> >> >> Confidentiality: This communication and any attachments are intended for >> the above-named persons only and may be confidential and/or legally >> privileged. Any opinions expressed in this communication are not >> necessarily those of NICE Actimize. If this communication has come to you >> in error you must take no action based on it, nor must you copy or show it >> to anyone; please delete/destroy and inform the sender by e-mail >> immediately. >> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. >> Viruses: Although we have taken steps toward ensuring that this e-mail >> and attachments are free from any virus, we advise that in keeping with >> good computing practice the recipient should ensure they are actually virus >> free. >> >> >>