Hi Team, Is it ok to spawn multiple spark jobs within a main spark job, my main spark job's driver which was launched on yarn cluster, will do some preprocessing and based on it, it needs to launch multilple spark jobs on yarn cluster. Not sure if this right pattern.
Please share your thoughts. Sample code i ve is as below for better understanding.. --------------------- Object Mainsparkjob { main(...){ val sc=new SparkContext(..) Fetch from hive..using hivecontext Fetch from hbase //spawning multiple Futures.. Val future1=Future{ Val sparkjob= SparkLauncher(...).launch; spark.waitFor } Similarly, future2 to futureN. future1.onComplete{...} } }// end of mainsparkjob ----------------------