Hi Team,
Is it ok to spawn multiple spark jobs within a main spark job, my main
spark job's driver which was launched on yarn cluster, will do some
preprocessing and based on it, it needs to launch multilple spark jobs on
yarn cluster. Not sure if this right pattern.
Please share your thoughts.
Sample code i ve is as below for better understanding..
---------------------
Object Mainsparkjob {
main(...){
val sc=new SparkContext(..)
Fetch from hive..using hivecontext
Fetch from hbase
//spawning multiple Futures..
Val future1=Future{
Val sparkjob= SparkLauncher(...).launch; spark.waitFor
}
Similarly, future2 to futureN.
future1.onComplete{...}
}
}// end of mainsparkjob
----------------------
