So basically you have lots of small ML tasks you want to run concurrently? With "I've used repartition and cache to store the sub-datasets on only one machine" you mean that you reduced each RDD to have one partition only?
Maybe you want to give the fair scheduler a try to get more of your tasks executing concurrently. Just an idea... Regards, Jeff 2015-02-25 12:06 GMT+01:00 Staffan <staffan.arvids...@gmail.com>: > Hi, > Is there a good way (recommended way) to control and run multiple Spark > jobs > within the same application? My application is like follows; > > 1) Run one Spark job on a 'ful' dataset, which then creates a few thousands > of RDDs containing sub-datasets from the complete dataset. Each of the > sub-datasets are independent from the others (the 'ful' dataset is simply a > dump from a database containing several different types of records). > 2) Run some filtration and manipulations on each of the RDD and finally do > some ML on the data. (Each of the created RDD's from step 1) is completely > independent so this should be run concurrently). > > I've implemented this by using Scala Futures and executing the Spark jobs > in > 2) from a separate thread for each RDD. This works and improves runtime > compared to a naive for-loop over step 2). Scaling is however not as good > as > I would expect it to be. (28 minutes for 4 cores on 1 machine -> 19 minutes > for 12 cores on 3 machines). > > Each of the sub-datasets are fairly small so I've used 'repartition' and > 'cache' to store the sub-datasets on only one machine in step 1), this > improved runtime a few %. > > So, either do anyone have a suggestion of how to do this in a better way or > perhaps if there a higher level workflow tool that I can use on top of > Spark? (The cool solution would have been to use nestled RDDs and just map > over them in a high level way, but as this is not supported afaik). > > Thanks! > Staffan > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-efficiently-control-concurrent-Spark-jobs-tp21800.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
