You should try out TorrentBroadcast (NOT BitTorrentBroadcast) from the 0.8.1 branch. In your config file, set spark.broadcast.factory= org.apache.spark.broadcast.TorrentBroadcastFactory It should perform significantly better than HttpBroadcast (some benchmarks here: https://github.com/apache/incubator-spark/pull/68). I expect a 10X improvement over the default. Make sure you have enough memory in slaves.
-- Mosharaf Chowdhury http://www.mosharaf.com/ On Wed, Dec 25, 2013 at 7:32 PM, purav aggarwal <[email protected]>wrote: > Hi all, > > I have a large file ( > 5 gigs) which I need to lookup. Since each slave > need to perform the search operation on the hashmap (built out of the file) > in parallel I need to broadcast the file. I was wondering if broadcasting > such a huge file is really a good idea. Do we have any benchmarks for the > broadcast variables. I am on a Standalone cluster and machine configuration > is not a problem at the moment. > Has anyone exploited broadcast to such an extent ? > > Thanks, > Purav >
