Thanks Imran, I'll try your suggestions. I eventually got this to run by 'checkpointing' the joined RDD (according to Akhil's suggestion) before performing the reduceBy, and then checkpointing it again afterward. i.e.
> val rdd2 = rdd.join(rdd, numPartitions=1000) > .map(fp=>((fp._2._1, fp._2._2), 1)) > .persist(MEMORY_AND_DISK_SER) > val rdd3 = rdd2.reduceByKey((x,y)=>x+y).persist(MEMORY_AND_DISK_SER) > rdd3.count() It takes a while, but at least it runs. So, I'll be sure to try your suggestions for further speed-up. Thanks again for your help. On 18 February 2015 at 18:47, Imran Rashid <[email protected]> wrote: > Hi Tom, > > there are a couple of things you can do here to make this more efficient. > first, I think you can replace your self-join with a groupByKey. on your > example data set, this would give you > > (1, Iterable(2,3)) > (4, Iterable(3)) > > this reduces the amount of data that needs to be shuffled, and that way > you can produce all of your pairs just from the Iterable(2,3). > > second, if you expect the same pairs to appear many times in your dataset, > you might first want to replace them with a count. eg., if you start with > > (1,2) > (1,2) > (1,2) > ... > (1,2) > (1,3) > (1,3) > (4,3) > ... > > you might want to first convert that to get a count of each pair > > val pairCounts = rdd.map{x => (x,1)}.reduceByKey{_ + _} > > to give you something like: > > ((1,2), 145) > ((1,3), 2) > ((4,3), 982) > ... > > and then with a little more massaging you can group by key and also keep > the counts of each item: > > val groupedCounts: RDD[(Int, Iterable[(Int,Int)])] = > pairCounts.map{case((key, value), counts) => > key -> (value,counts) > }.groupByKey > > which would give you something like > > (1, Iterable((2,145), (3, 2)) > (4, Iterable((3, 982)) > > > hope this helps > Imran > > On Wed, Feb 18, 2015 at 1:43 AM, Tom Walwyn <[email protected]> wrote: > >> Thanks for the reply, I'll try your suggestions. >> >> Apologies, in my previous post I was mistaken. rdd is actually an PairRDD >> of (Int, Int). I'm doing the self-join so I can count two things. First, I >> can count the number of times a value appears in the data set. Second I can >> count number of times values occur with the same key. For example, if I >> have the following: >> >> (1,2) >> (1,3) >> (4,3) >> >> Then joining with itself I get: >> >> (1,(2,2)) - map - ((2,2),1) - reduceByKey - ((2,2),1) >> (1,(2,3)) - map - ((2,3),1) - reduceByKey - ((2,3),1) >> (1,(3,2)) - map - ((3,2),1) - reduceByKey - ((3,2),1) >> (1,(3,3)) - map - ((3,3),1) - reduceByKey - ((3,3),2) >> (4,(3,3)) - map - ((3,3),1) _| >> >> Note that I want to keep the duplicates (2,2) and reflections. >> >> Rgds >> >> On 18 February 2015 at 09:00, Akhil Das <[email protected]> >> wrote: >> >>> Why are you joining the rdd with itself? >>> >>> You can try these things: >>> >>> - Change the StorageLevel of both rdds to MEMORY_AND_DISK_2 or >>> MEMORY_AND_DISK_SER, so that it doesnt need to keep everything up in memory. >>> >>> - Set your default Serializer to Kryo (.set("spark.serializer", >>> "org.apache.spark.serializer.KryoSerializer")) >>> >>> - Enable rdd compression (.set("spark.rdd.compress","true")) >>> >>> >>> Thanks >>> Best Regards >>> >>> On Wed, Feb 18, 2015 at 12:21 PM, Tom Walwyn <[email protected]> wrote: >>> >>>> Hi All, >>>> >>>> I'm a new Spark (and Hadoop) user and I want to find out if the cluster >>>> resources I am using are feasible for my use-case. The following is a >>>> snippet of code that is causing a OOM exception in the executor after about >>>> 125/1000 tasks during the map stage. >>>> >>>> > val rdd2 = rdd.join(rdd, numPartitions=1000) >>>> > .map(fp=>((fp._2._1, fp._2._2), 1)) >>>> > .reduceByKey((x,y)=>x+y) >>>> > rdd2.count() >>>> >>>> Which errors with a stack trace like: >>>> >>>> > 15/02/17 16:30:11 ERROR executor.Executor: Exception in task 98.0 in >>>> stage 2.0 (TID 498) >>>> > java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> > at >>>> scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:168) >>>> > at >>>> scala.collection.mutable.ListBuffer.$plus$eq(ListBuffer.scala:45) >>>> > at >>>> scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48) >>>> > at >>>> scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48) >>>> > at scala.collection.immutable.List.foreach(List.scala:318) >>>> >>>> rdd is a PairRDD of (Int, (Int, Int)). The idea is to get the count of >>>> co-occuring values by key in the dataset, i.e. 'These two numbers occurred >>>> with the same key n times'. I intentionally don't want to filter out >>>> duplicates and reflections. rdd is about 3.6 million records, which has a >>>> size in memory of about 120MB, and results in a 'joined' RDD (before the >>>> reduceByKey stage) of around 460 million records, with a size in memory of >>>> about 35GB. >>>> >>>> My cluster setup is as follows. I have 3 nodes, where each node has 2 >>>> cores and about 7.5GB of memory. I'm running Spark on YARN. The driver and >>>> executors are allowed 1280m each and the job has 5 executors and 1 driver. >>>> Additionally, I have set spark.storage.memoryFraction to 0.06, and >>>> spark.shuffle.memoryFraction to 0.65 in the hopes that this would mitigate >>>> the issue. I've also tried increasing the number of partitions after the >>>> join dramatically (up to 15000). Nothing has been effective. Thus, I'm >>>> beginning to suspect I don't have enough resources for the job. >>>> >>>> Does anyone have a feeling about what the resource requirements would >>>> be for a use-case like this? I could scale the cluster up if necessary, but >>>> would like to avoid it. I'm willing to accept longer computation times if >>>> that is an option. >>>> >>>> Warm Regards, >>>> Thomas >>>> >>>> >>> >> >
