combine rdds?

2014-10-27 Thread Josh J
Hi, How could I combine rdds? I would like to combine two RDDs if the count in an RDD is not above some threshold. Thanks, Josh

Re: combine rdds?

2014-10-27 Thread Koert Kuipers
this requires evaluation of the rdd to do the count. val x: RDD[X] = ... val y: RDD[X] = ... x.cache val z = if(x.count < thres) x.union(y) else x On Oct 27, 2014 7:51 PM, "Josh J" wrote: Hi, How could I combine rdds? I would like to combine two RDDs if the count in an RDD is n