how to convert RDD to PairRDDFunctions ?
Hi all, sorry for fooly question, but how can I get PairRDDFunctions RDD? I'm doing it to perform leftOuterJoin aftewards currently I do in this was (it seems incorrect): val parRDD = new PairRDDFunctions( oldRdd.map(i = (i.key, i)) ) I guess this constructor is definitely wrong... Thank you, Konstantin Kudryavtsev
Re: how to convert RDD to PairRDDFunctions ?
If your RDD contains pairs, like an RDD[(String,Integer)] or something, then you get to use the functions in PairRDDFunctions as if they were declared on RDD. On Tue, Jul 8, 2014 at 6:25 PM, Konstantin Kudryavtsev kudryavtsev.konstan...@gmail.com wrote: Hi all, sorry for fooly question, but how can I get PairRDDFunctions RDD? I'm doing it to perform leftOuterJoin aftewards currently I do in this was (it seems incorrect): val parRDD = new PairRDDFunctions( oldRdd.map(i = (i.key, i)) ) I guess this constructor is definitely wrong... Thank you, Konstantin Kudryavtsev
Re: how to convert RDD to PairRDDFunctions ?
See Working with Key-Value Pairs http://spark.apache.org/docs/latest/programming-guide.html. In particular: In Scala, these operations are automatically available on RDDs containing Tuple2 objects (the built-in tuples in the language, created by simply writing (a, b)), as long as you import org.apache.spark.SparkContext._ in your program to enable Spark’s implicit conversions. On Tue, Jul 8, 2014 at 10:25 AM, Konstantin Kudryavtsev kudryavtsev.konstan...@gmail.com wrote: Hi all, sorry for fooly question, but how can I get PairRDDFunctions RDD? I'm doing it to perform leftOuterJoin aftewards currently I do in this was (it seems incorrect): val parRDD = new PairRDDFunctions( oldRdd.map(i = (i.key, i)) ) I guess this constructor is definitely wrong... Thank you, Konstantin Kudryavtsev
