heya, Without a bit of gymnastic at the type level, nope. Actually RDD doesn't share any functions with the scala lib (the simple reason I could see is that the Spark's ones are lazy, the default implementations in Scala aren't).
However, it'd be possible by implementing an implicit converter from a SeqLike (f.i.) to an RDD, nonetheless it'd be cumbersome because the overlap between the two world isn't entire (for instance, flatMap haven't the same semantic, drop is hard, etc). Also, it'd scary me a bit to have this kind of bazooka waiting me a next corner, by letting me think that a iterative like process can be ran in a distributed world :-). OTOH, the inverse is quite easy, an implicit conv from RDD to an Array is simply a call to collect (take care that RDD is not covariant -- I think it'd be related to the fact that the ClassTag is needed!?) only my .2 ¢ aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] <http://about.me/noootsab> On Mon, Jul 21, 2014 at 7:01 PM, Philip Ogren <philip.og...@oracle.com> wrote: > It is really nice that Spark RDD's provide functions that are often > equivalent to functions found in Scala collections. For example, I can > call: > > myArray.map(myFx) > > and equivalently > > myRdd.map(myFx) > > Awesome! > > My question is this. Is it possible to write code that works on either an > RDD or a local collection without having to have parallel implementations? > I can't tell that RDD or Array share any supertypes or traits by looking > at the respective scaladocs. Perhaps implicit conversions could be used > here. What I would like to do is have a single function whose body is like > this: > > myData.map(myFx) > > where myData could be an RDD[Array[String]] (for example) or an > Array[Array[String]]. > > Has anyone had success doing this? > > Thanks, > Philip > > >
