Re: A question about rdd transformation

2017-06-22 Thread Lionel Luffy
Now I found the root cause is a Wrapper class in AnyRef is not Serializable, but even though I changed it to implements Serializable. the 'rows' still cannot get data... Any suggestion? On Fri, Jun 23, 2017 at 10:56 AM, Lionel Luffy wrote: > Hi there, > I'm trying to do below action while it alw

A question about rdd transformation

2017-06-22 Thread Lionel Luffy
Hi there, I'm trying to do below action while it always return java.io.NotSerializableException in the shuffle task. I've checked that Array is serializable. how can I get the data of rdd in newRDD? step 1: val rdd: RDD[(AnyRef, Array[AnyRef]] {..} step2 : rdd .partitio