And Since sortByKey serializes the classes, I guess it has something to do with Serialization thing.
On Mon, Dec 9, 2013 at 11:19 AM, Archit Thakur <archit279tha...@gmail.com>wrote: > I did make the classes Serialized. But now running the same command > sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_ > func).sortByKey().count(), gives me java.lang.NoSuchMethodError. > > For the Collection class which I made Serialized accesses one static > variable that > > static com.xyz.logging.Logger Logger = new > com.xyz.logging.Logger(Collection.class) and It throws > > java.lang.NoSuchMethodError: > com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V > at > com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17) > > but it doesn't do that when I don't sortByKey, ie when I run > sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_ > func).count() it doesn't throw the error. > > Thanks and Regards, > Archit Thakur. > > > > On Mon, Dec 9, 2013 at 10:48 AM, Patrick Wendell <pwend...@gmail.com>wrote: > >> It's because sorting serializes the data during the shuffle phase. >> >> On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <archit279tha...@gmail.com> >> wrote: >> > Hi, >> > >> > When I did >> > >> > sc.sequenceFile(file, classOf[Text], >> > classOf[Text]).flatMap(map_func).count() >> > It gave me result of 365. >> > >> > However, when I did >> > sc.sequenceFile(file, classOf[Text], >> > classOf[Text]).flatMap(map_func).sortByKey().count(), >> > >> > It threw java.io.NotSerializableException for Key Class returned by >> flapMap. >> > My question is >> > Why does sortByKey require the Key/Value Classes to be Serialized.? >> > >> > Thanks and Regards, >> > Archit Thakur. >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Spark Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to spark-users+unsubscr...@googlegroups.com. >> > For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Spark Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to spark-users+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> > >
