Hi Nick,
Yeah I saw that. I actually used sc.sequenceFile file to load data from
HDFS to RDD. Also both my key class and value class implements
WritableComparable of Hadoop. Still I got the error
"java.io.NotSerializableException", When I used sortByKey.
Hierarchy of my classes:
Collection
KeyCo
1 update:
Now both of the commands with and without sortByKey have started throwing
an error.
Loss was due to java.lang.NoSuchMethodError
java.lang.NoSuchMethodError:
com.guavus.logging.Logger.(Ljava/lang/Class;)V
at
com.guavus.mapred.common.collection.Collection.(Collection.java:17)
And Since sortByKey serializes the classes, I guess it has something to do
with Serialization thing.
On Mon, Dec 9, 2013 at 11:19 AM, Archit Thakur wrote:
> I did make the classes Serialized. But now running the same command
> sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
> fu
I did make the classes Serialized. But now running the same command
sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
func).sortByKey().count(), gives me java.lang.NoSuchMethodError.
For the Collection class which I made Serialized accesses one static
variable that
static com.xyz.l
It's because sorting serializes the data during the shuffle phase.
On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur wrote:
> Hi,
>
> When I did
>
> sc.sequenceFile(file, classOf[Text],
> classOf[Text]).flatMap(map_func).count()
> It gave me result of 365.
>
> However, when I did
> sc.sequenceFile(fi
Hi,
When I did
sc.sequenceFile(file, classOf[Text],
classOf[Text]).flatMap(map_func).count()
It gave me result of 365.
However, when I did
sc.sequenceFile(file, classOf[Text],
classOf[Text]).flatMap(map_func).sortByKey().count(),
It threw java.io.NotSerializableException for Key Class returned
