Hi,
I'm implementing KryoSerializer for my custom class. Here is class
public class ImpressionFactsValue implements KryoSerializable {
private int hits;
public ImpressionFactsValue() {
}
public int getHits() {
return hits;
}
public void setHits(int hits) {
this.hits = hits;
}
public void read(Kryo kryo, Input input) {
this.hits = input.readInt();
//kryo.readClass(input);
}
public void write(Kryo arg0, Output output) {
output.writeInt(this.hits);
//arg0.writeClass(output, ImpressionFactsValue.class);
}
}
Here is my KryoRegistrator
class MyRegistrator implements KryoRegistrator {
public void registerClasses(Kryo kryo) {
kryo.register(ImpressionFactsValue.class);
}
}
Here is spark conf
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
conf.set("spark.kryo.registrator", "com.MyRegistrator");
Getting exception
14/10/01 17:15:38 ERROR serializer.KryoSerializer: Failed to run
spark.kryo.registrator
java.lang.IllegalAccessException: Class
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2 can not access
a member of class com.MyRegistrator with modifiers ""
at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:95)
at java.lang.Class.newInstance0(Class.java:366)
at java.lang.Class.newInstance(Class.java:325)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:78)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:76)
at scala.Option.foreach(Option.scala:236)
at
org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:76)
at
org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:133)
at
org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:95)
at
org.apache.spark.util.collection.ExternalAppendOnlyMap.(ExternalAppendOnlyMap.scala:109)
at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:57)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:96)
at
org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:95)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594)
at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.Task.run(Task.scala:51)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Please help to resolve this
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org