Hi,
I am using spark 0.9 and the Java API. I am configuring Kyro serialization
for my custom classes as follows:
SparkConf conf = new SparkConf();
conf.setMaster(args[0]);
conf.setAppName("Reifier");
conf.setJars(JavaSparkContext.jarOfClass(Reifier.class));
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
LOG.debug("Conf is " + conf.toDebugString());
JavaSparkContext ctx = new JavaSparkContext(conf);
The above prints
Conf is spark.app.name=Reifier
spark.jars=
spark.kryo.registrator=shark.KryoRegistrator
spark.master=local
spark.serializer=org.apache.spark.serializer.KryoSerializer
However, running the code gives me
org.apache.spark.SparkException: Job aborted: Task not serializable:
java.io.NotSerializableException:reifier.myClass
My questions are:
1. Is Kyro indeed being used, as the error above points to Java
serialization ?
2. I am using some external libaries which I can not set Serializable. How
can I avoid the above error?
Thanks a lot in advance for helping out.
Best Regards,
Sonal
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
