Re: ClassNotFoundException when registering classes with Kryo
Thanks for the notification! For now, I'll use the Kryo serializer without registering classes until the bug fix has been merged into the next version of Spark (I guess that will be 1.3, right?). arun On Sun, Feb 1, 2015 at 10:58 PM, Shixiong Zhu wrote: > It's a bug that has been fixed in > https://github.com/apache/spark/pull/4258 but not yet been merged. > > Best Regards, > Shixiong Zhu > > 2015-02-02 10:08 GMT+08:00 Arun Lists : > >> Here is the relevant snippet of code in my main program: >> >> === >> >> sparkConf.set("spark.serializer", >> "org.apache.spark.serializer.KryoSerializer") >> sparkConf.set("spark.kryo.registrationRequired", "true") >> val summaryDataClass = classOf[SummaryData] >> val summaryViewClass = classOf[SummaryView] >> sparkConf.registerKryoClasses(Array( >> >> summaryDataClass, summaryViewClass)) >> >> === >> >> I get the following error: >> >> Exception in thread "main" java.lang.reflect.InvocationTargetException >> ... >> >> Caused by: org.apache.spark.SparkException: Failed to load class to >> register with Kryo >> ... >> >> Caused by: java.lang.ClassNotFoundException: >> com.dtex.analysis.transform.SummaryData >> >> >> Note that the class in question SummaryData is in the same package as the >> main program and hence in the same jar. >> >> What do I need to do to make this work? >> >> Thanks, >> arun >> >> >> >
Re: ClassNotFoundException when registering classes with Kryo
It's a bug that has been fixed in https://github.com/apache/spark/pull/4258 but not yet been merged. Best Regards, Shixiong Zhu 2015-02-02 10:08 GMT+08:00 Arun Lists : > Here is the relevant snippet of code in my main program: > > === > > sparkConf.set("spark.serializer", > "org.apache.spark.serializer.KryoSerializer") > sparkConf.set("spark.kryo.registrationRequired", "true") > val summaryDataClass = classOf[SummaryData] > val summaryViewClass = classOf[SummaryView] > sparkConf.registerKryoClasses(Array( > > summaryDataClass, summaryViewClass)) > > === > > I get the following error: > > Exception in thread "main" java.lang.reflect.InvocationTargetException > ... > > Caused by: org.apache.spark.SparkException: Failed to load class to > register with Kryo > ... > > Caused by: java.lang.ClassNotFoundException: > com.dtex.analysis.transform.SummaryData > > > Note that the class in question SummaryData is in the same package as the > main program and hence in the same jar. > > What do I need to do to make this work? > > Thanks, > arun > > >
ClassNotFoundException when registering classes with Kryo
Here is the relevant snippet of code in my main program: === sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") sparkConf.set("spark.kryo.registrationRequired", "true") val summaryDataClass = classOf[SummaryData] val summaryViewClass = classOf[SummaryView] sparkConf.registerKryoClasses(Array( summaryDataClass, summaryViewClass)) === I get the following error: Exception in thread "main" java.lang.reflect.InvocationTargetException ... Caused by: org.apache.spark.SparkException: Failed to load class to register with Kryo ... Caused by: java.lang.ClassNotFoundException: com.dtex.analysis.transform.SummaryData Note that the class in question SummaryData is in the same package as the main program and hence in the same jar. What do I need to do to make this work? Thanks, arun