Re: KryoSerializer exception in Spark Streaming JAVA
Also please make sure you are specifying the fully qualified name of registrator class in the sparkconf configuration correctly. On Dec 11, 2014 5:57 PM, "bonnahu" wrote: > class MyRegistrator implements KryoRegistrator { > > public void registerClasses(Kryo kryo) { > kryo.register(ImpressionFactsValue.class); > } > > } > > change this class to public and give a try > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479p20647.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: KryoSerializer exception in Spark Streaming JAVA
class MyRegistrator implements KryoRegistrator { public void registerClasses(Kryo kryo) { kryo.register(ImpressionFactsValue.class); } } change this class to public and give a try -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479p20647.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
KryoSerializer exception in Spark Streaming JAVA
Hi, I'm implementing KryoSerializer for my custom class. Here is class public class ImpressionFactsValue implements KryoSerializable { private int hits; public ImpressionFactsValue() { } public int getHits() { return hits; } public void setHits(int hits) { this.hits = hits; } public void read(Kryo kryo, Input input) { this.hits = input.readInt(); //kryo.readClass(input); } public void write(Kryo arg0, Output output) { output.writeInt(this.hits); //arg0.writeClass(output, ImpressionFactsValue.class); } } Here is my KryoRegistrator class MyRegistrator implements KryoRegistrator { public void registerClasses(Kryo kryo) { kryo.register(ImpressionFactsValue.class); } } Here is spark conf conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer"); conf.set("spark.kryo.registrator", "com.MyRegistrator"); Getting exception 14/10/01 17:15:38 ERROR serializer.KryoSerializer: Failed to run spark.kryo.registrator java.lang.IllegalAccessException: Class org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2 can not access a member of class com.MyRegistrator with modifiers "" at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:95) at java.lang.Class.newInstance0(Class.java:366) at java.lang.Class.newInstance(Class.java:325) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:78) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$2.apply(KryoSerializer.scala:76) at scala.Option.foreach(Option.scala:236) at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:76) at org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:133) at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:95) at org.apache.spark.util.collection.ExternalAppendOnlyMap.(ExternalAppendOnlyMap.scala:109) at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:57) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:96) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$1.apply(PairRDDFunctions.scala:95) at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594) at org.apache.spark.rdd.RDD$$anonfun$15.apply(RDD.scala:594) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.Task.run(Task.scala:51) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Please help to resolve this -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-exception-in-Spark-Streaming-JAVA-tp15479.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: KryoSerializer Exception
43) java.lang.reflect.Method.invoke(Method.java:606) java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63) org.apache.spark.scheduler.ShuffleMapTask$.deserializeInfo(ShuffleMapTask.scala:63) org.apache.spark.scheduler.ShuffleMapTask.readExternal(ShuffleMapTask.scala:135) java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1837) java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:63) org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:85) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:169) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Driver stacktrace:at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633) at scala.Option.foreach(Option.scala:236)at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)* 2014-05-27 16:25 GMT+02:00 jaranda : > I am experiencing the same issue (I tried both using Kryo as serializer and > increasing the buffer size up to 256M, my objects are much smaller though). > I share my registrator class just in case: > > https://gist.github.com/JordiAranda/5cc16cf102290c413c82 > > Any hints would be highly appreciated. > > Thanks, > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-Exception-tp5435p6428.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. >
Re: KryoSerializer Exception
I am experiencing the same issue (I tried both using Kryo as serializer and increasing the buffer size up to 256M, my objects are much smaller though). I share my registrator class just in case: https://gist.github.com/JordiAranda/5cc16cf102290c413c82 Any hints would be highly appreciated. Thanks, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/KryoSerializer-Exception-tp5435p6428.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: KryoSerializer Exception
Hi Andrea, What version of Spark are you using? There were some improvements in how Spark uses Kryo in 0.9.1 and to-be 1.0 that I would expect to improve this. Also, can you share your registrator's code? Another possibility is that Kryo can have some difficulty serializing very large objects. Do you have a sense of how large the serialized items in your RDD are? Andrew On Sat, May 10, 2014 at 6:32 AM, Andrea Esposito wrote: > UP, doesn't anyone know something about it? ^^ > > > 2014-05-06 12:05 GMT+02:00 Andrea Esposito : > > Hi there, >> >> sorry if i'm posting a lot lately. >> >> i'm trying to add the KryoSerializer but i receive this exception: >> 2014 - 05 - 06 11: 45: 23 WARN TaskSetManager: 62 - Loss was due to >> java.io.EOFException >> java.io.EOFException >> at >> org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala: >> 105) >> at org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala: >> 165) >> at >> org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala: 56) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: >> 57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: >> 43) >> at java.lang.reflect.Method.invoke(Method.java: 606) >> >> I set the serializer as: >> System.setProperty("spark.serializer", >> "org.apache.spark.serializer.KryoSerializer") >> System.setProperty("spark.kryo.registrator", "test.TestKryoRegistrator") >> >> With or without register my custom registrator it throws the exception. >> >> Seems something related to broadcast.. but isn't Kryo already ok out of >> the box just setting it as default serializer? >> > >
Re: KryoSerializer Exception
UP, doesn't anyone know something about it? ^^ 2014-05-06 12:05 GMT+02:00 Andrea Esposito : > Hi there, > > sorry if i'm posting a lot lately. > > i'm trying to add the KryoSerializer but i receive this exception: > 2014 - 05 - 06 11: 45: 23 WARN TaskSetManager: 62 - Loss was due to > java.io.EOFException > java.io.EOFException > at > org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala: > 105) > at org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala: 165) > at > org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala: 56) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: > 43) > at java.lang.reflect.Method.invoke(Method.java: 606) > > I set the serializer as: > System.setProperty("spark.serializer", > "org.apache.spark.serializer.KryoSerializer") > System.setProperty("spark.kryo.registrator", "test.TestKryoRegistrator") > > With or without register my custom registrator it throws the exception. > > Seems something related to broadcast.. but isn't Kryo already ok out of > the box just setting it as default serializer? >
KryoSerializer Exception
Hi there, sorry if i'm posting a lot lately. i'm trying to add the KryoSerializer but i receive this exception: 2014 - 05 - 06 11: 45: 23 WARN TaskSetManager: 62 - Loss was due to java.io.EOFException java.io.EOFException at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala: 105) at org.apache.spark.broadcast.HttpBroadcast$.read(HttpBroadcast.scala: 165) at org.apache.spark.broadcast.HttpBroadcast.readObject(HttpBroadcast.scala: 56) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: 57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java: 43) at java.lang.reflect.Method.invoke(Method.java: 606) I set the serializer as: System.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer") System.setProperty("spark.kryo.registrator", "test.TestKryoRegistrator") With or without register my custom registrator it throws the exception. Seems something related to broadcast.. but isn't Kryo already ok out of the box just setting it as default serializer?