Re: Kryo exception : Encountered unregistered class ID: 13994
You need to do few more things or you will eventually run into these issues
val conf = new SparkConf()
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
* .set("spark.kryoserializer.buffer.mb",
arguments.get("buffersize").get)*
* .set("spark.kryoserializer.buffer.max.mb",
arguments.get("maxbuffersize").get)*
.registerKryoClasses(Array(classOf[com.ebay.ep.poc.spark.reporting.process.model.dw.SpsLevelMetricSum]))
-Deepak
On Mon, Apr 13, 2015 at 1:19 PM, mehdisinger
wrote:
> Hello,
>
>
>
> Thank you for your answer.
>
>
>
> I’m already registering my classes as you’re suggesting…
>
>
>
> Regards
>
>
>
> *De :* tsingfu [via Apache Spark User List] [mailto:ml-node+[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=22471&i=0>]
> *Envoyé :* lundi 13 avril 2015 03:48
> *À :* Mehdi Singer
> *Objet :* Re: Kryo exception : Encountered unregistered class ID: 13994
>
>
>
> Hi,
> error message is mentioned:
> >com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: 13994
>
> So I think this is issue with kryo, You should use
> `kryo.register(classOf[your_class_name])` in your app code.
>
> ----------
>
> *If you reply to this email, your message will be added to the discussion
> below:*
>
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437p22468.html
>
> To unsubscribe from Kryo exception : Encountered unregistered class ID:
> 13994, click here.
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> ------
> View this message in context: RE: Kryo exception : Encountered
> unregistered class ID: 13994
> <http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437p22471.html>
>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
--
Deepak
RE: Kryo exception : Encountered unregistered class ID: 13994
Hello, Thank you for your answer. I'm already registering my classes as you're suggesting... Regards De : tsingfu [via Apache Spark User List] [mailto:ml-node+s1001560n22468...@n3.nabble.com] Envoyé : lundi 13 avril 2015 03:48 À : Mehdi Singer Objet : Re: Kryo exception : Encountered unregistered class ID: 13994 Hi, error message is mentioned: >com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: >13994 So I think this is issue with kryo, You should use `kryo.register(classOf[your_class_name])` in your app code. If you reply to this email, your message will be added to the discussion below: http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437p22468.html To unsubscribe from Kryo exception : Encountered unregistered class ID: 13994, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=22437&code=bWVoZGkuc2luZ2VyQGxhbXBpcmlzLmJlfDIyNDM3fC0xNDI5MjI3OTAz>. NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437p22471.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Kryo exception : Encountered unregistered class ID: 13994
Hi, From my experience, those errors happen under very high memory pressure, and/or with machines with bad hardware (memory, network card,..) I have had a few of them, as well as Snappy uncompress errors, on a machine with a slightly failing memory stick. Given the large amount of data transiting on the wire during shuffles, the probability of an error occuring during deserialization or uncompression is relatively high. In general, reducing the memory pressure also helps a lot. Guillaume Hi, I'm facing an issue when I try to run my Spark application. I keep getting the following exception: 15/04/09 15:14:07 ERROR Executor: Exception in task 5.0 in stage 1.0 (TID 5) com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994 Serialization trace: ord (org.apache.spark.util.BoundedPriorityQueue) at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:599) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:133) at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:236) at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I'm not sure where this exception occurs exactly... Does anyone know about this issue? I'm running Spark version 1.1.0. My Master and workers are running on different machines (cluster mode), all with the exact same architecture/configuration Can anyone help? Regards -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org -- eXenSa *Guillaume PITEL, Président* +33(0)626 222 431 eXenSa S.A.S. <http://www.exensa.com/> 41, rue Périer - 92120 Montrouge - FRANCE Tel +33(0)184 163 677 / Fax +33(0)972 283 705
Re: Kryo exception : Encountered unregistered class ID: 13994
Is there custom class involved in your application ? I assume you have called sparkConf.registerKryoClasses() for such class(es). Cheers On Thu, Apr 9, 2015 at 7:15 AM, mehdisinger wrote: > Hi, > > I'm facing an issue when I try to run my Spark application. I keep getting > the following exception: > > 15/04/09 15:14:07 ERROR Executor: Exception in task 5.0 in stage 1.0 (TID > 5) > com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: > 13994 > Serialization trace: > ord (org.apache.spark.util.BoundedPriorityQueue) > at > > com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) > at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) > at > > com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:599) > at > > com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) > at > > org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:133) > at > > org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:236) > at > > org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at > > org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) > at > > org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > I'm not sure where this exception occurs exactly... > Does anyone know about this issue? > > I'm running Spark version 1.1.0. > My Master and workers are running on different machines (cluster mode), all > with the exact same architecture/configuration > > Can anyone help? > > Regards > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Kryo exception : Encountered unregistered class ID: 13994
Hi, I'm facing an issue when I try to run my Spark application. I keep getting the following exception: 15/04/09 15:14:07 ERROR Executor: Exception in task 5.0 in stage 1.0 (TID 5) com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994 Serialization trace: ord (org.apache.spark.util.BoundedPriorityQueue) at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:599) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:133) at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:236) at org.apache.spark.broadcast.TorrentBroadcast.readObject(TorrentBroadcast.scala:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I'm not sure where this exception occurs exactly... Does anyone know about this issue? I'm running Spark version 1.1.0. My Master and workers are running on different machines (cluster mode), all with the exact same architecture/configuration Can anyone help? Regards -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
