Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

examples on how to print all classes, but how to print all classes of kryo classloader - no idea. On 8 September 2015 at 16:43, Nick Peterson <nrpeter...@gmail.com> wrote: > Yes, the jar contains the class: > > $ jar -tf lumiata-evaluation-assembly-1.0.jar | grep 2028/Document

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

egarding your question about classloader - no idea, probably there is, I >> remember stackoverflow has some examples on how to print all classes, but >> how to print all classes of kryo classloader - no idea. >> >> On 8 September 2015 at 16:43, Nick Peterson <nrpeter...@gmai

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

> remember stackoverflow has some examples on how to print all classes, but > how to print all classes of kryo classloader - no idea. > > On 8 September 2015 at 16:43, Nick Peterson <nrpeter...@gmail.com> wrote: > >> Yes, the jar contains the class: >> >> $ jar

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

;>> appears 2 times or more...? or more generally - any 2 jars that can contain >>> this class by any chance >>> >>> regarding your question about classloader - no idea, probably there is, >>> I remember stackoverflow has some examples on how to print all classes

Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

I'm trying to run a Spark 1.4.1 job on my CDH5.4 cluster, through Yarn. Serialization is set to use Kryo. I have a large object which I send to the executors as a Broadcast. The object seems to serialize just fine. When it attempts to deserialize, though, Kryo throws a ClassNotFoundException

how to register CompactBuffer in Kryo

Hi, all I wrote a spark program which uses the Kryo serialization. When I count a rdd which type is RDD[(String,String)], it reported an Exception like the following : * Class is not registered: org.apache.spark.util.collection.CompactBuffer[] * Note: To register this class use: kryo.register

Re: how to register CompactBuffer in Kryo

For the exception w.r.t. ManifestFactory , there is SPARK-6497 which is Open. FYI On Fri, Aug 28, 2015 at 8:25 AM, donhoff_h 165612...@qq.com wrote: Hi, all I wrote a spark program which uses the Kryo serialization. When I count a rdd which type is RDD[(String,String)], it reported

Spark - Serialization with Kryo

Hello, I´ve got a problem using Spark with Geomesa. I´m not quite sure where the error comes from, but I assume its problem with Spark. A ClassNotFoundException is thrown with following content: Failed to register classes with Kryo. Please have a look at https://github.com/apache/spark/pull/4258

Kryo Serializer on Worker doesn't work by default.

Hello. I have an issue with CustomKryoRegistrator, which causes ClassNotFound on Worker. The issue is resolved if call SparkConf.setJar with path to the same jar I run. It is a workaround, but it requires to specify the same jar file twice. The first time I use it to actually run the job, and

Re: Kryo Serializer on Worker doesn't work by default.

What I seem to be don’t get is how my code ends up being on Worker node. My understanding was that jar file, which I use to start the job should automatically be copied into Worker nodes and added to classpath. It seems to be not the case. But if my jar is not copied into Worker nodes, then how

Re: Why Kryo Serializer is slower than Java Serializer in TeraSort

actually correct? Hope this helps.. Regards, Gylfi. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Why-Kryo-Serializer-is-slower-than-Java-Serializer-in-TeraSort-tp23621p23659.html Sent from the Apache Spark User List mailing list archive

Why Kryo Serializer is slower than Java Serializer in TeraSort

Hi, I am using TeraSort benchmark from ehiggs's branch https://github.com/ehiggs/spark-terasort https://github.com/ehiggs/spark-terasort . Then I noticed that in TeraSort.scala, it is using Kryo Serializer. So I made a small change from org.apache.spark.serializer.KryoSerializer

Re: Why Kryo Serializer is slower than Java Serializer in TeraSort

Looks like, it spend more time writing/transferring the 40GB of shuffle when you used kryo. And surpirsingly, JavaSerializer has 700MB of shuffle? Thanks Best Regards On Sun, Jul 5, 2015 at 12:01 PM, Gavin Liu ilovesonsofanar...@gmail.com wrote: Hi, I am using TeraSort benchmark from

Re: Why Kryo Serializer is slower than Java Serializer in TeraSort

That code doesn't appear to be registering classes with Kryo, which means the fully-qualified classname is stored with every Kryo record. The Spark documentation has more on this: https://spark.apache.org/docs/latest/tuning.html#data-serialization Regards, Will On July 5, 2015, at 2:31 AM

Re: Kryo fails to serialise output

Kryo serialization is used internally by Spark for spilling or shuffling intermediate results, not for writing out an RDD as an action. Look at Sandy Ryza's examples for some hints on how to do this: https://github.com/sryza/simplesparkavroapp Regards, Will On July 3, 2015, at 2:45 AM

Kryo fails to serialise output

I have a rather simple avro schema to serialize Tweets (message, username, timestamp). Kryo and twitter chill are used to do so. For my dev environment the Spark context is configured as below val conf: SparkConf = new SparkConf() conf.setAppName(kryo_test) conf.setMaster(“local[4]) conf.set

Re: Kryo serialization of classes in additional jars

to the cluster manually, and then using spark.executor.extraClassPath On Wed, Apr 29, 2015 at 6:42 PM, Akshat Aranya aara...@gmail.com mailto:aara...@gmail.com wrote: Hi, Is it possible to register kryo serialization for classes contained in jars that are added with spark.jars? In my

Re: UDF accessing hive struct array fails with buffer underflow from kryo

This is a kryo issue. https://github.com/EsotericSoftware/kryo/issues/124. It has to do with the lengths of the fieldnames. This issue is fixed in Kryo 2.23. What's weird is this doesn't break on Hive itself, only when using SparkSQL. Attached is the full stacktrace. It might be how SparkSQL

UDF accessing hive struct array fails with buffer underflow from kryo

table. Running the same query on Hive works... what's going on here? Any suggestions on how to debug this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/UDF-accessing-hive-struct-array-fails-with-buffer-underflow-from-kryo-tp23078.html Sent from the Apache

Re: Kryo serialization of classes in additional jars

I cherry-picked this commit into my local 1.2 branch. It fixed the problem with setting spark.serializer, but I get a similar problem with spark.closure.serializer org.apache.spark.SparkException: Failed to register classes with Kryo at org.apache.spark.serializer.KryoSerializer.newKryo

Kryo read method never called before reducing

NullPointerException on right1 (left1.merge(left2), right1.merge(right2)) }) Tuples of graphs are serialized correctly (we use a custom serializer). However, in the reduce method we get null instead of the right graph. The problem seems to be cause by the Kryo Serialization. In fact, when

Spark Kryo read method never called before reducing

to be cause by the Kryo Serialization. In fact, when we used java serialization Tuples are deserialized correctly. In addition, using two mapPartition/reduce, the first one for left graphs and the second one for right graphs the objects are deserialized correctly by kryo. This is our serializer class

Re: Kryo serialization of classes in additional jars

to the cluster manually, and then using spark.executor.extraClassPath On Wed, Apr 29, 2015 at 6:42 PM, Akshat Aranya aara...@gmail.com wrote: Hi, Is it possible to register kryo serialization for classes contained in jars that are added with spark.jars? In my experiment it doesn't seem to work, likely

Re: spark kryo serialization question

is pretty minimal, so as long as you do this within reason, I think you're OK. Imran On Thu, Apr 30, 2015 at 12:34 AM, 邓刚[技术中心] triones.d...@vipshop.com wrote: Hi all We know that spark support Kryo serialization, suppose there is a map function which map C to K,V(here C,K,V

Re: ClassNotFoundException for Kryo serialization

(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) I verified that the same configuration works without using Kryo serialization. On Fri, May 1, 2015 at 9:44 AM, Akshat Aranya aara...@gmail.com wrote: I cherry-picked the fix for SPARK-5470 and the problem has gone away. On Fri, May

ClassNotFoundException for Kryo serialization

Hi, I'm getting a ClassNotFoundException at the executor when trying to register a class for Kryo serialization: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method

Re: ClassNotFoundException for Kryo serialization

a ClassNotFoundException at the executor when trying to register a class for Kryo serialization: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance

Re: ClassNotFoundException for Kryo serialization

about Schema$MyRow ? On Fri, May 1, 2015 at 8:05 AM, Akshat Aranya aara...@gmail.com wrote: Hi, I'm getting a ClassNotFoundException at the executor when trying to register a class for Kryo serialization: java.lang.reflect.InvocationTargetException

Re: ClassNotFoundException for Kryo serialization

at the executor when trying to register a class for Kryo serialization: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57

Kryo serialization of classes in additional jars

Hi, Is it possible to register kryo serialization for classes contained in jars that are added with spark.jars? In my experiment it doesn't seem to work, likely because the class registration happens before the jar is shipped to the executor and added to the classloader. Here's the general idea

spark kryo serialization question

Hi all We know that spark support Kryo serialization, suppose there is a map function which map C to K,V(here C,K,V are instance of class C,K,V), when we register kryo serialization, should I register all of these three class? Best Wishes Triones Deng 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人

Execption while using kryo with broadcast

Hi All I am getting below exception while using Kyro serializable with broadcast variable. I am broadcating a hasmap with below line. MapLong, MatcherReleventData matchData =RddForMarch.collectAsMap(); final BroadcastMapLong, MatcherReleventData dataMatchGlobal = jsc.broadcast(matchData);

Re: Execption while using kryo with broadcast

Yes Without Kryo it did work out.when I remove kryo registration it did worked out On 15 April 2015 at 19:24, Jeetendra Gangele gangele...@gmail.com wrote: its not working with the combination of Broadcast. Without Kyro also not working. On 15 April 2015 at 19:20, Akhil Das ak

Re: Execption while using kryo with broadcast

Is it working without kryo? Thanks Best Regards On Wed, Apr 15, 2015 at 6:38 PM, Jeetendra Gangele gangele...@gmail.com wrote: Hi All I am getting below exception while using Kyro serializable with broadcast variable. I am broadcating a hasmap with below line. MapLong, MatcherReleventData

Re: Execption while using kryo with broadcast

, Jeetendra Gangele gangele...@gmail.com wrote: Yes Without Kryo it did work out.when I remove kryo registration it did worked out On 15 April 2015 at 19:24, Jeetendra Gangele gangele...@gmail.com wrote: its not working with the combination of Broadcast. Without Kyro also not working. On 15

Re: Execption while using kryo with broadcast

this is a really strange exception ... I'm especially surprised that it doesn't work w/ java serialization. Do you think you could try to boil it down to a minimal example? On Wed, Apr 15, 2015 at 8:58 AM, Jeetendra Gangele gangele...@gmail.com wrote: Yes Without Kryo it did work out.when I

Re: Execption while using kryo with broadcast

, MatcherReleventData dataMatchGlobal = jsc.broadcast(tmp); Can you please clarify: * Does it work w/ java serialization in the end? Or is this kryo only? * which Spark version you are using? (one of the relevant bugs was fixed in 1.2.1 and 1.3.0) On Wed, Apr 15, 2015 at 9:06 AM, Jeetendra Gangele gangele

Re: Execption while using kryo with broadcast

its not working with the combination of Broadcast. Without Kyro also not working. On 15 April 2015 at 19:20, Akhil Das ak...@sigmoidanalytics.com wrote: Is it working without kryo? Thanks Best Regards On Wed, Apr 15, 2015 at 6:38 PM, Jeetendra Gangele gangele...@gmail.com wrote: Hi All

Re: Execption while using kryo with broadcast

it work w/ java serialization in the end? Or is this kryo only? * which Spark version you are using? (one of the relevant bugs was fixed in 1.2.1 and 1.3.0) On Wed, Apr 15, 2015 at 9:06 AM, Jeetendra Gangele gangele...@gmail.com wrote: This looks like known issue? check this out http

RE: Kryo exception : Encountered unregistered class ID: 13994

Hello, Thank you for your answer. I'm already registering my classes as you're suggesting... Regards De : tsingfu [via Apache Spark User List] [mailto:ml-node+s1001560n22468...@n3.nabble.com] Envoyé : lundi 13 avril 2015 03:48 À : Mehdi Singer Objet : Re: Kryo exception : Encountered

Re: Kryo exception : Encountered unregistered class ID: 13994

my classes as you’re suggesting… Regards *De :* tsingfu [via Apache Spark User List] [mailto:ml-node+[hidden email] http:///user/SendEmail.jtp?type=nodenode=22471i=0] *Envoyé :* lundi 13 avril 2015 03:48 *À :* Mehdi Singer *Objet :* Re: Kryo exception : Encountered unregistered class ID

Re: Kryo exception : Encountered unregistered class ID: 13994

/Kryo-exception-Encountered-unregistered-class-ID-13994-tp22437.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands

Java and Kryo Serialization, Java.io.OptionalDataException

I have set Kryo Serializer as default serializer in SparkConf and Spark UI confirms it too, but in the Spark logs I'm getting this exception, java.io.OptionalDataException at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1370) at java.io.ObjectInputStream.readObject

Re: Spark SQL UDT Kryo serialization, Unable to find class

You probably don't cause a shuffle (which requires serialization) unless there is a join or group by. It's possible that we are need to pass the spark class loader to kryo when creating a new instance (you can get it from Utils I believe). We never run Otto this problem since this API

Re: Spark SQL UDT Kryo serialization, Unable to find class

/scala/org/apache/spark/sql/test/ExamplePointUDT.scala example. But I'm having Kryo Serialization issues, here is stack trace: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 22

Spark SQL UDT Kryo serialization, Unable to find class

Hi, I want to introduce custom type for SchemaRDD, I'm following this https://github.com/apache/spark/blob/branch-1.2/sql/core/src/main/scala/org/apache/spark/sql/test/ExamplePointUDT.scala example. But I'm having Kryo Serialization issues, here is stack trace: org.apache.spark.SparkException

Re: Workaround for spark 1.2.X roaringbitmap kryo problem?

to be registered (the error is only when I turn on kryo). However the code is running smoothly with kryo turned off. On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid iras...@cloudera.com wrote: I'm not sure what you mean. Are you asking how you can recompile all of spark and deploy it, instead

Re: Workaround for spark 1.2.X roaringbitmap kryo problem?

saying that it needs to be registered (the error is only when I turn on kryo). However the code is running smoothly with kryo turned off. On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid iras...@cloudera.com wrote: I'm not sure what you mean. Are you asking how you can recompile all of spark

Re: Workaround for spark 1.2.X roaringbitmap kryo problem?

]) If I don't register it, I get a runtime error saying that it needs to be registered (the error is only when I turn on kryo). However the code is running smoothly with kryo turned off. On Wed, Mar 11, 2015 at 5:38 PM, Imran Rashid iras...@cloudera.com wrote: I'm not sure what you mean

Re: Workaround for spark 1.2.X roaringbitmap kryo problem?

compiles fine, so I'm not sure what problem you are running into -- we'd need a lot more info to help On Tue, Mar 10, 2015 at 6:54 PM, Arun Luthra arun.lut...@gmail.com wrote: Does anyone know how to get the HighlyCompressedMapStatus to compile? I will try turning off kryo in 1.2.0 and hope things

Re: Workaround for spark 1.2.X roaringbitmap kryo problem?

Does anyone know how to get the HighlyCompressedMapStatus to compile? I will try turning off kryo in 1.2.0 and hope things don't break. I want to benefit from the MapOutputTracker fix in 1.2.0. On Tue, Mar 3, 2015 at 5:41 AM, Imran Rashid iras...@cloudera.com wrote: the scala syntax

Re: Workaround for spark 1.2.X roaringbitmap kryo problem?

the scala syntax for arrays is Array[T], not T[], so you want to use something: kryo.register(classOf[Array[org.roaringbitmap.RoaringArray$Element]]) kryo.register(classOf[Array[Short]]) nonetheless, the spark should take care of this itself. I'll look into it later today. On Mon, Mar 2, 2015

Re: Workaround for spark 1.2.X roaringbitmap kryo problem?

I think this is a Java vs scala syntax issue. Will check. On Thu, Feb 26, 2015 at 8:17 PM, Arun Luthra arun.lut...@gmail.com wrote: Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949 I tried this as a workaround: import org.apache.spark.scheduler._ import

Workaround for spark 1.2.X roaringbitmap kryo problem?

Problem is noted here: https://issues.apache.org/jira/browse/SPARK-5949 I tried this as a workaround: import org.apache.spark.scheduler._ import org.roaringbitmap._ ... kryo.register(classOf[org.roaringbitmap.RoaringBitmap]) kryo.register(classOf[org.roaringbitmap.RoaringArray])

Re: Custom Kryo serializer

I was able to get this working by extending KryoRegistrator and setting the spark.kryo.registrator property. On Thu, Feb 12, 2015 at 12:31 PM, Corey Nolet cjno...@gmail.com wrote: I'm trying to register a custom class that extends Kryo's Serializer interface. I can't tell exactly what Class

Custom Kryo serializer

I'm trying to register a custom class that extends Kryo's Serializer interface. I can't tell exactly what Class the registerKryoClasses() function on the SparkConf is looking for. How do I register the Serializer class?

Spark Streaming: Flume receiver with Kryo serialization

Hi, I want to include if possible Kryo serialization in a project and first I'm trying to run FlumeEventCount with Kryo. If I comment setAll method, runs correctly, but if I use Kryo params it returns several errors. 15/02/11 11:42:16 ERROR SparkDeploySchedulerBackend: Asked to remove non

ClassNotFoundException when registering classes with Kryo

to load class to register with Kryo ... Caused by: java.lang.ClassNotFoundException: com.dtex.analysis.transform.SummaryData Note that the class in question SummaryData is in the same package as the main program and hence in the same jar. What do I need to do to make this work? Thanks, arun

Re: ClassNotFoundException when registering classes with Kryo

Thanks for the notification! For now, I'll use the Kryo serializer without registering classes until the bug fix has been merged into the next version of Spark (I guess that will be 1.3, right?). arun On Sun, Feb 1, 2015 at 10:58 PM, Shixiong Zhu zsxw...@gmail.com wrote: It's a bug that has

Re: ClassNotFoundException when registering classes with Kryo

)) === I get the following error: Exception in thread main java.lang.reflect.InvocationTargetException ... Caused by: org.apache.spark.SparkException: Failed to load class to register with Kryo ... Caused by: java.lang.ClassNotFoundException

Re: Duplicate key when sorting BytesWritable with Kryo?

this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Duplicate-key-when-sorting-BytesWritable-with-Kryo-tp21447.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e

Re: Duplicate key when sorting BytesWritable with Kryo?

://apache-spark-user-list.1001560.n3.nabble.com/Duplicate-key-when-sorting-BytesWritable-with-Kryo-tp21447.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr

Re: Duplicate key when sorting BytesWritable with Kryo?

the default configuration values set. Using the external shuffle service and disabling spill compression makes no difference. Is this a bug? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Duplicate-key-when-sorting-BytesWritable-with-Kryo-tp21447.html

Duplicate key when sorting BytesWritable with Kryo?

/Duplicate-key-when-sorting-BytesWritable-with-Kryo-tp21447.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h

Kryo buffer overflows

A search shows several historical threads for similar Kryo issues, but none seem to have a definitive solution. Currently using Spark 1.2.0. While collecting/broadcasting/grouping moderately sized data sets (~500MB - 1GB), I regularly see exceptions such as the one below. I’ve tried increasing

Spark attempts to de/serialize using JavaSerializer despite being configured to use Kryo

I'm new to Spark and have run into issues using Kryo for serialization instead of Java. I have my SparkConf configured as such: val conf = new SparkConf().setMaster(local).setAppName(test) .set(spark.kryo.registrationRequired,false) .set(spark.serializer, classOf[KryoSerializer

Custom receiver runtime Kryo exception

Hello all, I am using Spark 1.0.2 and I have a custom receiver that works well. I tried adding Kryo serialization to SparkConf: val spark = new SparkConf() ….. .set(spark.serializer, org.apache.spark.serializer.KryoSerializer) and I am getting a strange error that I am not sure how

Locality level and Kryo

that kryo works well). I'm running Spark 1.0.0 with the following characteristics: - 18 executors with 30G each - Yarn client mode - ulimit is defined in 500k - Input data: hdfs file with 1000 partitions and 10 GB of size Please any hint would be appreciated -- View this message in context

Locality Level Kryo

Hi guys, I get Kryo exceptions of the type unregistered class id and cannot cast to class when the locality level of the tasks go beyond LOCAL. However I get no Kryo exceptions during shuffling operations. If the locality level never goes beyond LOCAL everything works fine. Is there a special

Re: KryoRegistrator exception and Kryo class not found while compiling

Is the class com.dataken.spark.examples.MyRegistrator public? if not, change it to public and give a try. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/KryoRegistrator-exception-and-Kryo-class-not-found-while-compiling-tp10396p20646.html Sent from

Re: Serializing with Kryo NullPointerException - Java

version2.24.0/version /dependency fixed this On 2014-12-03 18:15, Robin Keunen wrote: Hi all, I am having troubles using Kryo and being new to this kind of serialization, I am not sure where to look. Can someone please help me? :-) Here is my custom class: public class

Serializing with Kryo NullPointerException - Java

Hi all, I am having troubles using Kryo and being new to this kind of serialization, I am not sure where to look. Can someone please help me? :-) Here is my custom class: public class *DummyClass* implements KryoSerializable { private static final Logger LOGGER = LoggerFactory.getLogger

Re: Kryo NPE with Array

was instead scala.collection.convert.Wrappers.JListWrapper, as noted in the exception above. This type was not registered with Kryo and so that's why I got the exception. Registering the type did not solve the problem. However, an additional call to .toBuffer did solve the problem, since the Buffer class

Kryo exception for CassandraSQLRow

I am using Cassandra-Spark connector to pull data from Cassandra, process it and write it back to Cassandra. Now I am getting the following exception, and apparently it is Kryo serialisation. Does anyone what is the reason and how this can be solved? I also tried to register

RE: Kryo exception for CassandraSQLRow

Don't know if this'll solve it, but if you're on Spark 1.1, the Cassandra Connector version 1.1.0 final fixed the guava back compat issue. Maybe taking the guava exclusions might help? Date: Mon, 1 Dec 2014 10:48:25 +0100 Subject: Kryo exception for CassandraSQLRow From: shahab.mok...@gmail.com

Re: Kryo NPE with Array

I guess I already have the answer of what I have to do here, which is to configure the kryo object with the strategy as above. Now the question becomes: how can I pass this custom kryo configuration to the spark kryo serializer / kryo registrator? I've had a look at the code but I am still fairly

Re: Unable to use Kryo

The problem was I didn't use the correct class name, it should be org.apache.spark.*serializer*.KryoSerializer On Mon, Nov 24, 2014 at 11:12 PM, Daniel Haviv danielru...@gmail.com wrote: Hi, I want to test Kryo serialization but when starting spark-shell I'm hitting the following error

Kryo NPE with Array

(ObjectField.java:125) I have been running into similar issues when using avro classes, that I was able to resolve by registering them with a Kryo serializer that uses chill-avro. However, in this case the field is in a case class and it seems that registering the class does not help. I found this stack

Unable to use Kryo

Hi, I want to test Kryo serialization but when starting spark-shell I'm hitting the following error: java.lang.ClassNotFoundException: org.apache.spark.KryoSerializer the kryo-2.21.jar is on the classpath so I'm not sure why it's not picking it up. Thanks for your help, Daniel

Kryo serialization in examples.streaming.TwitterAlgebirdCMS/HLL

Hi, If I look inside algebird Monoid implementation it uses java.io.Serializable... But when we use CMS/HLL in examples.streaming.TwitterAlgebirdCMS, I don't see a KryoRegistrator for CMS and HLL monoid... In these examples we will run with Kryo serialization on CMS and HLL or they will be java

Re: registering Array of CompactBuffer to Kryo

How about this? Class.forName([Lorg.apache.spark.util.collection.CompactBuffer;) On Tue, Sep 30, 2014 at 5:33 PM, Andras Barjak andras.bar...@lynxanalytics.com wrote: Hi, what is the correct scala code to register an Array of this private spark class to Kryo

Re: registering Array of CompactBuffer to Kryo

([Lorg.apache.spark.util.collection.CompactBuffer;) On Tue, Sep 30, 2014 at 5:33 PM, Andras Barjak andras.bar...@lynxanalytics.com wrote: Hi, what is the correct scala code to register an Array of this private spark class to Kryo? java.lang.IllegalArgumentException: Class is not registered

registering Array of CompactBuffer to Kryo

Hi, what is the correct scala code to register an Array of this private spark class to Kryo? java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.util.collection.CompactBuffer[] Note: To register this class use: kryo.register

Kryo UnsupportedOperationException

We're running into an error (below) when trying to read spilled shuffle data back in. Has anybody encountered this before / is anybody familiar with what causes these Kryo UnsupportedOperationExceptions? any guidance appreciated, Sandy --- com.esotericsoftware.kryo.KryoException

Re: Kryo UnsupportedOperationException

shuffle data back in. Has anybody encountered this before / is anybody familiar with what causes these Kryo UnsupportedOperationExceptions? any guidance appreciated, Sandy --- com.esotericsoftware.kryo.KryoException (com.esotericsoftware.kryo.KryoException

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

.1001560.n3.nabble.com/Kryo-fails-with-avro-having-Arrays-and-unions-but-succeeds-with-simple-avro-tp14549p14649.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Hi Mohan, It’s a bit convoluted to follow in their source, but they essentially typedef KSerializer as being a KryoSerializer, and then their serializers all extend KSerializer. Spark should identify them properly as Kryo Serializers, but I haven’t tried it myself. Regards, Frank Austin

Re: Kryo fails with avro having Arrays and unions, but succeeds with simple avro.

Hi frank, thanks for the info, thats great. but im not saying Avro serializer is failing. Kryo is failing but im using kryo serializer. and registering Avro generated classes with kryo. sparkConf.set(spark.serializer, org.apache.spark.serializer.KryoSerializer); sparkConf.set

spark kryo serilizable exception

hi all, In RDD map , i invoke an object that is *Serialized* by java standard , and exception :: com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 13 at com.esotericsoftware.kryo.io.Output.require(Output.java:138) at

RE: spark kryo serilizable exception

Hi,I was able to set this parameter in my application to resolve this issue: set(spark.kryoserializer.buffer.mb, 256) Please let me know if this helps. Date: Mon, 18 Aug 2014 21:50:02 +0800 From: dujinh...@hzduozhun.com To: user@spark.apache.org Subject: spark kryo serilizable exception

kryo out of buffer exception

Hi All, I was doing a groupBy and apparently some keys were very frequent making the serializer fail with buffer overflow exception. I did not need a groupBy so I switched to combineByKey in this case but would like to know how to increase the kryo buffer sizes to avoid this error. I hope

Re: Issue using kryo serilization

any pointers to this issue. Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Issue-using-kryo-serilization-tp11129p11191.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Kryo Issue on Spark 1.0.1, Mesos 0.18.2

PM, Gary Malouf malouf.g...@gmail.com wrote: After upgrading to Spark 1.0.1 from 0.9.1 everything seemed to be going well. Looking at the Mesos slave logs, I noticed: ERROR KryoSerializer: Failed to run spark.kryo.registrator java.lang.ClassNotFoundException: com/mediacrossing/verrazano/kryo

Kryo NoSuchMethodError on Spark 1.0.0 standalone

Hi there, I've been sucessfully using the precompiled Spark 1.0.0 Java api on a small cluster in standalone mode. However, when I try to use Kryo serializer by adding conf.set(spark.serializer,org.apache.spark.serializer.KryoSerializer); as suggested, Spark crashes out with the following error

RE: Kryo is slower, and the size saving is minimal

Message- From: wxhsdp [mailto:wxh...@gmail.com] Sent: Wednesday, July 09, 2014 5:47 PM To: u...@spark.incubator.apache.org Subject: Re: Kryo is slower, and the size saving is minimal i'am not familiar with kryo and my opinion may be not right. in my case, kryo only saves about 5% of the original

Error with Stream Kafka Kryo

Hi My setup is to use localMode standalone, Sprak 1.0.0 release version, scala 2.10.4 I made a job that receive serialized object from Kafka broker. The objects are serialized using kryo. The code : val sparkConf = new SparkConf().setMaster(local[4]).setAppName(SparkTest) .set

Kryo is slower, and the size saving is minimal

Hi, For my test case, using Kryo serializer does not help. It is slower than default Java serializer, and the size saving is minimal. I've registered almost all classes to the Kryo registrator. What is happening to my test case? Have Anyone experienced a case like this?

Re: Crazy Kryo Exception

Macbeth so...@yieldbot.com wrote: Hallo, I've getting this rather crazy kryo exception trying to run my spark job: Exception in thread main org.apache.spark.SparkException: Job aborted: Exception while deserializing and fetching task: com.esotericsoftware.kryo.KryoException

using kryo for spark.closure.serializer with a registrator doesn't work

Is this supposed to be supported? It doesn't work, at least in mesos fine grained mode. First it fails a bunch of times because it can't find my registrator class because my assembly jar hasn't been fetched like so: java.lang.ClassNotFoundException: pickles.kryo.PicklesRegistrator at

Re: Crazy Kryo Exception

it seems that it dying while trying to fetch results from my tasks to return back to the driver. Am I close? On Fri, May 2, 2014 at 3:35 PM, Soren Macbeth so...@yieldbot.com wrote: Hallo, I've getting this rather crazy kryo exception trying to run my spark job: Exception in thread main

<    1   2   3   >