[SQL] Why does a small two-source JDBC query take ~150-200ms with all optimizations (AQE, CBO, pushdown, Kryo, unsafe) enabled? (v3.4.0-SNAPSHOT)

I did some basic testing of multi-source queries with the most recent Spark: https://github.com/GavinRay97/spark-playground/blob/44a756acaee676a9b0c128466e4ab231a7df8d46/src/main/scala/Application.scala#L46-L115 The output of "spark.time()" surprised me: SELECT p.id, p.name, t.id, t.title FROM

Supporting Kryo registration in DSv2

Hello all, Is there a way to register classes within a datasourcev2 implementation in the Kryo serializer? I've attempted the following in both the constructor and static block of my toplevel class: SparkContext context = SparkContext.getOrCreate(); SparkConf conf

Re: intermittent Kryo serialization failures in Spark

trator >> >> class MyKryoRegistrator extends KryoRegistrator { >> override def registerClasses(kryo: Kryo): Unit = { >> kryo.register(Class.forName("[[B")) // byte[][] >> kryo.register(classOf[java.lang.Class[_]]) >> } >> } >&g

Re: intermittent Kryo serialization failures in Spark

tends KryoRegistrator { > override def registerClasses(kryo: Kryo): Unit = { > kryo.register(Class.forName("[[B")) // byte[][] > kryo.register(classOf[java.lang.Class[_]]) > } > } > ``` > > then run with > > 'spark.kryo.referenceTr

Re: intermittent Kryo serialization failures in Spark

at this is a bit different from your code and I'm wondering > whether there's any functional difference or if these are two ways to get > to the same end. Our code is directly adapted from the Spark documentation > on how to use the Kryo serializer but maybe there's some subtlety here that &

Re: intermittent Kryo serialization failures in Spark

.registerKryoClasses(KryoRegistrar.classesToRegister) > I notice that this is a bit different from your code and I'm wondering whether there's any functional difference or if these are two ways to get to the same end. Our code is directly adapted from the Spark documentation on how to use the Kryo serialize

Re: intermittent Kryo serialization failures in Spark

Pre-register your classes: ``` import com.esotericsoftware.kryo.Kryo import org.apache.spark.serializer.KryoRegistrator class MyKryoRegistrator extends KryoRegistrator { override def registerClasses(kryo: Kryo): Unit = { kryo.register(Class.forName("[[B")) // byte[][] kry

Re: intermittent Kryo serialization failures in Spark

. org.apache.spark.SparkException: Failed to register classes with Kryo > at > org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140) > at > org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializ

intermittent Kryo serialization failures in Spark

Hi all, I am experiencing a strange intermittent failure of my Spark job that results from serialization issues in Kryo. Here is the stack trace: Caused by: java.lang.ClassNotFoundException: com.mycompany.models.MyModel > at java.net.URLClassLoader.findClass(URLClassLoader.java:

Question regarding kryo and java encoders in datasets

spark Web UI. Useful part - ID RDD Name Size in Memory 2 LocalTableScan [value#0] 56.5 MB 13 LocalTableScan [age#6, id#7L, name#8, salary#9L] 23.3 MB Few questions: * Shouldn't size of kryo serialized RDD be less than

Internal Spark class is not registered by Kryo

Hi all: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register() and

Internal Spark class is not registered by Kryo

Hi all: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register() and

Internal Spark class is not registered by Kryo

Hi developers: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by

Internal Spark class is not registered by Kryo

Hi developers: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register()

Internal Spark class is not registered by Kryo

Hi developers: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register()

Spark internal class is not registered by Kryo

Hi developers: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register()

Internal Spark class is not registered by Kryo

Hi developers: I have set spark.kryo.registrationRequired=true, but an exception occured: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run the program. I tried to register it manually by kryo.register()

How Kryo serializer allocates buffer in Spark

failure: Lost task 3.9 in stage 4.0 (TID 16, iadprd01mpr005.mgmt-a.xactly.iad.dc.local): org.apache.spark.SparkException: *Kryo serialization failed: Buffer overflow. Available: 0, required: 19. To avoid this, increase spark.kryoserializer.buffer.max value

Re: Kryo serialization failed: Buffer overflow : Broadcast Join

I am using spark 2.1.0 On Fri, Feb 2, 2018 at 5:08 PM, Pralabh Kumar wrote: > Hi > > I am performing broadcast join where my small table is 1 gb . I am > getting following error . > > I am using > > > org.apache.spark.SparkException: > . Available: 0, required:

Kryo serialization failed: Buffer overflow : Broadcast Join

Hi I am performing broadcast join where my small table is 1 gb . I am getting following error . I am using org.apache.spark.SparkException: . Available: 0, required: 28869232. To avoid this, increase spark.kryoserializer.buffer.max value I increase the value to

Re: Kryo not registered class

with scala and I'm registering all classes > with kryo, and I have a problem registering this class, > > org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$ > SerializableFileStatus$SerializableBlockLocation[] > > I can't register with classOf[Arra

Kryo not registered class

Hello, I'm with spark 2.1.0 with scala and I'm registering all classes with kryo, and I have a problem registering this class, org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[] I can't register with classOf[Array

spark 2.1.1 ml.LogisticRegression with large feature set cause Kryo serialization failed: Buffer overflow

, the spark.driver.maxResultSize is 40G. The execution fails with the following messages: +WARN TaskSetManager: Lost task 2.1 in stage 25.0 (TID 1415, Blackstone064183, executor 15): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 3, required: 8 Serialization trace: currMin

Fwd: Saving RDD as Kryo (broken in 2.1)

Hi, all! I have a code, serializing RDD as Kryo, and saving it as sequence file. It works fine in 1.5.1, but, while switching to 2.1.1 it does not work. I am trying to serialize RDD of Tuple2<> (got from PairRDD). 1. RDD consists of different heterogeneous objects (aggregates, like

Saving RDD as Kryo (broken in 2.1)

Hi, all! I have a code, serializing RDD as Kryo, and saving it as sequence file. It works fine in 1.5.1, but, while switching to 2.1.1 it does not work. I am trying to serialize RDD of Tuple2<> (got from PairRDD). 1. RDD consists of different heterogeneous objects (aggregates, like

Saving RDD as Kryo (broken in 2.1)

Hi, all! I have a code, serializing RDD as Kryo, and saving it as sequence file. It works fine in 1.5.1, but, while switching to 2.1.1 it does not work. I am trying to serialize RDD of Tuple2<> (got from PairRDD). 1. RDD consists of different heterogeneous objects (aggregates, like

Re: dataset aggregators with kryo encoder very slow

in spark itself i can for v2.1.0 but not for >> master. i guess it has been fixed >> >> On Fri, Jan 20, 2017 at 4:57 PM, Koert Kuipers <ko...@tresata.com> wrote: >> >>> i started printing out when kryo serializes my buffer data structure for >>> my aggr

Re: dataset aggregators with kryo encoder very slow

<ko...@tresata.com> wrote: > >> i started printing out when kryo serializes my buffer data structure for >> my aggregator. >> >> i would expect every buffer object to ideally get serialized only once: >> at the end of the map-side before the shuffle (so after all

Re: dataset aggregators with kryo encoder very slow

trying to replicate this in spark itself i can for v2.1.0 but not for master. i guess it has been fixed On Fri, Jan 20, 2017 at 4:57 PM, Koert Kuipers <ko...@tresata.com> wrote: > i started printing out when kryo serializes my buffer data structure for > my aggregator. > > i

Kryo (with Spark 1.6.3) class registration slows down processing

Hello, Here is something I am unable to explain and goes against Kryo's documentation, numerous suggestions on the web and on this list as well as pure intuition. Our Spark application runs in a single JVM (perhaps this is relevant, hence mentioning it). We have been using Kryo serialization

Re: dataset aggregators with kryo encoder very slow

i started printing out when kryo serializes my buffer data structure for my aggregator. i would expect every buffer object to ideally get serialized only once: at the end of the map-side before the shuffle (so after all the values for the given key within the partition have been reduced

dataset aggregators with kryo encoder very slow

we just converted a job from RDD to Dataset. the job does a single map-red phase using aggregators. we are seeing very bad performance for the Dataset version, about 10x slower. in the Dataset version we use kryo encoders for some of the aggregators. based on some basic profiling of spark

Re: Kryo On Spark 1.6.0

f you don’t mind, could please share me with the scala solution? I tried > to use kryo but seamed not work at all. I hope to get some practical > example. THX > > On 2017年1月10日, at 19:10, Enrico DUrso <enrico.du...@everis.com> wrote: > > Hi, > > I am trying to use Kry

Re: Sporadic ClassNotFoundException with Kryo

I faced a similar issue and had to do two things; 1. Submit Kryo jar with the spark-submit 2. Set spark.executor.userClassPathFirst true in Spark conf On Fri, Nov 18, 2016 at 7:39 PM, chrism <christopher.martens...@cics.se> wrote: > Regardless of the different ways we have tried deploy

Kryo and Spark 1.6.0 - Does it require a default empty constructor?

Hi, I have a doubt about Kryo and Spark 1.6.0. I read that for using Kryo, the class that you want to serialize must have a default constructor. I created a simple class avoiding to insert such a constructor and If I try to serialize manually, it does not work. But If I use that class in Spark

RE: Kryo On Spark 1.6.0 [Solution in this email]

Yes sure, you can find it here: http://stackoverflow.com/questions/34736587/kryo-serializer-causing-exception-on-underlying-scala-class-wrappedarray hope it works, I did not try, I am using Java. To be precise I found the solution for my problem: To sum up, I had problems in registering

Re: Kryo On Spark 1.6.0

If you don’t mind, could please share me with the scala solution? I tried to use kryo but seamed not work at all. I hope to get some practical example. THX > On 2017年1月10日, at 19:10, Enrico DUrso <enrico.du...@everis.com> wrote: > > Hi, > > I am trying to use Kryo on Spa

RE: Kryo On Spark 1.6.0

in according to how Spark works. How can I register all those classes? cheers, From: Richard Startin [mailto:richardstar...@outlook.com] Sent: 10 January 2017 11:18 To: Enrico DUrso; user@spark.apache.org Subject: Re: Kryo On Spark 1.6.0 Hi Enrico, Only set spark.kryo.registrationRequired if you want

Re: Kryo On Spark 1.6.0

tml> spark.apache.org Spark Configuration. Spark Properties. Dynamically Loading Spark Properties; Viewing Spark Properties; Available Properties. Application Properties; Runtime Environment To enable kryo, you just need spark.serializer=org.apache.spark.serializer.KryoSerializer. There is some info

Kryo On Spark 1.6.0

Hi, I am trying to use Kryo on Spark 1.6.0. I am able to register my own classes and it works, but when I set "spark.kryo.registrationRequired " to true, I get an error about a scala class: "Class is not registered: scala.collection.mutable.WrappedArray$ofRef". Any of yo

Re: Spark kryo serialization register Datatype[]

I already set .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") to enable kryo and .set("spark.kryo.registrationRequired", "true") to force kryo. Strangely, I see the issue of this missing Dataset[] Trying to register regular class

Re: Spark kryo serialization register Datatype[]

to enable kryo serializer you just need to pass `spark.serializer=org.apache.spark.serializer.KryoSerializer` the `spark.kryo.registrationRequired` controls the following behavior: Whether to require registration with Kryo. If set to 'true', Kryo will > throw an exception if an unregiste

Spark kryo serialization register Datatype[]

To force spark to use kryo serialization I set spark.kryo.registrationRequired to true. Now spark complains that: Class is not registered: org.apache.spark.sql.types.DataType[] is not registered. How can I fix this? So far I could not successfully register this class. -- View this message

Kryo Exception: NegativeArraySizeException

Hi, I'm trying to broadcast a map of 2.6GB but I'm getting a weird Kryo exception. I tried to set -XX:hashCode=0 in executor and driver, following this copmment: https://github.com/broadinstitute/gatk/issues/1524#issuecomment-189368808 But it didn't change anything. Are you aware of this problem

Sporadic ClassNotFoundException with Kryo

Regardless of the different ways we have tried deploying a jar together with Spark, when running a Spark Streaming job with Kryo as serializer on top of Mesos, we sporadically get the following error (I have truncated a bit): /16/11/18 08:39:10 ERROR OneForOneBlockFetcher: Failed while starting

BiMap BroadCast Variable - Kryo Serialization Issue

Hi, I am getting Nullpointer exception due to Kryo Serialization issue, while trying to read a BiMap broadcast variable. Attached is the code snippets. Pointers shared here didn't help - link1 <http://stackoverflow.com/questions/33156095/spark-serialization-issue-with-hashmap>, link2

Kryo on Zeppelin

Hi All, I am running some spark scala code on zeppelin on CDH 5.5.1 (Spark version 1.5.0). I customized the Spark interpreter to use org.apache.spark. serializer.KryoSerializer as spark.serializer. And in the dependency I added Kyro-3.0.3 as following: com.esotericsoftware:kryo:3.0.3 When I

Re: mutable.LinkedHashMap kryo serialization issues

quot;) 3. sparkConf().set("spark.kryo.registrator", "registrator1, registrator2") In the first two methods, which set the classes to register in Kryo, what I get are empty mutable.LinkedHashMaps after calling collect on the RDD. To my best understanding this should not happen (non

Re: mutable.LinkedHashMap kryo serialization issues

zes differ because the elements of the LinkedHashMap were never copied over Anyway I think I've tracked down the issue and it doesn't seem to be a spark or kryo issue. For those it concerns LinkedHashMap has this serialization issue because it has transient members for firstEntry and lastEntry. T

Re: mutable.LinkedHashMap kryo serialization issues

Hi Rahul, You have probably already figured this one out, but anyway... You need to register the classes that you'll be using with Kryo because it does not support all Serializable types and requires you to register the classes you’ll use in the program in advance. So when you don't register

Re:Do we still need to use Kryo serializer in Spark 1.6.2 ?

The way to use Kryo serializer is similar as Scala, like below, the only different is lack of convenient method "conf.registerKryoClasses", but it should be easy to make one by yourself conf=SparkConf() conf.set("spark.serializer", "org.apache.spark.serializer.

Re: Do we still need to use Kryo serializer in Spark 1.6.2 ?

Hi, I've not heard this. And moreover I see Kryo supported in Encoders (SerDes) in Spark 2.0. https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/Encoders.scala#L151 Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache

Do we still need to use Kryo serializer in Spark 1.6.2 ?

I heard that Kryo will get phased out at some point but not sure which Spark release. I'm using PySpark, does anyone has any docs on how to call / use Kryo Serializer in PySpark ? Thanks. -- -eric ho

mutable.LinkedHashMap kryo serialization issues

Hi, Just sending this again to see if others have had this issue. I recently switched to using kryo serialization and I've been running into errors with the mutable.LinkedHashMap class. If I don't register the mutable.LinkedHashMap class then I get an ArrayStoreException seen below. If I do

mutable.LinkedHashMap kryo serialization issues

Hi, I recently switched to using kryo serialization and I've been running into errors with the mutable.LinkedHashMap class. If I don't register the mutable.LinkedHashMap class then I get an ArrayStoreException seen below. If I do register the class, then when the LinkedHashMap is collected

Spark, Kryo Serialization Issue with ProtoBuf field

Hi, I am seeing an error when running my spark job relating to Serialization of a protobuf field when transforming an RDD. com.esotericsoftware.kryo.KryoException: java.lang.UnsupportedOperationException Serialization trace: otherAuthors_

Re: Kryo ClassCastException during Serialization/deserialization in Spark Streaming

sire...@gmail.com> wrote: > >> Hi, >> >> I keep getting the following error in my Spark Streaming every now and >> then >> after the job runs for say around 10 hours. I have those 2 classes >> registered in kryo as shown below. sampleMap is a field in

Re: Kryo ClassCastException during Serialization/deserialization in Spark Streaming

ve those 2 classes > registered in kryo as shown below. sampleMap is a field in SampleSession > as shown below. Any suggestion as to how to avoid this would be of great > help!! > > public class SampleSession implements Serializable, Cloneable{ >private Map<String, Sample> sampleM

Kryo ClassCastException during Serialization/deserialization in Spark Streaming

Hi, I keep getting the following error in my Spark Streaming every now and then after the job runs for say around 10 hours. I have those 2 classes registered in kryo as shown below. sampleMap is a field in SampleSession as shown below. Any suggestion as to how to avoid this would be of great

Re: Dataset kryo encoder fails on Collections$UnmodifiableCollection

t; On Sun, May 22, 2016 at 2:50 PM, Amit Sela <amitsel...@gmail.com> wrote: > >> I've been using Encoders with Kryo to support encoding of generically >> typed Java classes, mostly with success, in the following manner: >> >> public static Encoder encoder() { >>

Re: Dataset kryo encoder fails on Collections$UnmodifiableCollection

Can you open a JIRA? On Sun, May 22, 2016 at 2:50 PM, Amit Sela <amitsel...@gmail.com> wrote: > I've been using Encoders with Kryo to support encoding of generically > typed Java classes, mostly with success, in the following manner: > > public static Encoder enco

Re: kryo

e.DateTimeZone.convertUTCToLocal(DateTimeZone.java:925) > > > > > > Any ideas? > > > > Thanks > > > > > > *From:* Ted Yu [mailto:yuzhih...@gmail.com] > *Sent:* May-11-16 5:32 PM > *To:* Younes Naguib > *Cc:* user@spark.apache.org > *Subject:* Re: kr

RE: kryo

) at org.joda.time.DateTimeZone.convertUTCToLocal(DateTimeZone.java:925) Any ideas? Thanks From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: May-11-16 5:32 PM To: Younes Naguib Cc: user@spark.apache.org Subject: Re: kryo Have you seen this thread ? http://search-hadoop.com/m/q3RTtpO0qI3cp06/JodaDateTimeSerializer+spark

Re: kryo

alizer. > I set it in the spark-default.conf, but I statred getting issues with > datetimes. > > As I understand, I need to disable it. > Anyways to keep using kryo? > > It's seems I can use JodaDateTimeSerializer for datetimes, just not sure > how to set it, and register i

kryo

Hi all, I'm trying to get to use spark.serializer. I set it in the spark-default.conf, but I statred getting issues with datetimes. As I understand, I need to disable it. Anyways to keep using kryo? It's seems I can use JodaDateTimeSerializer for datetimes, just not sure how to set

Re: How to verify if spark is using kryo serializer for shuffle

As far as I know driver is just driving shuffle operation but not >>>>> actually doing anything within its own system that will cause memory >>>>> issue. >>>>> Can you explain in what circumstances I could see this error in driv

Re: How to verify if spark is using kryo serializer for shuffle

ue. >>>> Can you explain in what circumstances I could see this error in driver >>>> logs? I don't do any collect or any other driver operation that would cause >>>> this. It fails when doing aggregateByKey operation but that should happen >&g

Re: How to verify if spark is using kryo serializer for shuffle

m that will cause memory issue. Can you >>>> explain in what circumstances I could see this error in driver logs? I >>>> don't do any collect or any other driver operation that would cause this. >>>> It fails when doing aggregateByKey operation but that should happen in

Re: How to verify if spark is using kryo serializer for shuffle

fails when doing aggregateByKey operation but that should happen >>> in executor JVM NOT in driver JVM. >>> >>> >>> Thanks >>> >>> On Sat, May 7, 2016 at 11:58 AM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>&

Re: How to verify if spark is using kryo serializer for shuffle

ih...@gmail.com> wrote: >> >>> bq. at akka.serialization.JavaSerializer.toBinary( >>> Serializer.scala:129) >>> >>> It was Akka which uses JavaSerializer >>> >>> Cheers >>> >>> On Sat, May 7, 2016 at

Re: How to verify if spark is using kryo serializer for shuffle

JavaSerializer >> >> Cheers >> >> On Sat, May 7, 2016 at 11:13 AM, Nirav Patel <npa...@xactlycorp.com >> <javascript:_e(%7B%7D,'cvml','npa...@xactlycorp.com');>> wrote: >> >>> Hi, >>> >>> I thou

Re: How to verify if spark is using kryo serializer for shuffle

16 at 11:13 AM, Nirav Patel <npa...@xactlycorp.com> > wrote: > >> Hi, >> >> I thought I was using kryo serializer for shuffle. I could verify it >> from spark UI - Environment tab that >> spark.serializer

Re: How to verify if spark is using kryo serializer for shuffle

bq. at akka.serialization.JavaSerializer.toBinary(Serializer.scala:129) It was Akka which uses JavaSerializer Cheers On Sat, May 7, 2016 at 11:13 AM, Nirav Patel <npa...@xactlycorp.com> wrote: > Hi, > > I thought I was using kryo serializer for shuffle. I could verify it

How to verify if spark is using kryo serializer for shuffle

Hi, I thought I was using kryo serializer for shuffle. I could verify it from spark UI - Environment tab that spark.serializer org.apache.spark.serializer.KryoSerializer spark.kryo.registrator com.myapp.spark.jobs.conf.SparkSerializerRegistrator But when I see following error in Driver logs

Re: Kryo serialization mismatch in spark sql windowing function

rt.maxRetries -> 999 spark.executor.port -> 45250 spark.driver.extraClassPath -> ... On Wed, Apr 6, 2016 at 6:59 PM, Josh Rosen <joshro...@databricks.com> wrote: > > Spark is compiled against a custom fork of Hive 1.2.1 which added shading > of Protobuf and removed shading

Re: Kryo serialization mismatch in spark sql windowing function

Spark is compiled against a custom fork of Hive 1.2.1 which added shading of Protobuf and removed shading of Kryo. What I think that what's happening here is that stock Hive 1.2.1 is taking precedence so the Kryo instance that it's returning is an instance of shaded/relocated Hive version rather

Kryo serialization mismatch in spark sql windowing function

Hi folks, I have a build of Spark 1.6.1 on which spark sql seems to be functional outside of windowing functions. For example, I can create a simple external table via Hive: CREATE EXTERNAL TABLE PSTable (pid int, tty string, time string, cmd string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','

DataFrames - Kryo registration issue

Hello All, If Kryo serialization is enabled, doesn't Spark take care of registration of built-in classes, i.e., are we not supposed to register just the custom classes? When using DataFrames, this does not seem to be the case. I had to register the following classes conf.registerKryoClasses

Re: Spark Streaming 1.6 mapWithState not working well with Kryo Serialization

folks and especially TD, > > I am using the Spark Streaming 1.6 mapWithState API, and I am trying to > enforce Kryo Serialization with > > SparkConf.set("spark.kryo.registrationRequired", "true") > > However, this appears to b

Spark Streaming 1.6 mapWithState not working well with Kryo Serialization

Hello Spark folks and especially TD, I am using the Spark Streaming 1.6 mapWithState API, and I am trying to enforce Kryo Serialization with SparkConf.set("spark.kryo.registrationRequired", "true") However, this appears to be impossible! I registered all the classes th

Re: Kryo serializer Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException:

Could you disable `spark.kryo.registrationRequired`? Some classes may not be registered but they work well with Kryo's default serializer. On Fri, Jan 8, 2016 at 8:58 AM, Ted Yu wrote: > bq. try adding scala.collection.mutable.WrappedArray > > But the hint said registering

Re: Kryo serializer Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException:

(point of post is to see if anyone has ideas about errors at end of post) In addition, the real way to test if it's working is to force serialization: In Java: Create array of all your classes: // for kyro serializer it wants to register all classes that need to be serialized Class[]

Re: Kryo serializer Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException:

bq. try adding scala.collection.mutable.WrappedArray But the hint said registering scala.collection.mutable.WrappedArray$ofRef.class , right ? On Fri, Jan 8, 2016 at 8:52 AM, jiml wrote: > (point of post is to see if anyone has ideas about errors at end of post) > >

Kryo serialization fails when using SparkSQL and HiveContext

Hi everyone, I'm using HiveContext and SparkSQL to query a Hive table and doing join operation on it. After changing the default serializer to Kryo with spark.kryo.registrationRequired = true, the Spark application failed with the following error: java.lang.IllegalArgumentException: Class

Re: Kryo serialization fails when using SparkSQL and HiveContext

wrote: > Hi everyone, > I'm using HiveContext and SparkSQL to query a Hive table and doing join > operation on it. > After changing the default serializer to Kryo with > spark.kryo.registrationRequired = true, the Spark application failed with > the following error: > > java

Re: Kryo Serialization in Spark

Are you sure you are using Kryo serialization. You are getting a java serialization error. Are you setting up your sparkcontext with kryo serialization enabled? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-Serialization-in-Spark-tp25628p25678.html

hbase Put object kryo serialisation error

I have registerd Put,and TreeMap in spark kryo serialiser. whats the issue of this error ? Full stack trace is : Exception in thread "main" com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 100 Serialization trace: familyMap (org.apache.hadoop.

Kryo Serialization in Spark

Hi All, I'm unable to use Kryo serializer in my Spark program. I'm loading a graph from an edgelist file using GraphLoader and performing a BFS using pregel API. But I get the below mentioned error while I'm running. Can anybody tell me what is the right way to serialize a class in Spark and what

ClassCastException in Kryo Serialization

Hi, I seem to be getting class cast exception in Kryo Serialization. Following is the error. Child1 class is a map in parent class. Child1 has a hashSet testObjects of the type Object1. I get an error when it tries to deserialize Object1. Any idea as to why this is happening

Re: Why is Kryo not the default serializer?

Array issue was also discussed in Apache Hive forum. This problem seems like it can be resolved by using Kryo 3.x. Will upgrading to Kryo 3.x allow Kryo to become the default SerDes? https://issues.apache.org/jira/browse/HIVE-12174

Why is Kryo not the default serializer?

If Kryo usage is recommended, why is Java serialization the default serializer instead of Kryo? Is there some limitation to using Kryo? I've read through the documentation but it just seem Kryo is a better choice and should be made a default. -- View this message in context: http://apache

Re: Why is Kryo not the default serializer?

I have seen some failures in our workloads with Kryo, one I remember is a scenario with very large arrays. We could not get Kryo to work despite using the different configuration properties. Switching to java serde was what worked. Regards Sab On Tue, Nov 10, 2015 at 11:43 AM, Hitoshi Ozawa

Kryo makes String data invalid

Hi all, I have a parquet file, which I am loading in a shell. When I launch the shell with -driver-java-options ="-Dspark.serializer=...kryo", makes a couple fields look like: 03-?? ??-?? ??-??? when calling > data.first I will confirm briefly, but I am utterly su

Enabling kryo serialization slows down machine learning app.

Hi, my team is setting up a machine-learning framework based on Spark's mlib, that currently uses logistic regression. I enabled Kryo serialization and enforced class registration, so I know that all the serialized classes are registered. However, the running times when Kryo serialization

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

Hi Nick/Igor, ​​ Any solution for this ? Even I am having the same issue and copying jar to each executor is not feasible if we use lot of jars. Thanks, Vipul

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

ough Yarn. > Serialization is set to use Kryo. > > I have a large object which I send to the executors as a Broadcast. The > object seems to serialize just fine. When it attempts to deserialize, > though, Kryo throws a ClassNotFoundException... for a class that I include > in the

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

with same class > > On 8 September 2015 at 06:38, Nicholas R. Peterson <nrpeter...@gmail.com> > wrote: > >> I'm trying to run a Spark 1.4.1 job on my CDH5.4 cluster, through Yarn. >> Serialization is set to use Kryo. >> >> I have a large object which I

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

classloader kryo is using? On Tue, Sep 8, 2015 at 6:34 AM Igor Berman <igor.ber...@gmail.com> wrote: > java.lang.ClassNotFoundException: com.i2028.Document.Document > > 1. so have you checked that jar that you create(fat jar) contains this class? > > 2. might be there is some stale c

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

, Sep 8, 2015 at 1:14 AM Igor Berman <igor.ber...@gmail.com> >>> wrote: >>> >>>> as a starting point, attach your stacktrace... >>>> ps: look for duplicates in your classpath, maybe you include another >>>> jar with same class >

Re: Spark on Yarn: Kryo throws ClassNotFoundException for class included in fat jar

;>>> >>>> This seems like a likely culprit. What could cause this, and how can I >>>> fix it? >>>> >>>> Best, >>>> Nick >>>> >>>> On Tue, Sep 8, 2015 at 1:14 AM Igor Berman <igor.ber...@gmail.com&

  1   2   3   >