I did some basic testing of multi-source queries with the most recent Spark:
https://github.com/GavinRay97/spark-playground/blob/44a756acaee676a9b0c128466e4ab231a7df8d46/src/main/scala/Application.scala#L46-L115
The output of "spark.time()" surprised me:
SELECT p.id, p.name, t.id, t.title
FROM
Hello all,
Is there a way to register classes within a datasourcev2 implementation in
the Kryo serializer?
I've attempted the following in both the constructor and static block of my
toplevel class:
SparkContext context = SparkContext.getOrCreate();
SparkConf conf
trator
>>
>> class MyKryoRegistrator extends KryoRegistrator {
>> override def registerClasses(kryo: Kryo): Unit = {
>> kryo.register(Class.forName("[[B")) // byte[][]
>> kryo.register(classOf[java.lang.Class[_]])
>> }
>> }
>&g
tends KryoRegistrator {
> override def registerClasses(kryo: Kryo): Unit = {
> kryo.register(Class.forName("[[B")) // byte[][]
> kryo.register(classOf[java.lang.Class[_]])
> }
> }
> ```
>
> then run with
>
> 'spark.kryo.referenceTr
at this is a bit different from your code and I'm wondering
> whether there's any functional difference or if these are two ways to get
> to the same end. Our code is directly adapted from the Spark documentation
> on how to use the Kryo serializer but maybe there's some subtlety here that
&
.registerKryoClasses(KryoRegistrar.classesToRegister)
>
I notice that this is a bit different from your code and I'm wondering
whether there's any functional difference or if these are two ways to get
to the same end. Our code is directly adapted from the Spark documentation
on how to use the Kryo serialize
Pre-register your classes:
```
import com.esotericsoftware.kryo.Kryo
import org.apache.spark.serializer.KryoRegistrator
class MyKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo): Unit = {
kryo.register(Class.forName("[[B")) // byte[][]
kry
.
org.apache.spark.SparkException: Failed to register classes with Kryo
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:140)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializ
Hi all,
I am experiencing a strange intermittent failure of my Spark job that
results from serialization issues in Kryo. Here is the stack trace:
Caused by: java.lang.ClassNotFoundException: com.mycompany.models.MyModel
> at java.net.URLClassLoader.findClass(URLClassLoader.java:
spark Web UI. Useful part -
ID RDD Name Size in Memory
2 LocalTableScan [value#0] 56.5 MB
13 LocalTableScan [age#6, id#7L, name#8, salary#9L] 23.3 MB
Few questions:
* Shouldn't size of kryo serialized RDD be less than
Hi all:
I have set spark.kryo.registrationRequired=true, but an exception occured:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I
run the program.
I tried to register it manually by kryo.register() and
Hi all:
I have set spark.kryo.registrationRequired=true, but an exception occured:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run
the program.
I tried to register it manually by kryo.register() and
Hi developers:
I have set spark.kryo.registrationRequired=true, but an exception occured:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run
the program.
I tried to register it manually by
Hi developers:
I have set spark.kryo.registrationRequired=true, but an exception occured:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run
the program.
I tried to register it manually by kryo.register()
Hi developers:
I have set spark.kryo.registrationRequired=true, but an exception occured:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run
the program.
I tried to register it manually by kryo.register()
Hi developers:
I have set spark.kryo.registrationRequired=true, but an exception occured:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run
the program.
I tried to register it manually by kryo.register()
Hi developers:
I have set spark.kryo.registrationRequired=true, but an exception occured:
java.lang.IllegalArgumentException: Class is not registered:
org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage when I run
the program.
I tried to register it manually by kryo.register()
failure: Lost task 3.9 in stage 4.0 (TID 16,
iadprd01mpr005.mgmt-a.xactly.iad.dc.local): org.apache.spark.SparkException:
*Kryo serialization failed: Buffer overflow. Available: 0, required: 19. To
avoid this, increase spark.kryoserializer.buffer.max value
I am using spark 2.1.0
On Fri, Feb 2, 2018 at 5:08 PM, Pralabh Kumar
wrote:
> Hi
>
> I am performing broadcast join where my small table is 1 gb . I am
> getting following error .
>
> I am using
>
>
> org.apache.spark.SparkException:
> . Available: 0, required:
Hi
I am performing broadcast join where my small table is 1 gb . I am getting
following error .
I am using
org.apache.spark.SparkException:
. Available: 0, required: 28869232. To avoid this, increase
spark.kryoserializer.buffer.max value
I increase the value to
with scala and I'm registering all classes
> with kryo, and I have a problem registering this class,
>
> org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$
> SerializableFileStatus$SerializableBlockLocation[]
>
> I can't register with classOf[Arra
Hello, I'm with spark 2.1.0 with scala and I'm registering all classes with
kryo, and I have a problem registering this class,
org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[]
I can't register with
classOf[Array
, the spark.driver.maxResultSize is
40G.
The execution fails with the following messages:
+WARN TaskSetManager: Lost task 2.1 in stage 25.0 (TID 1415,
Blackstone064183, executor 15): org.apache.spark.SparkException: Kryo
serialization failed: Buffer overflow. Available: 3, required: 8
Serialization trace:
currMin
Hi, all!
I have a code, serializing RDD as Kryo, and saving it as sequence file. It
works fine in 1.5.1, but, while switching to 2.1.1 it does not work.
I am trying to serialize RDD of Tuple2<> (got from PairRDD).
1. RDD consists of different heterogeneous objects (aggregates, like
Hi, all!
I have a code, serializing RDD as Kryo, and saving it as sequence file. It
works fine in 1.5.1, but, while switching to 2.1.1 it does not work.
I am trying to serialize RDD of Tuple2<> (got from PairRDD).
1. RDD consists of different heterogeneous objects (aggregates, like
Hi, all!
I have a code, serializing RDD as Kryo, and saving it as sequence file. It
works fine in 1.5.1, but, while switching to 2.1.1 it does not work.
I am trying to serialize RDD of Tuple2<> (got from PairRDD).
1. RDD consists of different heterogeneous objects (aggregates, like
in spark itself i can for v2.1.0 but not for
>> master. i guess it has been fixed
>>
>> On Fri, Jan 20, 2017 at 4:57 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> i started printing out when kryo serializes my buffer data structure for
>>> my aggr
<ko...@tresata.com> wrote:
>
>> i started printing out when kryo serializes my buffer data structure for
>> my aggregator.
>>
>> i would expect every buffer object to ideally get serialized only once:
>> at the end of the map-side before the shuffle (so after all
trying to replicate this in spark itself i can for v2.1.0 but not for
master. i guess it has been fixed
On Fri, Jan 20, 2017 at 4:57 PM, Koert Kuipers <ko...@tresata.com> wrote:
> i started printing out when kryo serializes my buffer data structure for
> my aggregator.
>
> i
Hello,
Here is something I am unable to explain and goes against Kryo's
documentation, numerous suggestions on the web and on this list as well as
pure intuition.
Our Spark application runs in a single JVM (perhaps this is relevant, hence
mentioning it). We have been using Kryo serialization
i started printing out when kryo serializes my buffer data structure for my
aggregator.
i would expect every buffer object to ideally get serialized only once: at
the end of the map-side before the shuffle (so after all the values for the
given key within the partition have been reduced
we just converted a job from RDD to Dataset. the job does a single map-red
phase using aggregators. we are seeing very bad performance for the Dataset
version, about 10x slower.
in the Dataset version we use kryo encoders for some of the aggregators.
based on some basic profiling of spark
f you don’t mind, could please share me with the scala solution? I tried
> to use kryo but seamed not work at all. I hope to get some practical
> example. THX
>
> On 2017年1月10日, at 19:10, Enrico DUrso <enrico.du...@everis.com> wrote:
>
> Hi,
>
> I am trying to use Kry
I faced a similar issue and had to do two things;
1. Submit Kryo jar with the spark-submit
2. Set spark.executor.userClassPathFirst true in Spark conf
On Fri, Nov 18, 2016 at 7:39 PM, chrism <christopher.martens...@cics.se>
wrote:
> Regardless of the different ways we have tried deploy
Hi,
I have a doubt about Kryo and Spark 1.6.0.
I read that for using Kryo, the class that you want to serialize must have a
default constructor.
I created a simple class avoiding to insert such a constructor and If I try to
serialize manually, it does not work.
But If I use that class in Spark
Yes sure,
you can find it here:
http://stackoverflow.com/questions/34736587/kryo-serializer-causing-exception-on-underlying-scala-class-wrappedarray
hope it works, I did not try, I am using Java.
To be precise I found the solution for my problem:
To sum up, I had problems in registering
If you don’t mind, could please share me with the scala solution? I tried to
use kryo but seamed not work at all. I hope to get some practical example. THX
> On 2017年1月10日, at 19:10, Enrico DUrso <enrico.du...@everis.com> wrote:
>
> Hi,
>
> I am trying to use Kryo on Spa
in according to
how Spark works.
How can I register all those classes?
cheers,
From: Richard Startin [mailto:richardstar...@outlook.com]
Sent: 10 January 2017 11:18
To: Enrico DUrso; user@spark.apache.org
Subject: Re: Kryo On Spark 1.6.0
Hi Enrico,
Only set spark.kryo.registrationRequired if you want
tml>
spark.apache.org
Spark Configuration. Spark Properties. Dynamically Loading Spark Properties;
Viewing Spark Properties; Available Properties. Application Properties; Runtime
Environment
To enable kryo, you just need
spark.serializer=org.apache.spark.serializer.KryoSerializer. There is some info
Hi,
I am trying to use Kryo on Spark 1.6.0.
I am able to register my own classes and it works, but when I set
"spark.kryo.registrationRequired " to true, I get an error about a scala class:
"Class is not registered: scala.collection.mutable.WrappedArray$ofRef".
Any of yo
I already set
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
to enable kryo and
.set("spark.kryo.registrationRequired", "true")
to force kryo. Strangely, I see the issue of this missing Dataset[]
Trying to register regular class
to enable kryo serializer you just need to pass
`spark.serializer=org.apache.spark.serializer.KryoSerializer`
the `spark.kryo.registrationRequired` controls the following behavior:
Whether to require registration with Kryo. If set to 'true', Kryo will
> throw an exception if an unregiste
To force spark to use kryo serialization I set
spark.kryo.registrationRequired to true.
Now spark complains that: Class is not registered:
org.apache.spark.sql.types.DataType[] is not registered.
How can I fix this? So far I could not successfully register this class.
--
View this message
Hi, I'm trying to broadcast a map of 2.6GB but I'm getting a weird Kryo
exception.
I tried to set -XX:hashCode=0 in executor and driver, following this
copmment:
https://github.com/broadinstitute/gatk/issues/1524#issuecomment-189368808
But it didn't change anything.
Are you aware of this problem
Regardless of the different ways we have tried deploying a jar together with
Spark, when running a Spark Streaming job with Kryo as serializer on top of
Mesos, we sporadically get the following error (I have truncated a bit):
/16/11/18 08:39:10 ERROR OneForOneBlockFetcher: Failed while starting
Hi,
I am getting Nullpointer exception due to Kryo Serialization issue, while
trying to read a BiMap broadcast variable. Attached is the code snippets.
Pointers shared here didn't help - link1
<http://stackoverflow.com/questions/33156095/spark-serialization-issue-with-hashmap>,
link2
Hi All,
I am running some spark scala code on zeppelin on CDH 5.5.1 (Spark version
1.5.0). I customized the Spark interpreter to use org.apache.spark.
serializer.KryoSerializer as spark.serializer. And in the dependency I
added Kyro-3.0.3 as following:
com.esotericsoftware:kryo:3.0.3
When I
quot;)
3. sparkConf().set("spark.kryo.registrator", "registrator1, registrator2")
In the first two methods, which set the classes to register in Kryo,
what I get are empty mutable.LinkedHashMaps after calling collect on the
RDD.
To my best understanding this should not happen (non
zes differ because the elements of the LinkedHashMap were never
copied over
Anyway I think I've tracked down the issue and it doesn't seem to be a
spark or kryo issue.
For those it concerns LinkedHashMap has this serialization issue because it
has transient members for firstEntry and lastEntry.
T
Hi Rahul,
You have probably already figured this one out, but anyway...
You need to register the classes that you'll be using with Kryo because it
does not support all Serializable types and requires you to register the
classes you’ll use in the program in advance. So when you don't register
The way to use Kryo serializer is similar as Scala, like below, the only
different is lack of convenient method "conf.registerKryoClasses", but it
should be easy to make one by yourself
conf=SparkConf()
conf.set("spark.serializer", "org.apache.spark.serializer.
Hi,
I've not heard this. And moreover I see Kryo supported in Encoders
(SerDes) in Spark 2.0.
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/Encoders.scala#L151
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache
I heard that Kryo will get phased out at some point but not sure which
Spark release.
I'm using PySpark, does anyone has any docs on how to call / use Kryo
Serializer in PySpark ?
Thanks.
--
-eric ho
Hi,
Just sending this again to see if others have had this issue.
I recently switched to using kryo serialization and I've been running into
errors
with the mutable.LinkedHashMap class.
If I don't register the mutable.LinkedHashMap class then I get an
ArrayStoreException seen below.
If I do
Hi,
I recently switched to using kryo serialization and I've been running into
errors
with the mutable.LinkedHashMap class.
If I don't register the mutable.LinkedHashMap class then I get an
ArrayStoreException seen below.
If I do register the class, then when the LinkedHashMap is collected
Hi,
I am seeing an error when running my spark job relating to Serialization of
a protobuf field when transforming an RDD.
com.esotericsoftware.kryo.KryoException:
java.lang.UnsupportedOperationException Serialization trace: otherAuthors_
sire...@gmail.com> wrote:
>
>> Hi,
>>
>> I keep getting the following error in my Spark Streaming every now and
>> then
>> after the job runs for say around 10 hours. I have those 2 classes
>> registered in kryo as shown below. sampleMap is a field in
ve those 2 classes
> registered in kryo as shown below. sampleMap is a field in SampleSession
> as shown below. Any suggestion as to how to avoid this would be of great
> help!!
>
> public class SampleSession implements Serializable, Cloneable{
>private Map<String, Sample> sampleM
Hi,
I keep getting the following error in my Spark Streaming every now and then
after the job runs for say around 10 hours. I have those 2 classes
registered in kryo as shown below. sampleMap is a field in SampleSession
as shown below. Any suggestion as to how to avoid this would be of great
t; On Sun, May 22, 2016 at 2:50 PM, Amit Sela <amitsel...@gmail.com> wrote:
>
>> I've been using Encoders with Kryo to support encoding of generically
>> typed Java classes, mostly with success, in the following manner:
>>
>> public static Encoder encoder() {
>>
Can you open a JIRA?
On Sun, May 22, 2016 at 2:50 PM, Amit Sela <amitsel...@gmail.com> wrote:
> I've been using Encoders with Kryo to support encoding of generically
> typed Java classes, mostly with success, in the following manner:
>
> public static Encoder enco
e.DateTimeZone.convertUTCToLocal(DateTimeZone.java:925)
>
>
>
>
>
> Any ideas?
>
>
>
> Thanks
>
>
>
>
>
> *From:* Ted Yu [mailto:yuzhih...@gmail.com]
> *Sent:* May-11-16 5:32 PM
> *To:* Younes Naguib
> *Cc:* user@spark.apache.org
> *Subject:* Re: kr
)
at org.joda.time.DateTimeZone.convertUTCToLocal(DateTimeZone.java:925)
Any ideas?
Thanks
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: May-11-16 5:32 PM
To: Younes Naguib
Cc: user@spark.apache.org
Subject: Re: kryo
Have you seen this thread ?
http://search-hadoop.com/m/q3RTtpO0qI3cp06/JodaDateTimeSerializer+spark
alizer.
> I set it in the spark-default.conf, but I statred getting issues with
> datetimes.
>
> As I understand, I need to disable it.
> Anyways to keep using kryo?
>
> It's seems I can use JodaDateTimeSerializer for datetimes, just not sure
> how to set it, and register i
Hi all,
I'm trying to get to use spark.serializer.
I set it in the spark-default.conf, but I statred getting issues with datetimes.
As I understand, I need to disable it.
Anyways to keep using kryo?
It's seems I can use JodaDateTimeSerializer for datetimes, just not sure how to
set
As far as I know driver is just driving shuffle operation but not
>>>>> actually doing anything within its own system that will cause memory
>>>>> issue.
>>>>> Can you explain in what circumstances I could see this error in driv
ue.
>>>> Can you explain in what circumstances I could see this error in driver
>>>> logs? I don't do any collect or any other driver operation that would cause
>>>> this. It fails when doing aggregateByKey operation but that should happen
>&g
m that will cause memory issue. Can you
>>>> explain in what circumstances I could see this error in driver logs? I
>>>> don't do any collect or any other driver operation that would cause this.
>>>> It fails when doing aggregateByKey operation but that should happen in
fails when doing aggregateByKey operation but that should happen
>>> in executor JVM NOT in driver JVM.
>>>
>>>
>>> Thanks
>>>
>>> On Sat, May 7, 2016 at 11:58 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>>
>>>&
ih...@gmail.com> wrote:
>>
>>> bq. at akka.serialization.JavaSerializer.toBinary(
>>> Serializer.scala:129)
>>>
>>> It was Akka which uses JavaSerializer
>>>
>>> Cheers
>>>
>>> On Sat, May 7, 2016 at
JavaSerializer
>>
>> Cheers
>>
>> On Sat, May 7, 2016 at 11:13 AM, Nirav Patel <npa...@xactlycorp.com
>> <javascript:_e(%7B%7D,'cvml','npa...@xactlycorp.com');>> wrote:
>>
>>> Hi,
>>>
>>> I thou
16 at 11:13 AM, Nirav Patel <npa...@xactlycorp.com>
> wrote:
>
>> Hi,
>>
>> I thought I was using kryo serializer for shuffle. I could verify it
>> from spark UI - Environment tab that
>> spark.serializer
bq. at akka.serialization.JavaSerializer.toBinary(Serializer.scala:129)
It was Akka which uses JavaSerializer
Cheers
On Sat, May 7, 2016 at 11:13 AM, Nirav Patel <npa...@xactlycorp.com> wrote:
> Hi,
>
> I thought I was using kryo serializer for shuffle. I could verify it
Hi,
I thought I was using kryo serializer for shuffle. I could verify it from
spark UI - Environment tab that
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator com.myapp.spark.jobs.conf.SparkSerializerRegistrator
But when I see following error in Driver logs
rt.maxRetries -> 999
spark.executor.port -> 45250
spark.driver.extraClassPath -> ...
On Wed, Apr 6, 2016 at 6:59 PM, Josh Rosen <joshro...@databricks.com> wrote:
>
> Spark is compiled against a custom fork of Hive 1.2.1 which added shading
> of Protobuf and removed shading
Spark is compiled against a custom fork of Hive 1.2.1 which added shading
of Protobuf and removed shading of Kryo. What I think that what's happening
here is that stock Hive 1.2.1 is taking precedence so the Kryo instance
that it's returning is an instance of shaded/relocated Hive version rather
Hi folks,
I have a build of Spark 1.6.1 on which spark sql seems to be functional
outside of windowing functions. For example, I can create a simple external
table via Hive:
CREATE EXTERNAL TABLE PSTable (pid int, tty string, time string, cmd string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
Hello All,
If Kryo serialization is enabled, doesn't Spark take care of registration
of built-in classes, i.e., are we not supposed to register just the custom
classes?
When using DataFrames, this does not seem to be the case. I had to register
the following classes
conf.registerKryoClasses
folks and especially TD,
>
> I am using the Spark Streaming 1.6 mapWithState API, and I am trying to
> enforce Kryo Serialization with
>
> SparkConf.set("spark.kryo.registrationRequired", "true")
>
> However, this appears to b
Hello Spark folks and especially TD,
I am using the Spark Streaming 1.6 mapWithState API, and I am trying to
enforce Kryo Serialization with
SparkConf.set("spark.kryo.registrationRequired", "true")
However, this appears to be impossible! I registered all the classes th
Could you disable `spark.kryo.registrationRequired`? Some classes may not
be registered but they work well with Kryo's default serializer.
On Fri, Jan 8, 2016 at 8:58 AM, Ted Yu wrote:
> bq. try adding scala.collection.mutable.WrappedArray
>
> But the hint said registering
(point of post is to see if anyone has ideas about errors at end of post)
In addition, the real way to test if it's working is to force serialization:
In Java:
Create array of all your classes:
// for kyro serializer it wants to register all classes that need to be
serialized
Class[]
bq. try adding scala.collection.mutable.WrappedArray
But the hint said registering scala.collection.mutable.WrappedArray$ofRef.class
, right ?
On Fri, Jan 8, 2016 at 8:52 AM, jiml wrote:
> (point of post is to see if anyone has ideas about errors at end of post)
>
>
Hi everyone,
I'm using HiveContext and SparkSQL to query a Hive table and doing join
operation on it.
After changing the default serializer to Kryo with
spark.kryo.registrationRequired = true, the Spark application failed with
the following error:
java.lang.IllegalArgumentException: Class
wrote:
> Hi everyone,
> I'm using HiveContext and SparkSQL to query a Hive table and doing join
> operation on it.
> After changing the default serializer to Kryo with
> spark.kryo.registrationRequired = true, the Spark application failed with
> the following error:
>
> java
Are you sure you are using Kryo serialization.
You are getting a java serialization error.
Are you setting up your sparkcontext with kryo serialization enabled?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Kryo-Serialization-in-Spark-tp25628p25678.html
I have registerd Put,and TreeMap in spark kryo serialiser.
whats the issue of this error ?
Full stack trace is :
Exception in thread "main" com.esotericsoftware.kryo.KryoException:
Encountered unregistered class ID: 100
Serialization trace:
familyMap (org.apache.hadoop.
Hi All,
I'm unable to use Kryo serializer in my Spark program.
I'm loading a graph from an edgelist file using GraphLoader and performing a
BFS using pregel API.
But I get the below mentioned error while I'm running.
Can anybody tell me what is the right way to serialize a class in Spark and
what
Hi,
I seem to be getting class cast exception in Kryo Serialization. Following
is the error. Child1 class is a map in parent class. Child1 has a hashSet
testObjects of the type Object1. I get an error when it tries to
deserialize Object1. Any idea as to why this is happening
Array issue was also discussed in Apache Hive forum. This problem seems like it
can be resolved by using Kryo 3.x. Will upgrading to Kryo 3.x allow Kryo to
become the default SerDes?
https://issues.apache.org/jira/browse/HIVE-12174
If Kryo usage is recommended, why is Java serialization the default
serializer instead of Kryo? Is there some limitation to using Kryo? I've
read through the documentation but it just seem Kryo is a better choice and
should be made a default.
--
View this message in context:
http://apache
I have seen some failures in our workloads with Kryo, one I remember is a
scenario with very large arrays. We could not get Kryo to work despite
using the different configuration properties. Switching to java serde was
what worked.
Regards
Sab
On Tue, Nov 10, 2015 at 11:43 AM, Hitoshi Ozawa
Hi all,
I have a parquet file, which I am loading in a shell. When I launch the shell
with -driver-java-options ="-Dspark.serializer=...kryo", makes a couple fields
look like:
03-?? ??-?? ??-???
when calling > data.first
I will confirm briefly, but I am utterly su
Hi,
my team is setting up a machine-learning framework based on Spark's mlib,
that currently uses
logistic regression. I enabled Kryo serialization and enforced class
registration, so I know
that all the serialized classes are registered. However, the running times
when Kryo
serialization
Hi Nick/Igor,
Any solution for this ?
Even I am having the same issue and copying jar to each executor is not
feasible if we use lot of jars.
Thanks,
Vipul
ough Yarn.
> Serialization is set to use Kryo.
>
> I have a large object which I send to the executors as a Broadcast. The
> object seems to serialize just fine. When it attempts to deserialize,
> though, Kryo throws a ClassNotFoundException... for a class that I include
> in the
with same class
>
> On 8 September 2015 at 06:38, Nicholas R. Peterson <nrpeter...@gmail.com>
> wrote:
>
>> I'm trying to run a Spark 1.4.1 job on my CDH5.4 cluster, through Yarn.
>> Serialization is set to use Kryo.
>>
>> I have a large object which I
classloader kryo is using?
On Tue, Sep 8, 2015 at 6:34 AM Igor Berman <igor.ber...@gmail.com> wrote:
> java.lang.ClassNotFoundException: com.i2028.Document.Document
>
> 1. so have you checked that jar that you create(fat jar) contains this class?
>
> 2. might be there is some stale c
, Sep 8, 2015 at 1:14 AM Igor Berman <igor.ber...@gmail.com>
>>> wrote:
>>>
>>>> as a starting point, attach your stacktrace...
>>>> ps: look for duplicates in your classpath, maybe you include another
>>>> jar with same class
>
;>>>
>>>> This seems like a likely culprit. What could cause this, and how can I
>>>> fix it?
>>>>
>>>> Best,
>>>> Nick
>>>>
>>>> On Tue, Sep 8, 2015 at 1:14 AM Igor Berman <igor.ber...@gmail.com&
1 - 100 of 204 matches
Mail list logo