That fixed it!. I still had the serializer registered as a workaround
for SPARK-12591.
Thanks so much for your help Ryan!
-Joey
On Wed, Oct 12, 2016 at 2:16 PM, Shixiong(Ryan) Zhu
wrote:
> Oh, OpenHashMapBasedStateMap is serialized using Kryo's
> "com.esotericsoftware.kryo.serializers.JavaSeria
Oh, OpenHashMapBasedStateMap is serialized using Kryo's
"com.esotericsoftware.kryo.serializers.JavaSerializer". Did you set it for
OpenHashMapBasedStateMap? You don't need to set anything for Spark's
classes in 1.6.2.
On Wed, Oct 12, 2016 at 7:11 AM, Joey Echeverria wrote:
> I tried with 1.6.2
I tried with 1.6.2 and saw the same behavior.
-Joey
On Tue, Oct 11, 2016 at 5:18 PM, Shixiong(Ryan) Zhu
wrote:
> There are some known issues in 1.6.0, e.g.,
> https://issues.apache.org/jira/browse/SPARK-12591
>
> Could you try 1.6.1?
>
> On Tue, Oct 11, 2016 at 9:55 AM, Joey Echeverria wrote:
>
There are some known issues in 1.6.0, e.g.,
https://issues.apache.org/jira/browse/SPARK-12591
Could you try 1.6.1?
On Tue, Oct 11, 2016 at 9:55 AM, Joey Echeverria wrote:
> I tried wrapping my Tuple class (which is generated by Avro) in a
> class that implements Serializable, but now I'm gettin
I tried wrapping my Tuple class (which is generated by Avro) in a
class that implements Serializable, but now I'm getting a
ClassNotFoundException in my Spark application. The exception is
thrown while trying to deserialize checkpoint state:
https://gist.github.com/joey/7b374a2d483e25f15e20c0c4cb8
I do, I get the stack trace in this gist:
https://gist.github.com/joey/d3bf040af31e854b3be374e2c016d7e1
The class it references, com.rocana.data.Tuple, is registered with
Kryo. Also, this is with 1.6.0 so if this behavior changed/got fixed
in a later release let me know.
-Joey
On Mon, Oct 10, 2
That's enough. Did you see any error?
On Mon, Oct 10, 2016 at 5:08 AM, Joey Echeverria wrote:
> Hi Ryan!
>
> Do you know where I need to configure Kryo for this? I already have
> spark.serializer=org.apache.spark.serializer.KryoSerializer in my
> SparkConf and I registered the class. Is there a
Hi Ryan!
Do you know where I need to configure Kryo for this? I already have
spark.serializer=org.apache.spark.serializer.KryoSerializer in my
SparkConf and I registered the class. Is there a different
configuration setting for the state map keys?
Thanks!
-Joey
On Sun, Oct 9, 2016 at 10:58 PM,
You can use Kryo. It also implements KryoSerializable which is supported by
Kryo.
On Fri, Oct 7, 2016 at 11:39 AM, Joey Echeverria wrote:
> Looking at the source code for StateMap[1], which is used by
> JavaPairDStream#mapWithState(), it looks like state keys are
> serialized using an ObjectOutp
Looking at the source code for StateMap[1], which is used by
JavaPairDStream#mapWithState(), it looks like state keys are
serialized using an ObjectOutputStream. I couldn't find a reference to
this restriction in the documentation. Did I miss that?
Unless I'm mistaken, I'm guessing there isn't a w
10 matches
Mail list logo
