Re: Spark Streaming 1.6 mapWithState not working well with Kryo Serialization

2016-03-02 Thread Shixiong(Ryan) Zhu
See https://issues.apache.org/jira/browse/SPARK-12591 After applying the patch, it should work. However, if you want to enable "registrationRequired", you still need to register "org.apache.spark.streaming.util.OpenHashMapBasedStateMap", "org.apache.spark.streaming.util.EmptyStateMap" and

Spark Streaming 1.6 mapWithState not working well with Kryo Serialization

2016-03-02 Thread Aris
Hello Spark folks and especially TD, I am using the Spark Streaming 1.6 mapWithState API, and I am trying to enforce Kryo Serialization with SparkConf.set("spark.kryo.registrationRequired", "true") However, this appears to be impossible! I registered all the classes that are my own, but I