Hello Spark folks and especially TD, I am using the Spark Streaming 1.6 mapWithState API, and I am trying to enforce Kryo Serialization with
SparkConf.set("spark.kryo.registrationRequired", "true") However, this appears to be impossible! I registered all the classes that are my own, but I problem with a class org.apache.spark.streaming.rdd.MapWithStateRDDRecord, which is set as private[streaming] . The error: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.streaming.rdd.MapWithStateRDDRecord Note: To register this class use: kryo.register(org.apache.spark.streaming.rdd.MapWithStateRDDRecord.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:442) at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79) at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:472) Since this class is private with spark streaming itself, I cannot actually register it with Kryo, and I cannot do registrationRequired in order to make sure *everything* has been serialized with Kryo. Is this a bug? Can I somehow solve this? Aris