DStream must be Serializable, it's metadata checkpointing. But you can use KryoSerializer for data checkpointing. The data checkpointing uses RDD.checkpoint which can be set by spark.serializer.
Best Regards, Shixiong Zhu 2015-07-08 3:43 GMT+08:00 Chen Song <chen.song...@gmail.com>: > In Spark Streaming, when using updateStateByKey, it requires the generated > DStream to be checkpointed. > > It seems that it always use JavaSerializer, no matter what I set for > spark.serializer. Can I use KryoSerializer for checkpointing? If not, I > assume the key and value types have to be Serializable? > > Chen >
