DStream must be Serializable, it's metadata checkpointing. But you can use
KryoSerializer for data checkpointing. The data checkpointing uses
RDD.checkpoint which can be set by spark.serializer.
Best Regards,
Shixiong Zhu
2015-07-08 3:43 GMT+08:00 Chen Song chen.song...@gmail.com:
In Spark Streaming, when using updateStateByKey, it requires the generated
DStream to be checkpointed.
It seems that it always use JavaSerializer, no matter what I set for
spark.serializer. Can I use KryoSerializer for checkpointing? If not, I
assume the key and value types have to be Serializable?
Chen
