The StreamingContext can be recreated from a checkpoint file, indeed.
check out the following Spark Streaming source files for details:
StreamingContext, Checkpoint, DStream, DStreamCheckpoint, and DStreamGraph.
On Wed, Jul 9, 2014 at 6:11 PM, Yan Fang yanfang...@gmail.com wrote:
Hi guys,
Hi guys,
I am a little confusing by the checkpointing in Spark Streaming. It
checkpoints the intermediate data for the stateful operations for sure.
Does it also checkpoint the information of StreamingContext? Because it
seems we can recreate the SC from the checkpoint in a driver node failure