The StreamingContext can be recreated from a checkpoint file, indeed. check out the following Spark Streaming source files for details: StreamingContext, Checkpoint, DStream, DStreamCheckpoint, and DStreamGraph.
On Wed, Jul 9, 2014 at 6:11 PM, Yan Fang <yanfang...@gmail.com> wrote: > Hi guys, > > I am a little confusing by the checkpointing in Spark Streaming. It > checkpoints the intermediate data for the stateful operations for sure. > Does it also checkpoint the information of StreamingContext? Because it > seems we can recreate the SC from the checkpoint in a driver node failure > scenario. When I looked at the checkpoint directory, did not find much > clue. Any help? Thank you very much. > > Best, > > Fang, Yan > yanfang...@gmail.com > +1 (206) 849-4108 >
