Re: Introduce Barriers in stream source

2018-08-13 Thread Fabian Hueske
Hi, It is sufficient to implement the CheckpointedFunction interface. Since SourceFunctions emit records in a separate thread, you need to ensure that not record is emitted while the shapshotState method is called. Flink provides a lock to synchronize data emission and state snapshotting. See the

Introduce Barriers in stream source

2018-08-13 Thread Darshan Singh
Hi, I am implementing a source and I want to use checkpointing and would like to restore the job from these external checkpoints. I used Kafka for my tests and it worked fine. However, I would like to know if I have my own source what do I need to do. I am sure that I will need to implement