Re: Flink Checkpoint & Offset Commit

2024-03-07 Thread Yanfei Lei
1][2] may be helpful to implement a custom Flink’s Table API connector. Specifically in terms of “Flink Checkpoint & Offset Commit”, the custom source needs to inherit the `SourceReader` interfaces, and you can override `snapshotState()` and `notifyCheckpointComplete()` into your implementations. [3] i

Re: Flink Checkpoint & Offset Commit

2024-03-07 Thread xia rui
Hi Jacob. Flink uses "notification" to let an operator callback the completion of a checkpoint. After gathering all checkpoint done messages from TMs, JM sends a "notify checkpoint completed" RPC to all TMs. Operators will handle this notification, where checkpoint success callbacks are invoked.

Fwd: Flink Checkpoint & Offset Commit

2024-03-07 Thread Jacob Rollings
Hello, I am implementing proof of concepts based Flink realtime streaming solutions. I came across below lines in out-of-the-box Flink Kafka connector documents. *https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/*