Hi All, I am using Apache Beam to construct the pipeline, and this pipeline is running with Flink Runner.
Both Source and Sink are Kafka topics, I have enabled Beam Exactly once semantics. I believe how it works in beam is: the messages will be cached and not processed by the KafkaExactlyOnceSink, until the checkpoint completes and all the cached messages are checkpointed, then it will start processing those messages. So is there any benefit to enable increment checkpointing when using RocksDB as backend. Because I see the states as consumer offsets, and cached messages in between checkpoints. Delta seems to be the complete new checkpointed states. Thanks a lot! Eleanore
