Re: Is incremental checkpoints needed?

Hi Arvid, Thank you for the clarification! Best, Eleanore On Tue, Mar 10, 2020 at 12:32 PM Arvid Heise wrote: > Hi Eleanore, > > incremental checkpointing would be needed if you have a large state > (GB-TB), but between two checkpoints only little changes happen (KB-MB). > > There are two

Re: Is incremental checkpoints needed?

Hi Arvid, Thank you for the clarification! Best, Eleanore On Tue, Mar 10, 2020 at 12:32 PM Arvid Heise wrote: > Hi Eleanore, > > incremental checkpointing would be needed if you have a large state > (GB-TB), but between two checkpoints only little changes happen (KB-MB). > > There are two

Re: Is incremental checkpoints needed?

Hi Eleanore, incremental checkpointing would be needed if you have a large state (GB-TB), but between two checkpoints only little changes happen (KB-MB). There are two reasons for large state: large user state or large operator state coming from joins, windows, or grouping. In the end, you will

Is incremental checkpoints needed?

Hi All, I am using Apache Beam to construct the pipeline, and this pipeline is running with Flink Runner. Both Source and Sink are Kafka topics, I have enabled Beam Exactly once semantics. I believe how it works in beam is: the messages will be cached and not processed by the

Is incremental checkpoints needed?

Hi All, I am using Apache Beam to construct the pipeline, and this pipeline is running with Flink Runner. Both Source and Sink are Kafka topics, I have enabled Beam Exactly once semantics. I believe how it works in beam is: the messages will be cached and not processed by the