Information request: Reactive mode and Rescaling

2021-12-15 Thread Geldenhuys, Morgan Karl
Greetings, I would like to find out more about Flink's new reactive mode as well as the rescaling feature regarding fault tolerance. For the following question lets assume checkpointing is enabled using HDFS. So first question, if I have a job where the source(s) and sink(s) are configured t

Re: Information request: Reactive mode and Rescaling

2021-12-16 Thread Alexander Preuß
Hi Morgan, Regarding your first question, if the Kafka connector is configured to use exactly-once semantics it will check the offsets of partitions when recovering from the checkpoint, so there will be no data loss or duplication. I'm not quite sure I understood the second part of the first quest