jigar-bhati commented on PR #267: URL: https://github.com/apache/flink-connector-kafka/pull/267#issuecomment-5038788681
Thanks, Efrat. One clarification on what you mean by recovery: are you thinking specifically about Flink checkpoint restoration, or about any previously established concrete Kafka position becoming unavailable, including for an already-running reader? The motivating case is not limited to checkpoint restoration. A reader can start successfully and consume normally, then a Kafka leader change/log truncation can make its current position out of range (there is a similar existing report in [FLINK-35630](https://issues.apache.org/jira/browse/FLINK-35630)). In that case, retaining the startup initializer's `earliest`/`latest` reset strategy would still silently reset the running reader. The current PR intentionally preserves an explicitly configured `auto.offset.reset` for the lifetime of the consumer, while keeping the existing initializer-derived behavior when the property is absent. This allows, for example: ```text earliest startup + auto.offset.reset=none -> fresh source starts from earliest normally -> a restored or running concrete position later becomes out of range -> fail instead of silently resetting to earliest ``` Fresh earliest startup uses the earliest marker and seeks to the beginning; latest startup resolves and seeks to the current end. So `none` does not prevent those normal startups. It intentionally affects cases where Kafka cannot use the selected position (including a missing committed position or an invalid specific position). If your suggestion is to use the initializer strategy only for the initial position and the explicit strategy for any later out-of-range position, I think that is the behavior we want conceptually. Since an out-of-range can also occur during normal execution, though, limiting the change to `restoreEnumerator(...)` would not cover the leader-truncation case. Does that match the scenario you had in mind? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
