twthorn opened a new pull request, #17552: URL: https://github.com/apache/iceberg/pull/17552
Fixes https://github.com/apache/iceberg/issues/17551 More context in that ticket on the exact sequence of events. Overall, it's possible multiple coordinators exist, and one may commit old/stale offsets still in memory, which may be out of retention, and this sequence will cause data loss. We do a check before writing the offset. Note: it is possible that a race condition exists (eg coordinator A reads offset n, coordinator B reads offset n & commits offset n+2, and then coordinator A commits offset n+1). However, the committed offset still never drops below n, so the worst case is reprocessing a few records (ie duplicates), not data loss. Also add some logging that makes these scenarios much more clear (eg when a stale coordinator may exist, what the coordinators offsets are that they are committing). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
