twthorn commented on PR #17552: URL: https://github.com/apache/iceberg/pull/17552#issuecomment-5590108752
@danielcweeks These are different bugs dealing with different offset stores. This PR addresses the issue of kafka consumer offsets being rewound (by stale coordinators). If rewound out of retention, the consumer falls back to its auto offset reset strategy, causing potential data loss. The other PR addresses the offsets stored in the Iceberg snapshot summary (`kafka.connect.offsets...`). On restart, this value is [used](https://github.com/apache/iceberg/blob/main/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java#L274-L278) to skip records to avoid duplicates when committing to the iceberg table. When this value is rewound it leads to duplicate data. So, separate bugs, both fixes needed. One to prevent data loss. The other to prevent duplicates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
