akomisarek commented on issue #17634: URL: https://github.com/apache/iceberg/issues/17634#issuecomment-5292426517
I analyzed the source code, and I have a follow-up question: why not trigger a commit when the Coordinator wakes up for the first time? I can imagine in our case is a tuning problem, but we hit a loop that on it's own the connector cannot recover. For me the following events are possible and lead to a problem: 1. Coordinator announces commit. 2. As the consumers were paused for some time, there were possibly hundreds of thousands of messages that were written in a short period of time. So we end up with many of those messages on the control topic. https://github.com/apache/iceberg/blob/35889387c8c6ff0a3f57d17a304709dc1b7d9340/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Worker.java#L70-L72 3. Coordinator crashes mid-collecting stats. 4. New coordinator starts. 5. It hasn't reached the commit threshold, so it immediately begins to poll. - https://github.com/apache/iceberg/blob/35889387c8c6ff0a3f57d17a304709dc1b7d9340/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java#L122-L129 6. Polling for thousands of messages prepared in step 2. Never escaping the loop as the loop polls for new messages. - https://github.com/apache/iceberg/blob/35889387c8c6ff0a3f57d17a304709dc1b7d9340/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Channel.java#L121-L139 7. Now, simultaneously, a lot of writers constantly write new data. 8. After some time the committer escapes the loop to announce commit as maybe the threshold passed. (10 more minutes of new data at least being written). 9. The workers begin to dump new daa written in previous window. 10. Coordinator adds the new events as ready and hopefully performs commit. 11. If Coordinator crashes before finishing the whole thing restarts with even more data to process. This seems to be an ever-growing problematic cycle. Why not allow the coordinator to recover silently without announcing a new commit? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
