ZeleniJure opened a new issue, #15844: URL: https://github.com/apache/iceberg/issues/15844
### Query engine - AWS MSK kafka connect v 3.7.x and iceberg sink 1.10.1 - using Glue catalog, writing to S3 ### Question I have experimented with different settings of `iceberg.control.commit.interval-ms` and `iceberg.control.commit.timeout-ms`. Interval is usually set to 5 minutes, and timeout from 2 minutes up to 8. I even tried increasing `iceberg.kafka.session.timeout.ms` and `iceberg.kafka.heartbeat.interval.ms`. <img width="1589" height="193" alt="Image" src="https://github.com/user-attachments/assets/08c1d287-6295-4fed-9e59-f7415aa3fc42"; /> In all cases I see the control topic growing over time with this kind of a pattern: <img width="1589" height="193" alt="Image" src="https://github.com/user-attachments/assets/060ebb84-77d7-4793-8274-359515e7043f"; /> I have observed these in logs: - Received commit response when no commit in progress, this can happen during recovery. - Commit timeout reached. I do not see any other errors, data is flowing as expected, topics do get drained. Except the control one, which will blow up one day since it is constantly increasing. I have noticed that if I destroy the connector and re-create it after a short while, the control topic "will start growing from 0 again". Or maybe I was just lucky! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
