kbendick commented on issue #4971: URL: https://github.com/apache/iceberg/issues/4971#issuecomment-1154176676
As an aside, may I ask what you have you max concurrent checkpoints set to with Flink? I would advise you keep it at 1. Additionally, I would advise that you configure (via Flink configuration) a minimum time between checkpoints. Defining a minimum time between checkpoints (default is zero) ensures that if a checkpoint takes place, if it takes long enough to complete that another checkpoint would have already started, that checkpoint is skipped to allow the engine time to process more data. I know that an empty commit could still occur in some scenarios, for example a Kafka topic might not receive data at all for 10 minutes. But if you’re not using those two Flink configurations, I highly suggest you do as they will reduce the chances of this happening and also generally make your jobs more stable. 🙂 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
