nickdelnano opened a new issue, #6107: URL: https://github.com/apache/paimon/issues/6107
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Code path: https://github.com/apache/paimon/blob/ff02f6bf3ceccf8dcac38bc58cf6db390509bd46/paimon-core/src/main/java/org/apache/paimon/iceberg/IcebergCommitCallback.java#L305C49-L305C59 createMetadataWithoutBase is called under certain conditions like - enabling Iceberg compatibility for the first time - committing Iceberg metadata after a previous commit failed in the Iceberg layer (for many possible reasons) In some cases the Paimon table will have many snapshots already (e.g. snapshots 1 to 1000) and after calling createMetadataWithoutBase, only the latest Paimon commit will be synced to Iceberg. By syncing the whole Paimon history to Iceberg, the Iceberg compatibility feature becomes more reliable and suitable for production use cases that require Iceberg time travel. My use case is this: - Sync MySQL tables to Paimon using Flink CDC - Tag daily Paimon snapshots [automatically](https://paimon.apache.org/docs/master/maintenance/manage-tags/) - Iceberg readers read daily snapshots Upon any failure in the Iceberg committer, the metadata needs to be recreated. This pipeline breaks and cannot be recovered. Consider making this feature opt-in with configuration as it may be a costly operation to sync many Paimon snapshots to Iceberg and therefore reach the Flink checkpoint timeout ### Solution _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@paimon.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
