junmuz opened a new issue, #6183: URL: https://github.com/apache/paimon/issues/6183
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 77f130bde03381e3b26b86e4f0e95cff5c1dc622 ### Compute Engine Flink ### Minimal reproduce step I created a pkey Paimon table with the following table properties with Iceberg compatibility enabled. ``` changelog-producer: lookup full-compaction.delta-commits: 1 metadata.iceberg.storage-location: table-location metadata.iceberg.manifest-legacy-version: true metadata.iceberg.manifest-compression: snappy metadata.iceberg.previous-versions-max: 24 metadata.iceberg.storage: hive-catalog metadata.iceberg.hive-client-class: com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient metadata.iceberg.glue.skip-archive: true metadata.iceberg.uri: "" metadata.iceberg.hive-conf-dir: "" metadata.iceberg.hadoop-conf-dir: "" ``` I performed the inserts and update operations on the above created table. ### What doesn't meet your expectations? After performing the updates, I don't see the updates getting reflected from Iceberg compatibility layer. I was following the source code, and it appears that the delta-commits trigger only happens when the changelog-producer is set to full-compaction. The delta-commits is handled only inside the [GlobalFullCompactionSinkWrite](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/GlobalFullCompactionSinkWrite.java#L172-L176), which only gets invoked when the [changelog-producer is full-compaction](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StoreSinkWrite.java#L146-L146). Inside the docs as well, the full-compaction.delta-commits is only mentioned for [Full Compaction changelog-producer](https://paimon.apache.org/docs/1.2/primary-key-table/changelog-producer/#full-compact ion). When we have [changelog-producer as lookup](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/StoreSinkWrite.java#L160-L175), it relies on the [LookupSinkWrite](https://github.com/apache/paimon/blob/77f130bde03381e3b26b86e4f0e95cff5c1dc622/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/LookupSinkWrite.java#L36-L36) that doesn't take care of the delta-commit property. ### Anything else? Based on the above behavior, I think of the following 2 solutions. 1. Either allow the `full-compaction.delta-commits` for other changelog producers as well. This will have a drawback increasing the cost of commit operations as full-compaction will be triggered everytime. 2. Improve the documentation, and not allow setting `full-compaction.delta-commits` property when changelog producer is different from `full-compaction`. Let me know about your thoughts. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
