zhuxiangyi commented on PR #10105: URL: https://github.com/apache/paimon/pull/10105#issuecomment-6316853074
Done — this PR now keeps only Spark changes. We narrowed the scope to our actual use case: append-only log tables fed from Kafka, where a replayed micro-batch is duplicated because nothing merges the rows away. Chain tables are not part of it, so all core changes are reverted: the chain-table overwrite cleanup on retry, the `beforeOverwrite` hook and the `filterAndCommit` callback overload are gone, and the PR is now a single commit (28cb9fe) touching only `paimon-spark` and the docs. Without core changes: - the expiration check is done in the sink: it looks the commit user up first, validates that lookup against the marker, and then commits directly; a batch found already committed only goes through `filterAndCommit` without its files, to let the commit callbacks retry (e.g. partition registration), so a lookup that misses it publishes an empty commit rather than the batch again; - complete mode keeps the previous behaviour: every micro-batch, including a replay, is committed as an overwrite of the complete result, which leaves the same rows and still runs the overwrite callbacks. `paimon-spark-ut` passes in full on Spark 3.5 apart from the lumina vector-index suites (native library not available on my machine) and one time-based tag-expiry case that passes on rerun; the sink suites pass on Spark 3.2 / 3.4 / 3.5 / 4.0 / 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
