JingsongLi commented on PR #10431: URL: https://github.com/apache/paimon/pull/10431#issuecomment-6080839855
[P2] Reject snapshot changes without rolling back concurrent compaction The new strict snapshot check in `ConflictDetection.check_conflicts` returns a normal commit conflict. Maintenance sets `check_from_snapshot` on every message, which also sets `allow_rollback=True` in `FileStoreCommit.commit`. When a rollback-capable catalog has published a concurrent `COMPACT`, `_try_commit_once` consequently calls `CommitRollback.try_to_rollback`, rolls the table back to the preceding snapshot, and retries the maintenance commit. It can then publish successfully instead of rejecting the stale plan. This contradicts the documented strict guard and lets a scheduled index maintenance pass undo a completed compaction. Please make this snapshot-change conflict non-rollbackable, or disable compaction rollback for maintenance/index-only messages. The existing concurrent-append test uses a filesystem catalog without catalog rollback and does not exercise this branch. Validation: 90 targeted tests and 22 subtests passed locally, plus project Flake8. I also reproduced the rollback using real Parquet data, a real BTree rebuild and the production committer; only the catalog rollback endpoint was injected to delegate to local rollback. Planned snapshot 2 was followed by a physical-file-replacing COMPACT 3. Current code called rollback 3→2 and published maintenance as APPEND 3, restoring the original data file. Using the pre-PR conflict-check implementation instead published APPEND 4 with zero rollback calls and preserved the compacted file. Both variants retained the two logical rows; this finding does not claim data loss. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
