JingsongLi opened a new pull request, #10501: URL: https://github.com/apache/paimon/pull/10501
## Purpose Use Rust Native for existing PyPaimon writes to deletion-vector primary-key tables with partial-update/aggregation, and for postpone fixed-bucket DV writes. Depends on https://github.com/apache/paimon-rust/pull/1100. Keep this PR draft until the Rust changes are merged. Native CI continues building `apache/paimon-rust@main`. ## Changes - Remove the two Native eligibility gates for DV partial/aggregate writes and fixed-bucket postpone DV writes. - Continue passing Arrow input and commit messages through the existing Native writers. Rust owns routing, buffer folding, sequence restoration, and file preparation. - Add 51 REST integration cases comparing Python/Native writers and readers over fixed/dynamic/postpone buckets, batch/stream operations, and NONE/INPUT changelog. - Verify default DV scans hide level-0 files and merge-on-read exposes their merged state. - Check INPUT changelog preserves every input before folding, including null values; filter merged values while omitting filter columns from output; keep state correct over buffer flushes. - Cover dynamic restore snapshots, partition overwrite, existing postpone bucket layouts when defaults change, new partition bucket counts, and overwrite rescaling. ## Java semantics `MergeTreeWriter.flushWriteBuffer` uses the configured reducer to write level-0 files and emits INPUT changelog before folding. `CoreOptions.batchScanSkipLevel0` controls default DV visibility and the merge-on-read override. `KeyValueFileStore.newPostponeFixedBucketWrite` uses the normal fixed-bucket writer in write-only mode and accepts DV tables. The Native paths reuse those rules in core. ## Validation - Confirmed ordinary Python writes already support the partial-update/aggregation configurations before enabling Native. - Rebuilt the Rust extension from the companion branch. - 337 tests pass with `PYPAIMON_TEST_NATIVE_PLAN=1`, `PYPAIMON_TEST_NATIVE_READ=1`, `PYPAIMON_TEST_NATIVE_WRITE=1`, and `PYPAIMON_TEST_NATIVE_COMMIT=1`, including 51 new REST cases and the existing Native write/merge/changelog/precomputed-bucket/postpone suites. - Counters: plans=488, reads=363, writes=387, commits=160. - Flake8, Python 3.6 syntax parsing, and `git diff --check` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
