JingsongLi opened a new pull request, #10499: URL: https://github.com/apache/paimon/pull/10499
## Purpose Use Rust Native for PyPaimon's existing `TableUpdate.new_shard_updator()` operation. The public Python shard API stays the same; Rust owns the captured scan, balanced file groups, positional row-ID mapping, conversion rules and file staging. Depends on https://github.com/apache/paimon-rust/pull/1099. Keep this PR draft until that API is merged. Native CI continues building `apache/paimon-rust@main`; no workflow fork pin is added. ## Changes - Route supported batch and stream shard updates through the core updater before constructing the Python implementation. - Keep the Python wrapper limited to Arrow batch/schema and CommitMessage transport; nested output aliases retain their existing Python route. - Validate shard bounds before creating either implementation. Once a Native updater exists, its errors propagate without retrying through Python. - Add REST integration coverage for captured snapshots, balanced whole-file shards, overlapping column files, partitioned updates, deletion vectors, batch-boundary changes, column subsets/full updates, ARRAY/MAP/ROW values, invalid input, incomplete reads and prepared-file survival. - Add a required Native `shard` update counter alongside the existing update categories so the CI job proves this core path actually runs. ## Java semantics Core reuses the Data Evolution partial writer and row-ID file index. Column metadata, original physical row-ID ranges and snapshot conflict checks follow Java's `BaseAppendFileStoreWrite` and `FileStoreCommitImpl` rules. Deleted physical rows do not consume positional output values. File messages are preserved after close; their publication may have succeeded even when a commit response raises. ## Validation - Rebuilt Rust Native extension from the companion branch. - 527 tests pass with `PYPAIMON_TEST_NATIVE_PLAN=1`, `PYPAIMON_TEST_NATIVE_READ=1`, `PYPAIMON_TEST_NATIVE_WRITE=1`, `PYPAIMON_TEST_NATIVE_UPDATE=1`, and `PYPAIMON_TEST_NATIVE_COMMIT=1`. - Counters: 578 plans, 659 reads, 2323 writes, 74 commits; updates row_id=68, grouped=50, predicate=47, upsert=11, incremental=7, merge=24, shard=56. - All 8 existing shard tests also pass in the ordinary Python lane. - Flake8, Python 3.6 syntax parsing and `git diff --check` pass. - Independent core and Python contract review findings were fixed; core's 10 regression tests include actual I/O cancellation and failure followed by EOF. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
