JingsongLi opened a new pull request, #1208: URL: https://github.com/apache/paimon-rust/pull/1208
### Purpose PyPaimon's existing Daft distributed writes still hash dynamic keys, restore HASH indexes, and allocate buckets in Python. Provide these execution stages in Core so the Python Native backend can use Arrow throughout while Daft retains its grouping and scheduling. ### Brief change log - Add a schema-owned `DynamicBucketRouter` returning typed assigner IDs and Java BinaryRow key hashes in input order. Fixed and dynamic routing share required-column projection and write normalization. - Add `WriteBuilder::new_bucket_assigner` and `HashBucketAssigner::assign`. The factory captures one write restore snapshot, reuses `with_restore_snapshot` and overwrite settings, and preserves the REST write identity without requiring SELECT authorization. - Reuse the writer's allocation algorithm, including historical foreign-owned mappings, owned bucket holes, target rows and bucket limits. Read HASH indexes in bounded 8 MiB ranges and retain only requested mappings; this assignment stage creates no writers, HASH maintainers or files. - Validate the complete input's owner, lengths and nulls before restoring or changing state. Precomputed hashes only require partition columns, including zero-column unpartitioned batches. - Expose thin Python bindings, typed stubs and documentation. Final file writers continue to own complete HASH publication after regrouping by partition/bucket. ### Tests - Core unit suite: 4645 passed, 7 ignored, including 16 new behavioral tests. Current Java fixtures cover composite keys, NULL/noncompact DECIMAL/TIMESTAMP/LTZ, Unicode and signed hash boundaries. - Python SDK routing tests: 2 passed, exercising Arrow slicing, explicit hashes, invalid inputs, normal writes and restore/overwrite baselines. - Joint PyPaimon integration with the current SDK registered in a Cargo test executable: 119 passed, including real REST Daft initial/append/overwrite writes with Python routing/assignment constructors disabled and SELECT calls forbidden. No extension was installed or rebuilt for those Python tests. - Rust 1.98.0 workspace Clippy, all targets, fulltext/vortex: passed with `-D warnings`. - Rust 1.94.0 workspace/all-targets/all-features check: passed. ### API and Format Additive Core and Python SDK APIs. No storage-format, dependency or public PyPaimon API changes. No new compaction, rescaling or metadata functionality. ### Documentation Updated the Python SDK distributed input execution examples. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
