JingsongLi opened a new pull request, #1207: URL: https://github.com/apache/paimon-rust/pull/1207
### Purpose Move distributed input bucket computation and POSTPONE input statistics/inference into Rust Core so PyPaimon/Ray can use the same implementation as local writers without creating file writers for preclustering. ### Changes - Add schema-based `BucketRouter::new(&TableSchema)`, `with_bucket_plan` and input-order Arrow `route` APIs for fixed and POSTPONE buckets. Reuse existing bucket functions and write-array normalization; payload columns are not needed for routing. - Expose pure `PostponeBucketPlanner` with `input_statistics`, `plan` and `current_plan`. Worker statistics can be concatenated and duplicate partitions are aggregated before inference; known layouts are preserved and invalid statistics leave the plan unchanged. - Keep buffered automatic-writer ownership separate and reuse the same planner/router. Resolve write metadata through the existing write builder, preserving REST write authorization. - Make `PostponeBucketPlan` Arrow codecs depend on `TableSchema` instead of a live `Table`. Add typed Python binding classes and SDK docs; no new dependencies. - Match current Java's NULL BinaryRow layout for noncompact DECIMAL/TIMESTAMP/LTZ, including reserved variable slots and offsets. Cover exact Java bytes/hashes, nullable-key writer restarts, and exact NULL partition filtering. Existing format partition reconstruction uses the same typed-null helper to preserve its filtering behavior. ### Java semantics Reference fixtures were generated from current Apache Paimon Java `InternalRowSerializer`, `RowHelper` and `BinaryRowWriter` on paimon master `7befc15f18d81bc406679e14b48aa92cfe863335`. Tests include precision boundaries, a following variable-length string, fixed/mod/Hive buckets, nullable/sliced FixedSizeBinary partitions, option precedence, explicit-plan replacement, overflow/cap behavior, and normalized PK event inputs. ### Validation - Core focused suites: POSTPONE, BinaryRow, BucketRouter, partition filtering and partition reconstruction. - `nullable_primary_key_test` integration suite. - Registered Python SDK `write_routing` API test. - Joint PyPaimon execution against the registered current SDK: **116 passed**, including all 8 new real Native routing/planning cases and existing writer/authorization regressions; **36 Native plans exercised**. Used a temporary embedded test executable, without installing or replacing a Python extension. - Rust 1.98.0 Clippy and formatting checks. Paired PyPaimon PR will use these APIs for Ray preclustering and POSTPONE statistics/planning. PyPaimon public APIs remain unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
