JingsongLi opened a new pull request, #10770: URL: https://github.com/apache/paimon/pull/10770
### Purpose Use Native execution for Ray fixed/POSTPONE input bucket routing and POSTPONE input statistics/inference, which previously still ran Python row hashing and sizing before the Native writers received input. ### Changes - Select a Native input router through dispatch before distributed tasks consume rows. Ray scheduling/grouping and public APIs remain unchanged. - Keep Native adapters limited to Arrow/partition-plan transport and serializable configuration. The Native planner no longer inherits Python input statistics or inference; Rust Core owns these operations. - Drop SDK objects during pickle and reconstruct from the shared schema/plan in workers. Runtime failures after backend selection propagate without replaying input through Python. - Preserve positional statistics columns when partition keys are named `row_count`, `data_size` or `total_buckets`. - Match current Java's NULL noncompact DECIMAL/TIMESTAMP/LTZ BinaryRow bytes, including decimal-overflow-to-NULL serialization, so Python hashing agrees with Native writers. - Add backend-isolation, worker pickle, reordered/subset input, chunked/sliced Arrow, nullable FixedSizeBinary partition and exact Java hash regressions; update Ray docs. ### Validation - Serializer/routing/dispatch focused suites: **145 passed**. - Ray/postpone/table-write selection: **96 passed, 9 skipped**. - Joint current-SDK run: **116 passed**, including the 8 new actual Native routing/planning cases, existing write/authorization tests and NULL-layout regressions; **36 Native plans exercised**. - Scoped flake8 and diff checks passed. No native extension was installed or replaced. ### Dependency Requires https://github.com/apache/paimon-rust/pull/1207 and its `BucketRouter` and `PostponeBucketPlanner` SDK APIs. This PR is draft until that Rust change is merged. Native CI continues building Rust main; no SDK-version compatibility probes or fallback are introduced. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
