JingsongLi opened a new pull request, #10699: URL: https://github.com/apache/paimon/pull/10699
### Purpose Separate Python Execution and Native Execution while keeping the public PyPaimon API and independent Native plan/read/write/commit options. * Move Python scanners, file readers, merge execution, writers, committers, and updates into `pypaimon.execution.python`. * Move Rust runtime/table bridges and Native scan/read/search/write/commit/update adapters into `pypaimon.execution.native`. Existing `paimon_ftindex` SDK execution stays with the Python implementation. * Keep public builders and table operations in their existing packages. Shared predicates, schemas, plans, splits, commit messages, MERGE clauses, and input normalization remain outside both implementations. * Capture Native scan input in a shared `ScanRequest`, so Native planning does not access Python scanner internals. Share the existing chunk-shuffle and metadata-LIMIT rules. * Centralize write-side selection in `execution.dispatch`, preserving preparation-only fallbacks, fork-safety errors, callbacks, cancellation, and the prohibition on replaying a mutation through another implementation. MERGE input preparation errors still propagate. * Preserve public method signatures and existing serializable class imports. Update internal imports and test patch targets to the implementation that owns each operation, and document the dependency boundaries. No Rust changes, new Native features, or runtime dependencies are required. ### Tests * Broad Python regression: 9,036 passed and 641 skipped before the failure limit. The 19 stale-mock/test-isolation failures were repaired and rerun successfully; the one remaining failure also reproduces on the base commit (see below). * Rerun affected files: streaming scan 15 passed; stream builder 13 passed; field aggregators/write nullability 183 passed; execution-boundary checks 24 passed, 78 skipped. Remaining write/acceptance tail: 417 passed, 33 subtests passed. * Ray pipeline lifecycle and ingestion checks: 5 passed, including the corrected assertion that preserves a pre-existing cluster. * Native read/plan mocks: 152 passed, 78 skipped, 44 subtests passed. * Fork safety and execution dispatch regressions: 72 passed. * Scan request, Native plan, metadata-LIMIT and chunk-shuffle regressions: 93 passed, 44 subtests passed. * Wheel/sdist packaging tests, including the new execution packages: 9 passed. * Python 3.6's curated subset and Torch read coverage: 150 passed; the remaining distributed-launch test passed separately with `PET_LOCAL_ADDR=127.0.0.1` and `GLOO_SOCKET_IFNAME=lo0` for this host's rendezvous setup. * Python 3.6 grammar validation, whole-package Flake8, license headers, and `git diff --check` passed. * Real Native integration is left to CI against Rust main. The local extension was neither rebuilt nor installed; local Native adapter tests used mocks. The local Python suite has one existing failure in `RayReadByRowIdTest.test_empty_result_schema_matches_nonempty_read`: `_ROW_ID` is nullable in nonempty batches and non-null in the typed empty result. The same single test fails against the unchanged base commit `eeb765d405` with Python 3.13.5, Ray 2.54.0, and PyArrow 19.0.1. This refactor preserves that behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
