JingsongLi opened a new pull request, #10712: URL: https://github.com/apache/paimon/pull/10712
### Purpose Continue separating public PyPaimon configuration from execution after #10707. Public classes retain their existing paths, bases, signatures, configuration, validation, serialization and override hooks; explicit delegates execute against the original instance. * Move Ray read tasks, split balancing, output metadata and cached provider planning into shared `execution/datasource` helpers. Keep worker closures independent of datasource/provider objects and preserve independently selected plan/read backends. * Move Torch indexing, row/batch iteration, worker scheduling, prefetch, shuffle and conversion into shared datasource execution. Keep public dataset classes and their DataLoader/pickle behavior. * Move Ray worker writes, coordinator commits, bucket planning and distributed scheduling into shared `execution/datasink` helpers. Keep `PaimonDatasink` and `PaimonWriteResult` at their public paths. Shared Pandas transport and distributed writer inputs now live in `execution/write_input.py`. * Move Vindex selection and resource ownership into shared index execution. Public readers delegate without importing either backend during construction. Preserve Native-first selection and never replay Native opening/query failures through Python. Serialize initialization/close safely, including callbacks that reenter `close()`. * Move search reader/scan construction into Python execution factories while retaining existing public override hooks. * Move MERGE condition parsing and Arrow assignment/clause transforms out of the public Ray namespace. Local Python MERGE and Native MERGE no longer import Ray orchestration. Deduplicate callable input normalization and delegate public MERGE backend selection to shared update execution. Existing public Ray helper signatures and shared clause/reference types remain available. * Close retained Ray/Torch read sources on completion, failure and cancellation, preserving the original read/conversion exception. Prefetch producers check cancellation while waiting on a full queue, including with the default timeout settings. No new runtime dependencies, Rust features, Native version probes or extension rebuild/install steps. ### Verification * Compared 197 declared public contracts against merged master (`062b22e0ad`): signatures, annotations, decorators and class bases unchanged. Existing public classes keep their module/pickle identities and subclass/instance hooks. * New/related combined boundary regressions: **57 passed**, including fresh processes forbidding the other backend, public configuration without execution imports, serialization, hooks and cleanup. * Torch: **71 passed**, including two-rank/two-worker distributed execution. Follow-up cancellation regressions use default queue/put/join settings and verify producer termination and single source close. * Ray: **8 real write** and **9 real read** cases passed; write unit/boundary group **88 passed, 9 skipped**, provider/read boundary group **21 passed**, and Pandas write cases **2 passed**. * MERGE: initial broad Python/Ray run had **183 passed, 16 skipped** and two outdated mock/logger-path failures. Both were corrected to the execution module and passed in the **25-case** condition/boundary rerun. Final Python MERGE/source-schema group: **50 passed, 16 skipped**. The new process-isolation test fails on merged master for both backends and passes on this branch. * Vindex: **61 passed** for preparation, fallback, exceptions, concurrent initialization and reentrant close. Search/filter/hybrid/PK group: **107 passed**. Using the existing installed Native extension, **26 real Native Vindex cases passed**; it was neither rebuilt nor installed. * Wheel/sdist: **9 passed**, explicitly checking all new execution modules. Whole-package Flake8, all **1,181** license headers, changed-file Python 3.6 grammar (**45 files**) and diff checks passed. * Independent reviews covered public contracts, backend boundaries, Ray serialization, MERGE semantic equivalence, Torch cancellation and Vindex ownership. Review findings were fixed and rechecked. Focused groups overlap; the full Python/Native suites were not repeated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
