JingsongLi opened a new pull request, #10724: URL: https://github.com/apache/paimon/pull/10724
### Purpose Depends on apache/paimon-rust#1191. Draft until the new Rust task APIs reach main. The current Python master also uses the writer API from apache/paimon-rust#1188. Keep Ray as a scheduler while the selected Python or Native backend owns vector, batch-vector, full-text and hybrid search execution. Ray currently subclasses Python search readers and calls their private methods, forcing Python execution when Native is enabled. ### Changes - Add the public `SearchTaskExecutor` contract and optional `executor` on `read_plan` / `read_batch_plan`, plus `HybridSearchBuilder.execute(executor=...)`. Existing local calls and public hooks keep their behavior. - Move Python shard algorithms from Ray into `execution/python/read/search`. Ray uses public scan/read/execute methods and a generic bounded scheduler, without importing either backend. - Add a thin Native adapter for opaque Core-owned tasks and shared inputs. Live Native plans remain Native on workers and retain their captured schema/snapshot; execution failures propagate without replaying through Python. - Share task prerequisites and the query concurrency budget. Isolate cancellation between independent queries, propagate it across callback threads, preserve original failures and cancel pending work on iterator close. - Document the execution boundary and add packaging, backend isolation, snapshot transport, task scheduling and cancellation regressions. Keep global ANN selection before refinement and one raw BM25 corpus. ### Tests - Final public API/backend isolation/scheduling/cancellation group: 70 passed. - Adjacent local search/index/result/cleanup regressions: 237 passed. - Ray vector, batch-vector, full-text, refinement and filtering suites, including actual Ray cancellation and hybrid ranking tests. - Packaging: 9 passed; repository-configured Flake8, Python 3.6 grammar and ASF headers passed. - Whole test collection passed (22,384 tests). Native integration coverage is included for single/batch/full-text task transport with a new commit appended during worker scheduling. It requires the paired Rust PR and was not run locally; the Native extension was not rebuilt or installed. Native CI continues to build Rust main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
