JingsongLi opened a new pull request, #1191: URL: https://github.com/apache/paimon-rust/pull/1191
### Purpose Let an external scheduler execute the shard work of existing vector, batch-vector, full-text and hybrid search plans without reimplementing query semantics outside Rust Core. PyPaimon's Ray search currently selects Python readers directly, even when Native execution is enabled. ### Brief change log - Add opaque `SearchTask` / `SearchTaskResult` transport and a `SearchTaskExecutor` Core trait. Add `read_with_executor` to search readers and `execute_with_executor` to the hybrid builder; existing local methods retain their behavior. - Keep captured schema/snapshot, scalar and deletion-vector filtering, global ANN candidate selection, refinement and hybrid rank fusion in Core. Preserve complete data splits and one raw BM25 corpus. Limit scheduled task construction and raw result merging to windows of 16 tasks. - Share repeated immutable transport inputs and restore shared bitmap ownership on workers. Validate task/result identities, captured table/branch and worker authorization. - Add thin Python callbacks plus one worker task entry point. Yield blocking scheduler callbacks off the async runtime so hybrid routes can dispatch concurrently, preserve Python exception identity and close result iterators after decode failures. ### Tests - `cargo test -p paimon --features fulltext --lib de_vector_read:: -- --nocapture`: 56 passed, including shared-bitmap transport and bounded raw/index task windows. - `cargo test -p paimon --features fulltext --lib scheduled_ -- --nocapture`: 15 passed, covering snapshot/filter/deletion-vector parity, global refinement, BM25 corpus semantics, hybrid fusion and task identity/authorization. - Full-text reader/builder tests: 32 passed; hybrid builder tests: 21 passed. The later full-text window regression also passed in the scheduled group. - `cargo +1.98.0 test -p pypaimon_rust --lib search_task::tests -- --nocapture`: 3 passed for concurrent callbacks, cancellation exception identity and generator cleanup. - `cargo +1.98.0 clippy -p pypaimon_rust -- -D warnings`, `cargo +1.98.0 fmt --all --check` and Core checks with default and no default features passed. No Native Python extension was installed or rebuilt into the Python test environment. ### API and Format Adds optional search scheduling APIs. The opaque versioned task bytes are execution transport, with no table storage format change. Scheduled execution covers data-evolution searches already supported by PyPaimon's Ray integration; it does not add distributed primary-key search. ### Documentation API rustdocs and binding stubs are included. The paired PyPaimon PR documents the shared scheduling contract and Ray usage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
