TheR1sing3un opened a new pull request, #10584: URL: https://github.com/apache/paimon/pull/10584
Ray full-text search currently constructs live-row and scalar-filter bitmaps on the driver before dispatching shard searches. Candidate-only predicates can also read table data there, so these reads and bitmap allocations remain a driver bottleneck. Move deletion-vector reads, scalar-index evaluation and candidate refinement into the existing full-text workers. Workers use the snapshot pinned during planning and restrict live-row reads to their shard ranges. Reuse the vector path's shared scalar-filter ObjectRef dependency for full-text shards with identical scalar-index inputs, avoiding repeated evaluation without materializing the mask on the driver. Raw fallback still searches the complete original corpus so BM25 statistics remain unchanged. No new public API or benchmark code. This is independent of the candidate reduction in #10583. Validation: - 54 full-text scalar-filter and Ray full-text/hybrid tests passed, including worker-only bitmap/data reads and a single shared scalar evaluation across three text shards. - 23 vector scalar-filter exactness/sharing and full-text cleanup regressions passed. - Additional regression checks a delete committed after planning: worker filters retain the planned snapshot, while the next query sees the delete. - Project flake8 and git diff --check passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
