TheR1sing3un opened a new pull request, #10583: URL: https://github.com/apache/paimon/pull/10583
Full-text queries currently request every row in each shard before taking top-k. Ray then transports those candidate maps to the driver. With 10,000 matching documents and limit=10, the existing worker function returns 10,000 candidates. This change passes the requested limit into indexed and raw Python search. The native reader initially requests k+1 hits and expands only when the score boundary is tied: paimon-ftindex breaks ties by document ordinal, while Python uses row ID. Overlapping shard ranges retain exhaustive candidates to preserve first-shard precedence for duplicate IDs. Raw fallback keeps the same BM25 corpus. Returned score maps retain only selected candidates. This follows the intent of #9955 for the Python fallback and Ray paths, while preserving Python tie and duplicate semantics. No public API, file format, dependency, or benchmark code changes. Validation: - 126 focused full-text, filtering, cleanup, vector-filter, and hybrid execution tests passed. - 17 Ray full-text/hybrid integration tests passed (plus the 5 new top-k tests in that run). - Real native indexes cover shuffled row IDs, score ties, bitmap filtering, raw row-ID offsets, and bounded retrieval without boundary ties. - Project flake8 and git diff --check passed. A large boundary tie can still require fetching all matching candidates; correctness takes precedence over truncating tied rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
