github-actions[bot] commented on code in PR #66875:
URL: https://github.com/apache/doris/pull/66875#discussion_r3801120383
##########
be/src/storage/index/snii/snii_index_reader.cpp:
##########
@@ -153,6 +153,9 @@ bool
uses_plain_term_frequency_scoring(InvertedIndexQueryType query_type,
bool uses_phrase_frequency_scoring(InvertedIndexQueryType query_type,
const InvertedIndexQueryInfo& query_info) {
+ // A single-token phrase-prefix is a plain prefix query, not a
phrase-frequency query. Like V3,
+ // direct MATCH therefore publishes no BM25 score, while FunctionSearch
carries the bitmap as
+ // a constant-score BitSetQuery.
return query_info.term_infos.size() > 1 &&
Review Comment:
[P1] Gate scoring setup on the analyzed shape
With `size() > 1`, a one-token prefix has neither a phrase-frequency nor
plain-term scorer, but `_query()` still treats `actual_similarity` as
sufficient to call `SniiStatsProvider::open()` before checking either
predicate. This means scored `SEARCH` PREFIX still fails on a valid positional
SNII segment without CommonGrams scoring metadata: `SearchPredicateCollector`
intentionally excludes PREFIX, while `FunctionSearch` supplies a private
similarity sink for the constant-score fallback, and `open()` returns
`INVERTED_INDEX_NOT_SUPPORTED`. On scoring segments it also reads the whole
norms POD and bypasses cache/single-flight for a score it never publishes.
Please derive an actual frequency-scoring flag from the analyzed shape and use
it to gate the stats/cache policy so this case reaches the intended direct-0 /
SEARCH-1.0 result.
##########
be/src/storage/index/snii/snii_index_reader.cpp:
##########
@@ -153,6 +153,9 @@ bool
uses_plain_term_frequency_scoring(InvertedIndexQueryType query_type,
bool uses_phrase_frequency_scoring(InvertedIndexQueryType query_type,
const InvertedIndexQueryInfo& query_info) {
+ // A single-token phrase-prefix is a plain prefix query, not a
phrase-frequency query. Like V3,
+ // direct MATCH therefore publishes no BM25 score, while FunctionSearch
carries the bitmap as
+ // a constant-score BitSetQuery.
Review Comment:
[P1] Normalize the SEARCH prefix token before SNII analysis
The real `FunctionSearch` PREFIX path does not pass the cleaned term used by
these tests: the FE DSL preserves the terminal `*`, and FunctionSearch forwards
it unchanged for analysed SNII fields. That happens to work with tokenizers
that discard punctuation, but the scoring-capable CommonGrams provider used
here is a `char_group` tokenizer split only on whitespace, so `be*` remains one
term and SNII executes `prefix_query(..., "be*")`, which cannot match stored
terms such as `beta`. V3 routes the pattern through `WildcardQuery` and does
match. Please strip the one syntactic terminal `*` at the SNII FunctionSearch
boundary (or route an equivalent raw wildcard/prefix operation) and cover the
real SEARCH surface rather than an already-clean term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
