airborne12 commented on code in PR #66875:
URL: https://github.com/apache/doris/pull/66875#discussion_r3801865060
##########
be/src/storage/index/snii/snii_index_reader.cpp:
##########
@@ -153,6 +153,9 @@ bool
uses_plain_term_frequency_scoring(InvertedIndexQueryType query_type,
bool uses_phrase_frequency_scoring(InvertedIndexQueryType query_type,
const InvertedIndexQueryInfo& query_info) {
+ // A single-token phrase-prefix is a plain prefix query, not a
phrase-frequency query. Like V3,
+ // direct MATCH therefore publishes no BM25 score, while FunctionSearch
carries the bitmap as
+ // a constant-score BitSetQuery.
Review Comment:
Confirmed with a real custom-analyzer reproduction. `SearchDslParser` passes
`be*`; a whitespace-only char_group analyzer preserves the terminal `*`, so the
reader received a literal starred prefix and returned an empty bitmap. The fix
strips exactly the single PREFIX syntax marker before analyzed tokenization,
while keyword PREFIX remains on WILDCARD_QUERY and keeps `*`. Added
`FunctionSearchTest.TestSniiNativeAnalyzedPrefixStripsSearchDslMarker`; it
failed before the fix (`be*` vs `be`, empty vs `{0,2}`) and passes now.
##########
be/src/storage/index/snii/snii_index_reader.cpp:
##########
@@ -153,6 +153,9 @@ bool
uses_plain_term_frequency_scoring(InvertedIndexQueryType query_type,
bool uses_phrase_frequency_scoring(InvertedIndexQueryType query_type,
const InvertedIndexQueryInfo& query_info) {
+ // A single-token phrase-prefix is a plain prefix query, not a
phrase-frequency query. Like V3,
+ // direct MATCH therefore publishes no BM25 score, while FunctionSearch
carries the bitmap as
+ // a constant-score BitSetQuery.
return query_info.term_infos.size() > 1 &&
Review Comment:
Confirmed with a physical positional SNII segment that intentionally has no
scoring metadata. A similarity context caused the old code to open
`SniiStatsProvider` before considering the analyzed shape, producing `SNII
semantic scoring metadata is missing` for the single-token prefix. The fix
derives the scoring requirement after analysis: only term-/phrase-frequency
shapes open stats/norms. A single-token prefix keeps its established zero
direct score and can use query cache/singleflight. Added
`SniiIndexReaderCountFallback.PublicSingleTokenPhrasePrefixWithSimilarityDoesNotRequireScoringMetadata`,
including cold miss/insert and warm hit assertions. This is query-only and
does not change the storage format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
