airborne12 opened a new pull request, #67219:
URL: https://github.com/apache/doris/pull/67219
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
`support_phrase=true` asks the BE to persist term positions, and a position
is only
observable when a query can supply a second term to match it against.
`InvertedIndexAnalyzer::get_analyse_result()` short-circuits on
`!should_analyzer()` and
returns the entire search string as ONE term, for every analysis purpose.
Every phrase
variant — `MATCH_PHRASE`, `MATCH_PHRASE_PREFIX`, `MATCH_PHRASE_EDGE` —
sources its terms
from there, so a phrase against an index with no tokenizer is always a
single-term phrase,
which is a term query. The option describes something no query can reach.
FE was manufacturing exactly that combination. `Index`'s constructor
unconditionally
defaulted `support_phrase` to `true` for every inverted index, including
`normalizer`-only
indexes and `parser=none` ones, so a keyword index reached the BE carrying
an option it can
never honour. Downstream, `IndexReaderHelper::is_need_similarity_score()`
read that option
alone and reported such an index as scoreable, letting relevance work be
planned against
positions and norms that were never written.
**The fix aligns both sides on the same predicate:**
- FE keeps `support_phrase` only when the index actually tokenizes — an
explicit
`analyzer`, or a `parser` other than `none` — and removes the key
otherwise, so new
tablet metadata no longer carries it.
- BE gains `IndexReaderHelper::persists_scoring_inputs()`, which requires
both
`support_phrase=yes` and `InvertedIndexAnalyzer::should_analyzer()`. Both
`is_need_similarity_score()` overloads route through it. Tablet metadata
already written
to disk still holds the stale option, so the FE-side change alone is not
sufficient and
the guard has to be repeated on the read path.
**ARRAY deserves an explicit note, because the obvious argument does not
hold for it.**
`InvertedIndexUtil::checkInvertedIndexParser` forces every ARRAY inverted
index to
`parser=none`, yet an ARRAY row emits one term per element and
`SniiIndexColumnWriter::_add_array_values` does advance positions between
them (CLucene
writes the same shape). Those positions are nevertheless unreachable, for
the query-side
reason above — not because the document holds a single term.
`UntokenizedQueriesCannotObserveAPosition` pins that invariant so a future
analyzer change
cannot silently invalidate it.
For reference, Elasticsearch rejects the equivalent shape at mapping time: a
`keyword`
field's `index_options` accepts only `docs` and `freqs`, never `positions`.
### Release note
None
### Check List (For Author)
- Test
- [x] Unit Test
- FE: `IndexTest.testSupportPhraseKeptForTokenizingIndexes`,
`IndexTest.testSupportPhraseDroppedForNonTokenizingIndexes`
(covers the
normalizer-only, `parser=none` and keyword shapes)
- BE:
`IndexReaderHelperTest.IsNeedSimilarityScoreRequiresATokenizer`,
`IndexReaderHelperTest.UntokenizedQueriesCannotObserveAPosition`
- Both sides verified RED first: reverting `Index.java` fails the FE
normalizer
assertion; reverting `index_reader_helper.h` makes the keyword and
`parser=none`
shapes report `true`.
- Full BE inverted-index suite green (3205 tests / 285 suites); FE
suite of the 14
classes touching `Index` / `support_phrase` green (149 tests).
- Behavior changed:
- [x] Yes. An inverted index with no tokenizer no longer stores
`support_phrase`, and
is no longer reported as carrying scoring inputs. No query result
changes: such an
index can only ever be queried with a single-term phrase, which is a
term query.
Existing tablets keep their stored property; the BE-side guard handles
them.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
