airborne12 commented on code in PR #68661:
URL: https://github.com/apache/doris/pull/68661#discussion_r4197845548
##########
be/src/exprs/function/match.cpp:
##########
@@ -241,9 +355,12 @@ inline std::vector<segment_v2::TermInfo>
FunctionMatchBase::analyse_data_token(
auto reader =
doris::segment_v2::inverted_index::InvertedIndexAnalyzer::create_reader(
analyzer_ctx->char_filter_map);
reader->init(str_ref.data, (int)str_ref.size, true);
- data_tokens =
+ auto element_tokens =
doris::segment_v2::inverted_index::InvertedIndexAnalyzer::get_analyse_result(
reader, analyzer_ctx->analyzer.get());
+ for (auto& token : element_tokens) {
Review Comment:
The row-wide token accumulation was present in the analyzed array path.
Commit `d9aeeb226c63` processes one non-null element at a time: MATCH_ANY and
MATCH_REGEXP stop after a match, MATCH_ALL retains only flags for unmatched
query terms, and phrase variants retain a query-length token window. Retained
data tokens are now bounded by one element instead of the entire array row.
`FunctionMatchTest.*` passes 33/33, and the BE build passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
