This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new a8f9f5d419d branch-4.1: [fix](be) Score MATCH on keyword inverted
indexes like SEARCH (#69791)
a8f9f5d419d is described below
commit a8f9f5d419deaa4c1e04db575554d6cfdd46f48a
Author: Jack <[email protected]>
AuthorDate: Thu Nov 5 09:31:24 2026 +0800
branch-4.1: [fix](be) Score MATCH on keyword inverted indexes like SEARCH
(#69791)
### What problem does this PR solve?
Issue Number: None
Related PR: #69790
Problem Summary: score() of a MATCH on an untokenized (keyword) inverted
index is always 0, while SEARCH on the same field scores BM25:
INDEX idx_redirect (redirect) USING INVERTED
-- one row ('BP')
SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0
SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821
Elasticsearch scores a match, term or match_phrase query on a keyword
field with BM25 in query context: no norms and no frequencies, so
freq = dl = avgdl = 1 and the score is the idf of the whole value
(0.2876821 above, 0.6931471 for N=4, n=2). Only a bool filter scores 0.
SEARCH already gives exactly these values; MATCH on keyword behaved like
a filter, so `body MATCH 'apple' AND tag MATCH 'x'` ranked as if the tag
clause were not there.
Root cause on branch-4.1, three places keep keyword MATCH out of
scoring:
StringTypeInvertedIndexReader::query never sets is_similarity_score (and
answers score() queries from the query cache), MatchPredicateCollector
skips indexes without an analyzer, and is_need_similarity_score only
accepts phrase-capable or lowercase-normalizer indexes.
Fix: a keyword (non-analyzed) index is eligible for MATCH scoring
regardless of support_phrase; the keyword reader requests scoring for
the MATCH query types and bypasses the result cache like the full-text
reader does; MatchPredicateCollector collects the whole value as one
term (case and spaces kept), the term the reader looks up. The legacy
scorer already uses norm 1 when a field has none. Equality runs as
MATCH_ANY inside the reader but keeps EQUAL_QUERY for the decision, so
it stays a pure filter. MATCH_REGEXP and MATCH_PHRASE_PREFIX keep their
current scores.
### Release note
score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized)
inverted index now returns the BM25 score of the value, the same as
SEARCH and Elasticsearch, instead of 0.
### Check List (For Author)
- Test: Regression test / Unit Test / Manual test
- Regression test: new test_match_keyword_score (V2/V3, implicit and
explicit parser=none); its output equals the master PR's V2/V3
blocks
- Unit Test: IndexReaderHelperTest, CollectionStatisticsTest
- Manual test: the reported table on a branch-4.1 ASAN build; MATCH
equals SEARCH and Elasticsearch 9.5.5 to 7 digits, also for
ARRAY<STRING>, a column with both a keyword and an analyzed index,
and OR/AND mixes
- Behavior changed: Yes. Keyword MATCH contributes its BM25 score to
score() instead of 0.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../storage/compaction/collection_statistics.cpp | 17 ++-
be/src/storage/index/index_reader_helper.h | 46 ++++---
.../index/inverted/inverted_index_reader.cpp | 12 +-
be/src/storage/predicate_collector.cpp | 11 +-
.../compaction/collection_statistics_test.cpp | 16 ++-
.../storage/segment/index_reader_helper_test.cpp | 27 ++++
.../inverted_index_p0/test_match_keyword_score.out | 149 +++++++++++++++++++++
.../search/test_search_score_pattern_constant.out | 8 ++
.../test_match_keyword_score.groovy | 143 ++++++++++++++++++++
.../test_search_score_pattern_constant.groovy | 1 +
10 files changed, 393 insertions(+), 37 deletions(-)
diff --git a/be/src/storage/compaction/collection_statistics.cpp
b/be/src/storage/compaction/collection_statistics.cpp
index 94ab5cefec6..6e89aa157b2 100644
--- a/be/src/storage/compaction/collection_statistics.cpp
+++ b/be/src/storage/compaction/collection_statistics.cpp
@@ -88,6 +88,14 @@ Result<std::shared_ptr<lucene::search::IndexSearcher>>
open_optional_clucene_sta
}
return ResultError(status);
}
+ // Only text indexes have a segments file. A MATCH can reach a BKD index:
CAST(n AS STRING)
+ // MATCH '1' on an INT column, or on a VARIANT path v.n, which is typed
per segment and indexed
+ // as BKD where it holds numbers. Such an index has no terms and adds no
statistics, as a
+ // segment without the path does; its rows match without the index and
score nothing.
+ if (compound.value()->fileExists(
+
InvertedIndexDescriptor::get_temporary_bkd_index_data_file_name())) {
+ return std::shared_ptr<lucene::search::IndexSearcher> {};
+ }
lucene::index::IndexReader* reader = nullptr;
try {
reader = lucene::index::IndexReader::open(compound.value().get());
@@ -650,8 +658,13 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset, int3
}
auto searcher_variant =
inverted_index_cache_handle.get_index_searcher();
- auto index_searcher_ptr =
std::get<FulltextIndexSearcherPtr>(searcher_variant);
- index_searcher = index_searcher_ptr.get();
+ // CAST(v['n'] AS BIGINT) = 1 caches the BKD searcher of v.n under
this key, so a BKD index
+ // can also arrive from the cache: no terms, as in
open_optional_clucene_stats_searcher.
+ const auto* fulltext_searcher =
std::get_if<FulltextIndexSearcherPtr>(&searcher_variant);
+ if (fulltext_searcher == nullptr) {
+ continue;
+ }
+ index_searcher = fulltext_searcher->get();
index_reader = index_searcher->getReader();
#endif
// BM25 on an analyzed index needs the record length of every row, and
CLucene keeps
diff --git a/be/src/storage/index/index_reader_helper.h
b/be/src/storage/index/index_reader_helper.h
index 4b436feed5f..d251741ee4e 100644
--- a/be/src/storage/index/index_reader_helper.h
+++ b/be/src/storage/index/index_reader_helper.h
@@ -22,6 +22,7 @@
#include <string>
#include "storage/index/index_iterator.h"
+#include "storage/index/inverted/analyzer/analyzer.h"
#include "storage/index/inverted/inverted_index_reader.h"
namespace doris::segment_v2 {
@@ -112,18 +113,11 @@ public:
if (query_type == InvertedIndexQueryType::MATCH_REGEXP_QUERY) {
return true;
}
- if (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
- query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
- query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY ||
- query_type == InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY) {
- const auto& properties = index_meta->properties();
- if (get_parser_phrase_support_string_from_properties(properties) ==
- INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
- is_normalizer_only(properties)) {
- return true;
- }
- }
- return false;
+ return (query_type == InvertedIndexQueryType::MATCH_ANY_QUERY ||
+ query_type == InvertedIndexQueryType::MATCH_ALL_QUERY ||
+ query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY ||
+ query_type ==
InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY) &&
+ can_score_match(index_meta);
}
static bool is_need_similarity_score(TExprOpcode::type query_type,
@@ -132,17 +126,25 @@ public:
if (query_type == TExprOpcode::MATCH_REGEXP) {
return true;
}
- if (query_type == TExprOpcode::MATCH_ANY || query_type ==
TExprOpcode::MATCH_ALL ||
- query_type == TExprOpcode::MATCH_PHRASE ||
- query_type == TExprOpcode::MATCH_PHRASE_PREFIX) {
- const auto& properties = index_meta->properties();
- if (get_parser_phrase_support_string_from_properties(properties) ==
- INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
- is_normalizer_only(properties)) {
- return true;
- }
+ return (query_type == TExprOpcode::MATCH_ANY || query_type ==
TExprOpcode::MATCH_ALL ||
+ query_type == TExprOpcode::MATCH_PHRASE ||
+ query_type == TExprOpcode::MATCH_PHRASE_PREFIX) &&
+ can_score_match(index_meta);
+ }
+
+private:
+ static bool can_score_match(const TabletIndex* index_meta) {
+ const auto& properties = index_meta->properties();
+ // A keyword index keeps each value as one term, without positions or
norms, and its
+ // postings score with unit frequency and length whatever
support_phrase says, as an
+ // Elasticsearch keyword field does: over tags (x, x, y, z), tag MATCH
'x' scores
+ // idf(x) = ln(2) for both matching rows, the same as SEARCH('tag:x').
+ if
(!inverted_index::InvertedIndexAnalyzer::should_analyzer(properties)) {
+ return true;
}
- return false;
+ return get_parser_phrase_support_string_from_properties(properties) ==
+ INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES ||
+ is_normalizer_only(properties);
}
};
diff --git a/be/src/storage/index/inverted/inverted_index_reader.cpp
b/be/src/storage/index/inverted/inverted_index_reader.cpp
index 12c382fd062..1fd9dd7f5c7 100644
--- a/be/src/storage/index/inverted/inverted_index_reader.cpp
+++ b/be/src/storage/index/inverted/inverted_index_reader.cpp
@@ -537,8 +537,15 @@ Status StringTypeInvertedIndexReader::query(const
IndexQueryContextPtr& context,
search_str};
auto* cache = InvertedIndexQueryCache::instance();
InvertedIndexQueryCacheHandle cache_handler;
- if (handle_query_cache(context, cache, cache_key, &cache_handler,
bit_map)) {
- return Status::OK();
+ // A MATCH on a keyword index scores its whole value like SEARCH does,
so with score() it
+ // must run and publish its scores instead of answering from the
cached bitmap. Equality
+ // runs as MATCH_ANY below but keeps EQUAL_QUERY here, so it stays an
unscored filter.
+ const bool is_similarity_score =
+ IndexReaderHelper::is_need_similarity_score(query_type,
&_index_meta);
+ if (!(context->collection_similarity && is_similarity_score)) {
+ if (handle_query_cache(context, cache, cache_key, &cache_handler,
bit_map)) {
+ return Status::OK();
+ }
}
std::wstring column_name_ws =
StringUtil::string_to_wstring(column_name);
@@ -546,6 +553,7 @@ Status StringTypeInvertedIndexReader::query(const
IndexQueryContextPtr& context,
InvertedIndexQueryInfo query_info;
query_info.field_name = column_name_ws;
query_info.term_infos.emplace_back(search_str, 0);
+ query_info.is_similarity_score = is_similarity_score;
auto result = std::make_shared<roaring::Roaring>();
FulltextIndexSearcherPtr* searcher_ptr = nullptr;
diff --git a/be/src/storage/predicate_collector.cpp
b/be/src/storage/predicate_collector.cpp
index cfebb513f50..166386260eb 100644
--- a/be/src/storage/predicate_collector.cpp
+++ b/be/src/storage/predicate_collector.cpp
@@ -288,10 +288,6 @@ Status MatchPredicateCollector::collect(RuntimeState*
state, const TabletSchemaS
index_metas = {selected};
}
for (const auto* index_meta : index_metas) {
- if (!InvertedIndexAnalyzer::should_analyzer(index_meta->properties()))
{
- continue;
- }
-
if (!IndexReaderHelper::is_need_similarity_score(expr->op(),
index_meta)) {
continue;
}
@@ -305,7 +301,12 @@ Status MatchPredicateCollector::collect(RuntimeState*
state, const TabletSchemaS
options.timezone = &state->timezone_obj();
const std::string value = right_literal->value(options);
std::vector<TermInfo> term_infos;
- if (match_analyzer_ctx != nullptr) {
+ if (!InvertedIndexAnalyzer::should_analyzer(index_meta->properties()))
{
+ // A keyword index stores the whole value as one term, case and
spaces included, and
+ // its reader looks it up unanalyzed: tag MATCH 'Hello World'
scores the single term
+ // "Hello World", as SEARCH('tag:"Hello World"') does.
+ term_infos.emplace_back(value);
+ } else if (match_analyzer_ctx != nullptr) {
term_infos = DORIS_TRY(analyze_plain_query(value,
*match_analyzer_ctx));
} else {
term_infos = DORIS_TRY(analyze_plain_query(value,
index_meta->properties()));
diff --git a/be/test/storage/compaction/collection_statistics_test.cpp
b/be/test/storage/compaction/collection_statistics_test.cpp
index 705e2bb3b21..517068c614b 100644
--- a/be/test/storage/compaction/collection_statistics_test.cpp
+++ b/be/test/storage/compaction/collection_statistics_test.cpp
@@ -1766,10 +1766,9 @@ TEST_F(CollectionStatisticsTest,
CollectNotExtractedColumnSkipsFallback) {
EXPECT_TRUE(collect_infos.empty());
}
-// L1: Index whose properties do not request an analyzer
-// (should_analyzer returns false). The matching index_meta is iterated
-// but skipped before insertion.
-TEST_F(CollectionStatisticsTest, CollectSkipsIndexWithoutAnalyzer) {
+// L1: Index whose properties do not request an analyzer (should_analyzer
returns false). A keyword
+// MATCH scores like SEARCH on the same field, so its whole value is collected
as one term.
+TEST_F(CollectionStatisticsTest, CollectsKeywordIndexValueAsOneTerm) {
auto tablet_schema = std::make_shared<TabletSchema>();
constexpr int32_t kColUid = 1300;
@@ -1794,7 +1793,7 @@ TEST_F(CollectionStatisticsTest,
CollectSkipsIndexWithoutAnalyzer) {
auto match_expr =
std::make_shared<collection_statistics::MockVExpr>(TExprNodeType::MATCH_PRED);
auto slot_ref =
std::make_shared<collection_statistics::MockVSlotRef>("kw", SlotId(kSlotId));
- auto literal = std::make_shared<collection_statistics::MockVLiteral>("v");
+ auto literal =
std::make_shared<collection_statistics::MockVLiteral>("Hello World");
match_expr->_children.push_back(slot_ref);
match_expr->_children.push_back(literal);
@@ -1803,7 +1802,12 @@ TEST_F(CollectionStatisticsTest,
CollectSkipsIndexWithoutAnalyzer) {
auto status =
collector.collect(runtime_state_.get(), tablet_schema, match_expr,
&collect_infos);
ASSERT_TRUE(status.ok()) << status.msg();
- EXPECT_TRUE(collect_infos.empty());
+ ASSERT_EQ(collect_infos.size(), 1U);
+ const auto& collect_info = collect_infos.begin()->second;
+ ASSERT_NE(collect_info.index_meta, nullptr);
+ EXPECT_EQ(collect_info.index_meta->index_id(), 2300);
+ ASSERT_EQ(collect_info.term_infos.size(), 1U);
+ EXPECT_EQ(collect_info.term_infos.begin()->get_single_term(), "Hello
World");
}
// L2: Index whose analyzer is set (should_analyzer returns true) but does
diff --git a/be/test/storage/segment/index_reader_helper_test.cpp
b/be/test/storage/segment/index_reader_helper_test.cpp
index 352d6853249..187162c3344 100644
--- a/be/test/storage/segment/index_reader_helper_test.cpp
+++ b/be/test/storage/segment/index_reader_helper_test.cpp
@@ -439,6 +439,33 @@ TEST_F(IndexReaderHelperTest,
NormalizerOnlyPhraseAndScoringNeedNoPositions) {
InvertedIndexQueryType::MATCH_ANY_QUERY, &custom_meta));
}
+TEST_F(IndexReaderHelperTest, MatchScoringAcceptsKeywordIndexes) {
+ // Keyword indexes score with unit frequency and length, so phrase support
does not matter.
+ for (const auto& properties : std::vector<std::map<std::string,
std::string>> {
+ {}, {{"parser", "none"}}, {{"parser", "none"},
{"support_phrase", "true"}}}) {
+ SCOPED_TRACE(::testing::PrintToString(properties));
+ TabletIndexPB pb;
+ pb.set_index_id(1);
+ pb.set_index_name("keyword_index");
+ pb.set_index_type(IndexType::INVERTED);
+ pb.add_col_unique_id(1);
+ for (const auto& [key, value] : properties) {
+ (*pb.mutable_properties())[key] = value;
+ }
+ TabletIndex index;
+ index.init_from_pb(pb);
+ for (auto op : {TExprOpcode::MATCH_ANY, TExprOpcode::MATCH_ALL,
TExprOpcode::MATCH_PHRASE,
+ TExprOpcode::MATCH_PHRASE_PREFIX}) {
+ EXPECT_TRUE(IndexReaderHelper::is_need_similarity_score(op,
&index));
+ }
+ EXPECT_TRUE(IndexReaderHelper::is_need_similarity_score(
+ InvertedIndexQueryType::MATCH_ANY_QUERY, &index));
+
EXPECT_FALSE(IndexReaderHelper::is_need_similarity_score(TExprOpcode::EQ,
&index));
+ EXPECT_FALSE(IndexReaderHelper::is_need_similarity_score(
+ InvertedIndexQueryType::EQUAL_QUERY, &index));
+ }
+}
+
TEST_F(IndexReaderHelperTest, RegexpScoresConstantOnEveryIndex) {
// A regexp scores a constant 1 and reads no statistics, so neither the
analyzer nor phrase
// support decides whether it scores.
diff --git
a/regression-test/data/inverted_index_p0/test_match_keyword_score.out
b/regression-test/data/inverted_index_p0/test_match_keyword_score.out
new file mode 100644
index 00000000000..ec56ffcd1a1
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_match_keyword_score.out
@@ -0,0 +1,149 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !V2_single_row_match --
+1 0.287682
+
+-- !V2_match --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_match_all --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_match_phrase --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_search --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_none_match --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V2_none_match_phrase --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V2_none_search --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V2_analyzed_and_keyword --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V2_search_and --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V2_analyzed_and_equality --
+1 0.290716
+2 0.20021
+5 0.268636
+
+-- !V2_analyzed_and_numeric_cast --
+1 0.290716
+4 0.268636
+
+-- !V2_variant_numeric_path --
+1 0
+3 0
+4 0.287682
+
+-- !V2_variant_text_path --
+1 0.693147
+4 0.693147
+
+-- !V2_variant_sparse_path --
+1 0.980829
+
+-- !V2_variant_numeric_path_cached_bkd --
+1 0
+3 0
+4 0.287682
+
+-- !V3_single_row_match --
+1 0.287682
+
+-- !V3_match --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_match_all --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_match_phrase --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_search --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_none_match --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V3_none_match_phrase --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V3_none_search --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V3_analyzed_and_keyword --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V3_search_and --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V3_analyzed_and_equality --
+1 0.290716
+2 0.20021
+5 0.268636
+
+-- !V3_analyzed_and_numeric_cast --
+1 0.290716
+4 0.268636
+
+-- !V3_variant_numeric_path --
+1 0
+3 0
+4 0.287682
+
+-- !V3_variant_text_path --
+1 0.693147
+4 0.693147
+
+-- !V3_variant_sparse_path --
+1 0.980829
+
+-- !V3_variant_numeric_path_cached_bkd --
+1 0
+3 0
+4 0.287682
+
diff --git a/regression-test/data/search/test_search_score_pattern_constant.out
b/regression-test/data/search/test_search_score_pattern_constant.out
index f2a3c30f31d..6af1ee8e6c2 100644
--- a/regression-test/data/search/test_search_score_pattern_constant.out
+++ b/regression-test/data/search/test_search_score_pattern_constant.out
@@ -23,6 +23,10 @@
3 1
6 1
+-- !V2_match_regexp_keyword --
+6 1
+7 1
+
-- !V2_match_and_match_regexp --
2 2.023573
@@ -60,6 +64,10 @@
3 1
6 1
+-- !V3_match_regexp_keyword --
+6 1
+7 1
+
-- !V3_match_and_match_regexp --
2 2.023573
diff --git
a/regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy
b/regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy
new file mode 100644
index 00000000000..56daca02b31
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// score() over MATCH on a keyword (untokenized) inverted index. Like an
Elasticsearch match or term
+// query on a keyword field, it scores BM25 with unit frequency and length, so
every matching row
+// scores the idf of the whole value, the same as SEARCH on that field; it
used to score 0. An
+// equality predicate stays a pure filter.
+suite("test_match_keyword_score", "p0") {
+ // MATCH and SEARCH need index execution even when CI fuzzes these session
defaults.
+ sql "set enable_common_expr_pushdown = true"
+ sql "set enable_inverted_index_query = true"
+ sql "set enable_match_without_inverted_index = true"
+
+ for (def storageFormat : ["V2", "V3"]) {
+ sql "DROP TABLE IF EXISTS test_match_keyword_score"
+ sql """
+ CREATE TABLE test_match_keyword_score (
+ id INT NOT NULL,
+ body STRING NULL,
+ tag STRING NULL,
+ tag_none STRING NULL,
+ n INT NULL,
+ INDEX idx_body(body) USING INVERTED
+ PROPERTIES("parser" = "english", "support_phrase" =
"true"),
+ INDEX idx_tag(tag) USING INVERTED PROPERTIES("ignore_above" =
"1024"),
+ INDEX idx_tag_none(tag_none) USING INVERTED
+ PROPERTIES("parser" = "none", "support_phrase" = "true"),
+ INDEX idx_n(n) USING INVERTED
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default:1",
+ "inverted_index_storage_format" = "${storageFormat}"
+ )
+ """
+ // The first row alone is the reported case: one document scores idf =
ln(4/3) = 0.287682.
+ sql """INSERT INTO test_match_keyword_score VALUES (1, 'apple apple',
'BP', 'Hello World', 1)"""
+ sql "sync"
+ quickTest("${storageFormat}_single_row_match", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM test_match_keyword_score
+ WHERE tag MATCH 'BP' ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+
+ sql """INSERT INTO test_match_keyword_score VALUES
+ (2, 'apple pear', 'BP', 'hello world', 2), (3, 'pear', 'other',
'Hello World', 1),
+ (4, 'apple', 'x', 'Hello', 1), (5, 'apple', 'BP', NULL, NULL),
+ (6, 'apple', NULL, 'Hello World', 2)
+ """
+ sql "sync"
+
+ // The whole value is one term: MATCH, MATCH_ALL and MATCH_PHRASE
score as SEARCH does.
+ for (def query : [
+ ["match", "tag MATCH 'BP'"],
+ ["match_all", "tag MATCH_ALL 'BP'"],
+ ["match_phrase", "tag MATCH_PHRASE 'BP'"],
+ ["search", "SEARCH('tag:BP')"],
+ // Case and spaces are kept, and phrase support inherited by
parser=none is ignored.
+ ["none_match", "tag_none MATCH 'Hello World'"],
+ ["none_match_phrase", "tag_none MATCH_PHRASE 'Hello World'"],
+ ["none_search", "SEARCH('tag_none:\"Hello World\"')"],
+ // Both leaves score: idf(apple) from body plus idf(BP) from
tag.
+ ["analyzed_and_keyword", "body MATCH 'apple' AND tag MATCH
'BP'"],
+ ["search_and", "SEARCH('body:apple AND tag:BP')"],
+ // An equality is a filter: only the body leaf scores.
+ ["analyzed_and_equality", "body MATCH 'apple' AND tag = 'BP'"],
+ // A MATCH on a number cast to text is a filter over the BKD
index of n: only the
+ // body leaf scores. Opening that BKD index for statistics
used to abort the BE.
+ ["analyzed_and_numeric_cast", "body MATCH 'apple' AND CAST(n
AS STRING) MATCH '1'"]]) {
+ quickTest("${storageFormat}_${query[0]}", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM test_match_keyword_score
+ WHERE ${query[1]} ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+ }
+
+ // A VARIANT path is typed per segment: n holds numbers, indexed as
BKD, in the first
+ // insert and text in the second. The plain VARIANT index makes every
path a keyword
+ // index, so CAST(v['n'] AS STRING) MATCH collects statistics for n;
the BKD segment adds
+ // none instead of aborting the BE, and its rows match without the
index with score 0. p
+ // exists only in the first insert: the other segment has no index for
it and adds nothing.
+ sql "set default_variant_enable_typed_paths_to_sparse = false"
+ sql "set default_variant_enable_doc_mode = false"
+ sql "DROP TABLE IF EXISTS test_match_keyword_score_variant"
+ sql """
+ CREATE TABLE test_match_keyword_score_variant (
+ id INT NOT NULL,
+ v VARIANT NULL,
+ INDEX idx_v(v) USING INVERTED
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default:1",
+ "inverted_index_storage_format" = "${storageFormat}",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """INSERT INTO test_match_keyword_score_variant VALUES
+ (1, '{"n": 1, "s": "abc", "p": "abc"}'), (2, '{"n": 2, "s": "abd",
"p": "abd"}'),
+ (3, '{"n": 1, "s": "xyz", "p": "xyz"}')
+ """
+ sql """INSERT INTO test_match_keyword_score_variant VALUES (4, '{"n":
"1", "s": "abc"}')"""
+ sql "sync"
+ for (def query : [
+ ["variant_numeric_path", "CAST(v['n'] AS STRING) MATCH '1'"],
+ ["variant_text_path", "CAST(v['s'] AS STRING) MATCH 'abc'"],
+ ["variant_sparse_path", "CAST(v['p'] AS STRING) MATCH
'abc'"]]) {
+ quickTest("${storageFormat}_${query[0]}", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM
test_match_keyword_score_variant
+ WHERE ${query[1]} ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+ }
+ // A numeric predicate leaves the BKD searcher of n in the searcher
cache, where collecting
+ // statistics finds it next: it used to take it for a text searcher
and abort the BE.
+ sql "SELECT COUNT(*) FROM test_match_keyword_score_variant WHERE
CAST(v['n'] AS BIGINT) = 1"
+ quickTest("${storageFormat}_variant_numeric_path_cached_bkd", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM test_match_keyword_score_variant
+ WHERE CAST(v['n'] AS STRING) MATCH '1' ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+ }
+}
diff --git
a/regression-test/suites/search/test_search_score_pattern_constant.groovy
b/regression-test/suites/search/test_search_score_pattern_constant.groovy
index 430f313b10f..ad8117c9256 100644
--- a/regression-test/suites/search/test_search_score_pattern_constant.groovy
+++ b/regression-test/suites/search/test_search_score_pattern_constant.groovy
@@ -62,6 +62,7 @@ suite("test_search_score_pattern_constant", "p0") {
["search_prefix", "SEARCH('content:te*')"],
["search_keyword_regexp", "SEARCH('tag:/text.*/')"],
["match_regexp", "content MATCH_REGEXP 'ext'"],
+ ["match_regexp_keyword", "tag MATCH_REGEXP 'text'"],
// A pattern next to a scored clause adds exactly 1, as an
Elasticsearch bool must
// with a regexp clause does.
["match_and_match_regexp", "content MATCH 'search' AND content
MATCH_REGEXP 'ext'"],
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
