github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4070880347
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java:
##########
@@ -337,23 +339,48 @@ private static void
checkInvertedIndexProperties(Map<String, String> properties,
// dict_compression now silently ignores by V2/V3 inverted index
}
- // Normalize analyzer and normalizer names to lowercase for
case-insensitive matching
+ // Canonicalize built-ins while retaining the exact spelling of a
resolved legacy policy.
normalizeInvertedIndexProperties(properties);
}
/**
- * Normalize analyzer and normalizer names in index properties to
lowercase.
- * This ensures case-insensitive matching between table creation and query
time.
+ * Canonicalize analyzer and normalizer names in index properties. Legacy
metadata may contain
+ * case-distinct policy names, so a resolved custom policy must keep its
exact stored name.
*/
private static void normalizeInvertedIndexProperties(Map<String, String>
properties) {
+ resolvePolicyNames(properties);
AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(
properties,
- INVERTED_INDEX_ANALYZER_NAME_KEY,
- INVERTED_INDEX_NORMALIZER_NAME_KEY,
INVERTED_INDEX_PARSER_KEY,
INVERTED_INDEX_PARSER_KEY_ALIAS);
}
+ /** Store analyzer and normalizer names in the spelling BE dispatches on.
*/
+ public static void resolvePolicyNames(Map<String, String> properties) {
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_ANALYZER_NAME_KEY);
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_NORMALIZER_NAME_KEY);
+ }
+
+ private static void normalizeResolvedPolicyName(Map<String, String>
properties, String key) {
+ String name = properties.get(key);
+ if (name == null || name.isEmpty()) {
+ return;
+ }
+ properties.put(key, resolveAnalyzerName(name));
+ }
+
+ /** Resolve built-in names and retain the stored spelling of custom
policies. */
+ public static String resolveAnalyzerName(String name) {
+ String trimmedName = name.trim();
+ // Match the BE writer's case-sensitive built-in dispatch before
policy lookup.
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(trimmedName)
+ || IndexPolicy.BUILTIN_NORMALIZERS.contains(trimmedName)) {
+ return trimmedName;
Review Comment:
[P1] Keep mixed-case builtin resolution consistent with validation and BE
dispatch. With only a replayed `LOWERCASE` policy, `normalizer=LowerCase`
passes `validateNormalizerExists()` as the builtin (no exact `LowerCase`, then
normalized builtin), but this lookup falls through to the normalized legacy
policy and rewrites the stored value to exact `LOWERCASE`; BE then binds that
policy instead. Analyzer builtins have the same split with a normalized
wrong-family legacy policy. Please share an exact-policy -> builtin ->
normalized-policy resolver between validation and this property/MATCH path, and
cover mixed-case builtin spellings with replayed collisions.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -72,7 +177,7 @@ private static String resolveAnalyzerIdentity(String
analyzerName, String defaul
// Check if it's a built-in normalizer
if (IndexPolicy.BUILTIN_NORMALIZERS.contains(analyzerName)) {
Review Comment:
[P1] Resolve an exact legacy normalizer before assigning builtin identity.
BE's top-level order lets an exact replayed policy named `lowercase` shadow the
builtin, but this branch (and `customAnalyzerFoldContext`) always models
keyword-plus-lowercase first. A legacy `lowercase` normalizer backed by
`asciifolding`, for example, can therefore be rejected as a duplicate of a
distinct pipeline or admit an equivalent alias. Consult the exact binding
before the builtin special case and cover CREATE/ALTER identity with an exact
replayed `lowercase` policy.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -226,26 +858,381 @@ private static String resolveTokenFilterIdentity(String
filterList) {
* IMPORTANT: Order is preserved because filter order is semantically
significant.
*/
private static String resolveCharFilterIdentity(String filterList) {
+ return resolveCharFilterIdentity(filterList, null);
+ }
+
+ private static String resolveCharFilterIdentity(String filterList,
FoldContext downstreamFold) {
+ ArrayDeque<String> identities = new ArrayDeque<>();
+ walkCharFilters(filterList, downstreamFold, identities);
+ return String.join(",", identities);
+ }
+
+ /**
+ * Resolve the chain from its last filter to its first, collecting
identities, and return the
+ * case-folding context that a filter placed in front of the chain would
run in.
+ */
+ private static FoldContext walkCharFilters(
+ String filterList, FoldContext downstreamFold, Deque<String>
identities) {
+ FoldContext fold = downstreamFold;
if (Strings.isNullOrEmpty(filterList)) {
- return "";
+ return fold;
}
- StringBuilder sb = new StringBuilder();
String[] filters = filterList.split(",\\s*");
// DO NOT sort - filter order is semantically significant
- for (int i = 0; i < filters.length; i++) {
- String filter = filters[i].trim();
- if (i > 0) {
- sb.append(",");
+ for (int i = filters.length - 1; i >= 0; --i) {
+ String filterName = filters[i].trim();
+ String filter = resolveComponentIdentity(filterName,
IndexPolicyTypeEnum.CHAR_FILTER, fold);
+ if (Strings.isNullOrEmpty(filter)) {
+ continue;
}
+ identities.addFirst(filter);
+ fold = foldContextBefore(filterName, fold);
+ }
+ return fold;
+ }
- if (IndexPolicy.BUILTIN_CHAR_FILTERS.contains(filter)) {
- sb.append(filter);
- } else {
- sb.append(resolveComponentIdentity(filter,
IndexPolicyTypeEnum.CHAR_FILTER));
+ /**
+ * Context for the filter that runs before this one: a case fold starts a
fresh context, a
+ * char_replace filter adds the bytes it rewrites, and any other filter
ends the context.
+ */
+ private static FoldContext foldContextBefore(String filterName,
FoldContext fold) {
+ FoldContext caseFold = caseFoldingCharFilterContext(filterName);
+ if (caseFold != null) {
+ return caseFold;
+ }
+ if (fold == null) {
+ return null;
+ }
+ boolean[] sourceBytes = charReplaceSourceBytes(filterName);
+ if (sourceBytes == null) {
+ return null;
+ }
+ fold.block(sourceBytes);
+ return fold;
+ }
+
+ /** Bytes a named char_replace filter rewrites, or null for any other
filter. */
+ private static boolean[] charReplaceSourceBytes(String filterName) {
+ IndexPolicy policy = findPolicy(filterName,
IndexPolicyTypeEnum.CHAR_FILTER);
+ if (policy == null || policy.isInvalid() || policy.getProperties() ==
null) {
+ return null;
+ }
+ Map<String, String> properties = policy.getProperties();
+ String type = normalizeBuiltinComponentName(
+ properties.get(IndexPolicy.PROP_TYPE),
IndexPolicyTypeEnum.CHAR_FILTER);
+ String pattern = properties.get("pattern");
+ if (!"char_replace".equals(type) || pattern == null) {
+ return null;
+ }
+ // Replacing the single replacement byte with itself leaves the stream
unchanged.
+ String replacement = properties.getOrDefault("replacement", " ");
+ int replacementByte = replacement.length() == 1 &&
replacement.charAt(0) < 128 ? replacement.charAt(0) : -1;
+ boolean[] sourceBytes = new boolean[256];
+ for (int i = 0; i < pattern.length(); ++i) {
+ char patternByte = pattern.charAt(i);
+ if (patternByte < sourceBytes.length && patternByte !=
replacementByte) {
+ sourceBytes[patternByte] = true;
}
}
- return sb.toString();
+ return sourceBytes;
+ }
+
+ /** The named policy when one exists with the expected type, or null. */
+ private static IndexPolicy findPolicy(String name, IndexPolicyTypeEnum
expectedType) {
+ if (Strings.isNullOrEmpty(name)) {
+ return null;
+ }
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ IndexPolicy policy =
env.getIndexPolicyMgr().getPolicyByName(name);
+ if (policy != null && policy.getType() == expectedType) {
+ return policy;
+ }
+ }
+ } catch (RuntimeException e) {
+ // Treat lookup failures as an unknown policy.
+ }
+ return null;
+ }
+
+ /** Fold context started by a named or built-in case-folding char filter,
or null for any other filter. */
+ private static FoldContext caseFoldingCharFilterContext(String name) {
+ if (Strings.isNullOrEmpty(name)) {
+ return null;
+ }
+
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ IndexPolicy policy =
env.getIndexPolicyMgr().getPolicyByName(name);
+ if (policy != null && policy.getType() ==
IndexPolicyTypeEnum.CHAR_FILTER) {
+ if (policy.isInvalid()) {
+ return null;
+ }
+ Map<String, String> properties = policy.getProperties();
+ if (properties != null && !properties.isEmpty()) {
+ String type = normalizeBuiltinComponentName(
+ properties.get(IndexPolicy.PROP_TYPE),
IndexPolicyTypeEnum.CHAR_FILTER);
+ return "icu_normalizer".equals(type) ?
icuNormalizerFoldContext(properties) : null;
+ }
+ }
+ }
+ } catch (RuntimeException e) {
+ // Fall through to built-in resolution.
+ }
+
+ return "icu_normalizer".equals(normalizeBuiltinComponentName(name,
IndexPolicyTypeEnum.CHAR_FILTER))
+ ? FoldContext.unfiltered() : null;
+ }
+
+ /**
+ * Fold context of an icu_normalizer component: the default nfkc_cf form
folds case over every
+ * code point, or only inside a parsable non-empty unicode_set_filter.
Null for other forms.
+ */
+ private static FoldContext icuNormalizerFoldContext(Map<String, String>
properties) {
+ if (!"nfkc_cf".equals(icuNormalizerName(properties))) {
+ return null;
+ }
+ String filter = properties.get("unicode_set_filter");
+ if (filter == null || filter.isEmpty()) {
+ return FoldContext.unfiltered();
+ }
+ try {
+ UnicodeSet unicodeSet = new UnicodeSet(filter);
+ return unicodeSet.isEmpty() ? FoldContext.unfiltered() : new
FoldContext(unicodeSet.freeze());
+ } catch (IllegalArgumentException e) {
+ return null;
+ }
+ }
+
+ /** Whether an icu_normalizer component leaves ASCII letters as they are.
*/
+ private static boolean isAsciiCaseTransparentIcuNormalizer(Map<String,
String> properties) {
+ String name = icuNormalizerName(properties);
+ return "nfc".equals(name) || "nfd".equals(name) || "nfkc".equals(name)
|| "nfkd".equals(name);
+ }
+
+ private static String icuNormalizerName(Map<String, String> properties) {
+ return properties.getOrDefault("name",
"nfkc_cf").trim().toLowerCase(Locale.ROOT);
+ }
+
+ /** The outer char filter runs before everything else, so it takes the
analyzer's fold context. */
+ private static String appendOuterCharFilterIdentity(
+ String analyzerIdentity, Map<String, String> properties,
FoldContext fold) {
+ String type =
properties.get(InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE);
+ String pattern =
properties.get(InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN);
+ if (!"char_replace".equals(type) || Strings.isNullOrEmpty(pattern)) {
+ return analyzerIdentity;
+ }
+ String replacement = properties.getOrDefault(
+
InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT, " ");
+ String canonicalPattern = canonicalizeCharReplacePattern(pattern,
replacement, fold);
+ if (canonicalPattern.isEmpty()) {
+ return analyzerIdentity;
+ }
+ return analyzerIdentity + "|outer_char_filter=char_replace:"
+ + canonicalPattern.length() + ":" + canonicalPattern + ":"
+ + replacement.length() + ":" + replacement + ";";
+ }
+
+ /**
+ * Canonicalize the ASCII pattern to the BE filter's byte set.
+ * Order, duplicate bytes, and replacements of a byte with itself do not
change the stream.
+ */
+ private static String canonicalizeCharReplacePattern(
+ String pattern, String replacement, FoldContext fold) {
+ if (replacement.length() != 1) {
+ return pattern;
+ }
+ char replacementByte = replacement.charAt(0);
+ boolean[] replacedBytes = new boolean[256];
+ for (int i = 0; i < pattern.length(); ++i) {
+ char patternByte = pattern.charAt(i);
+ if (patternByte < replacedBytes.length && patternByte !=
replacementByte) {
+ replacedBytes[patternByte] = true;
+ }
+ }
+ if (fold != null && replacementByte >= 'a' && replacementByte <= 'z') {
+ // The downstream fold maps the upper-case byte to the replacement
anyway.
+ int upperByte = replacementByte - ('a' - 'A');
+ if (fold.foldsByte(upperByte, replacementByte)) {
+ replacedBytes[upperByte] = false;
+ }
+ }
+
+ StringBuilder canonical = new StringBuilder();
+ for (int i = 0; i < replacedBytes.length; ++i) {
+ if (replacedBytes[i]) {
+ canonical.append((char) i);
+ }
+ }
+ return canonical.toString();
+ }
+
+ private static FoldContext builtinIkFoldContext(String analyzerIdentity) {
+ return isDefaultLowercaseBuiltinIkIdentity(analyzerIdentity) ?
FoldContext.unfiltered() : null;
+ }
+
+ private static boolean isDefaultLowercaseBuiltinIkIdentity(String
analyzerIdentity) {
+ return (IndexPolicyTypeEnum.ANALYZER.name() +
":tokenizer=ik_smart;").equals(analyzerIdentity)
+ || (IndexPolicyTypeEnum.ANALYZER.name() +
":tokenizer=ik_max_word;").equals(analyzerIdentity);
+ }
+
+ /**
+ * Fold context for the outer char filter of a custom analyzer or
normalizer, which BE applies
+ * before the policy's own char filters. Unknown or unresolvable policies
get no context.
+ */
+ private static FoldContext customAnalyzerFoldContext(String analyzerName) {
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(analyzerName)) {
+ return null;
+ }
+ if (IndexPolicy.BUILTIN_NORMALIZERS.contains(analyzerName)) {
+ // The built-in normalizer lowercases keyword tokens without char
filters of its own.
+ return FoldContext.unfiltered();
+ }
+ IndexPolicy policy = findPolicy(analyzerName,
IndexPolicyTypeEnum.ANALYZER);
+ if (policy == null) {
+ policy = findPolicy(analyzerName, IndexPolicyTypeEnum.NORMALIZER);
+ }
+ if (policy == null || policy.isInvalid() || policy.getProperties() ==
null
+ || policy.getProperties().isEmpty()) {
+ return null;
+ }
+ Map<String, String> properties = policy.getProperties();
+ try {
+ String tokenizerIdentity = resolveComponentIdentity(
+ properties.get(IndexPolicy.PROP_TOKENIZER),
IndexPolicyTypeEnum.TOKENIZER);
+ return
walkCharFilters(properties.get(IndexPolicy.PROP_CHAR_FILTER),
+ foldsAsciiCaseAfterCharFilters(policy.getType(),
properties, tokenizerIdentity),
+ new ArrayDeque<>());
+ } catch (RuntimeException e) {
+ return null;
+ }
+ }
+
+ /**
+ * The fold the tokenizer and token filters apply to ASCII letters, so a
char filter that only
+ * lowercases such a letter cannot change the output, or null when they
keep case.
+ */
+ private static FoldContext foldsAsciiCaseAfterCharFilters(
+ IndexPolicyTypeEnum type, Map<String, String> properties, String
tokenizerIdentity) {
+ if (type == IndexPolicyTypeEnum.NORMALIZER) {
+ // A normalizer always tokenizes with keyword, which is case
transparent.
+ return
tokenFiltersFoldAsciiCase(properties.get(IndexPolicy.PROP_TOKEN_FILTER));
+ }
+ if ("ik_smart".equals(tokenizerIdentity) ||
"ik_max_word".equals(tokenizerIdentity)) {
+ return FoldContext.unfiltered();
+ }
+ return
isCaseTransparentTokenizer(properties.get(IndexPolicy.PROP_TOKENIZER))
+ ?
tokenFiltersFoldAsciiCase(properties.get(IndexPolicy.PROP_TOKEN_FILTER)) : null;
+ }
+
+ /** Whether the tokenizer splits and emits ASCII letters the same way
regardless of their case. */
+ private static boolean isCaseTransparentTokenizer(String name) {
+ TreeMap<String, String> settings = resolveComponentSettings(name,
IndexPolicyTypeEnum.TOKENIZER);
+ if (settings == null) {
+ return false;
+ }
+ switch (settings.get(IndexPolicy.PROP_TYPE)) {
Review Comment:
[P1] Compute case transparency from the same canonical tokenizer settings
used in the base identity. The base path drops NGram `custom_token_chars=A`
when `token_chars=letter` already covers it (and likewise drops a redundant
CharGroup literal `A`), but this later check reloads raw properties and treats
the redundant alias as case-sensitive. With a downstream lowercase filter,
identical aliases then differ only because `A->a` is retained for one identity,
so CREATE/ALTER can admit duplicate indexes. Canonicalize these settings before
the transparency check and cover both NGram and CharGroup aliases.
##########
be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp:
##########
@@ -74,6 +75,10 @@ Token* NGramTokenizer::next(Token* token) {
to_chars(_buffer, _buffer_start, _gram_size);
set(token, _utf8_buffer);
+ set_source_byte_offsets(_utf8_buffer, _offset);
+ token->setStartOffset(correct_source_offset(_offset));
Review Comment:
[P1] Preserve the full source span when an n-gram cuts through a char-filter
expansion. For `nfkc_cf`, source U+FB01 becomes `fi`; with 1-grams the second
gram uses normalized slice `[1,2)`, whose corrected endpoints are both byte 3.
Because `set_source_byte_offsets()` only sees that local slice, it publishes
`[0,0]`, and these new assignments expose token/provenance range `[3,3)` for
`i`; an offset-aware Pinyin filter cannot recover the missing start. Build
correction intervals for the full normalized input and slice them (or add a
range-aware correction API), and cover the ligature -> 1-gram chain through
reset.
##########
be/src/storage/index/inverted/tokenizer/icu/icu_tokenizer.cpp:
##########
@@ -54,26 +58,78 @@ Token* ICUTokenizer::next(Token* token) {
utf8Str_.clear();
int32_t length = std::min(end - start, LUCENE_MAX_WORD_LEN);
+ if (length < end - start && length > 0 && U16_IS_LEAD(buffer_.charAt(start
+ length - 1)) &&
+ U16_IS_TRAIL(buffer_.charAt(start + length))) {
+ --length;
+ }
auto subString = buffer_.tempSubString(start, length);
+ sourceUtf8Str_.clear();
Review Comment:
[P2] Skip this second UTF-16-to-UTF-8 conversion when source tracking is
disabled. `sourceUtf8Str_` is used only by `set_source_byte_offsets()`, whose
base implementation immediately returns in the ordinary non-opted-in ICU path;
with lowercase disabled it is also byte-for-byte identical to `utf8Str_`.
Normal indexing and query analysis now transcode every token twice and retain a
second scratch buffer for no observable result. Gate the source conversion on
the provenance opt-in (and reuse the output when possible), with
disabled/enabled coverage.
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java:
##########
@@ -93,14 +149,13 @@ public List<IndexPolicy> getCopiedIndexPolicies() {
public void validateAnalyzerExists(String analyzerName) throws
DdlException {
String normalizedName = normalizeKey(analyzerName);
- // Built-in analyzers are stored in lowercase, so use normalized name
for comparison
- if (IndexPolicy.BUILTIN_ANALYZERS.contains(normalizedName)) {
- return;
- }
-
readLock();
Review Comment:
[P2] Let a canonical builtin analyzer win validation before an exact
replayed policy. With an exact legacy TOKENIZER named `ik`, `analyzer=ik` now
binds that policy here and fails as the wrong family (an invalid analyzer
policy also fails), even though `resolveAnalyzerName("ik")` and BE's analyzer
factory both intercept canonical `ik` as the builtin before policy lookup. The
pre-change validator accepted this case. Check canonical analyzer builtins
first, while retaining exact-first behavior for noncanonical names and
normalizers, and cover CREATE/ALTER replay collisions.
##########
be/src/storage/index/inverted/tokenizer/pinyin/pinyin_tokenizer.cpp:
##########
@@ -48,6 +48,9 @@
PinyinTokenizer::PinyinTokenizer(std::shared_ptr<doris::segment_v2::PinyinConfig
void PinyinTokenizer::reset() {
DorisTokenizer::reset();
+ has_current_span_ = false;
+ ascii_buff_rune_starts_.clear();
Review Comment:
[P1] Avoid allocating and retaining these new provenance vectors when
offsets are ignored. With default Pinyin settings
(`ignore_pinyin_offset=true`), every byte of a large ASCII-alphanumeric value
still appends an `int32_t` to both vectors, even though `next()` discards the
candidate ranges and uses the whole-input span. A 100 MiB value thus adds
roughly 800 MiB of unnecessary peak and retained capacity to the cached
analyzer; reset only `clear()`s it. Gate collection on the effective offset
need, release oversized enabled-mode scratch on reset, and cover large ->
empty/small reuse.
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java:
##########
@@ -508,11 +602,10 @@ public void dropIndexPolicy(boolean isIfExists, String
indexPolicyName,
* tables, and indexes. In large-scale clusters with many tables, this can
be slow.
* Consider maintaining a reverse index (analyzer -> tables) if this
becomes a bottleneck.
*
- * @param analyzerName the analyzer name to check
+ * @param analyzer the analyzer policy to check
Review Comment:
[P2] Apply top-level builtin precedence when checking DROP dependencies. A
replayed policy `IK` plus an index stored with canonical `analyzer=ik` is
treated here as a reference to `IK` through normalized fallback, so `DROP
ANALYZER IK` is rejected even though runtime dispatch uses builtin `ik`.
Replayed `LOWERCASE` versus builtin normalizer `lowercase` has the same false
dependency. Use the runtime-equivalent top-level resolver for index references
(while keeping nested component lookup semantics) and add both collision cases.
##########
be/src/runtime/index_policy/index_policy_mgr.cpp:
##########
@@ -142,50 +205,61 @@ AnalyzerPtr IndexPolicyMgr::get_policy_by_name(const
std::string& name) {
AnalyzerPtr IndexPolicyMgr::get_analyzer_by_name(const std::string& name) {
std::shared_lock lock(_mutex);
const std::string normalized_name = normalize_name(name);
- auto name_it = _name_to_id.find(normalized_name);
- if (name_it == _name_to_id.end()) {
- if (is_builtin_normalizer(normalized_name)) {
- return build_builtin_normalizer(name);
+ bool builtin_normalizer = false;
+ const auto* index_policy = find_top_level_policy_locked(name,
&builtin_normalizer);
+ if (index_policy == nullptr) {
+ if (builtin_normalizer) {
+ return build_builtin_normalizer(normalized_name);
}
throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
name: " + name);
}
- auto policy_it = _policys.find(name_it->second);
- if (policy_it == _policys.end()) {
- throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
id: " + name);
- }
- if (policy_it->second.type == TIndexPolicyType::ANALYZER) {
- return build_analyzer_provider_from_config(
- build_analyzer_config_from_policy(policy_it->second),
{})
+ if (index_policy->type == TIndexPolicyType::ANALYZER) {
+ return
build_analyzer_provider_from_config(build_analyzer_config_from_policy(*index_policy),
+ {})
->get_analyzer();
}
- if (policy_it->second.type == TIndexPolicyType::NORMALIZER) {
- return build_normalizer_from_policy(policy_it->second);
+ if (index_policy->type == TIndexPolicyType::NORMALIZER) {
+ return build_normalizer_from_policy(*index_policy);
}
throw Exception(ErrorCode::INVALID_ARGUMENT, "Analyzer policy not found: "
+ name);
}
AnalyzerProviderPtr IndexPolicyMgr::get_analyzer_provider_by_name(
- const std::string& name, const std::map<std::string, std::string>&
outer_char_filter_map) {
+ const std::string& name, const std::map<std::string, std::string>&
outer_char_filter_map,
+ std::string* resolved_name, std::string* legacy_name) {
std::shared_lock lock(_mutex);
+ if (resolved_name != nullptr) {
+ *resolved_name = name;
+ }
+ if (legacy_name != nullptr) {
+ legacy_name->clear();
+ }
const std::string normalized_name = normalize_name(name);
- auto name_it = _name_to_id.find(normalized_name);
- if (name_it == _name_to_id.end()) {
- if (is_builtin_normalizer(normalized_name)) {
- return
std::make_shared<SingleAnalyzerProvider>(build_builtin_normalizer(name));
+ bool builtin_normalizer = false;
+ const auto* index_policy = find_top_level_policy_locked(name,
&builtin_normalizer);
+ if (index_policy == nullptr) {
+ if (builtin_normalizer) {
+ return std::make_shared<SingleAnalyzerProvider>(
+ build_builtin_normalizer(normalized_name));
}
throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
name: " + name);
}
- auto policy_it = _policys.find(name_it->second);
- if (policy_it == _policys.end()) {
- throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
id: " + name);
+ if (resolved_name != nullptr) {
Review Comment:
[P1] Do not publish a builtin normalizer name as a compatibility alias for
an exact custom policy. For a replayed exact `LOWERCASE` normalizer, normalized
lookup of `lowercase` resolves to that same policy, so this sets the alias even
though `lowercase` is reserved for the builtin normalizer. If no exact
`LOWERCASE` reader exists, selection can then use a builtin-`lowercase` index
while tokenizing the query with the custom pipeline. Exclude builtin
normalizers here too and cover the exact-custom/query versus
canonical-builtin-reader case.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -226,26 +858,381 @@ private static String resolveTokenFilterIdentity(String
filterList) {
* IMPORTANT: Order is preserved because filter order is semantically
significant.
*/
private static String resolveCharFilterIdentity(String filterList) {
+ return resolveCharFilterIdentity(filterList, null);
+ }
+
+ private static String resolveCharFilterIdentity(String filterList,
FoldContext downstreamFold) {
+ ArrayDeque<String> identities = new ArrayDeque<>();
+ walkCharFilters(filterList, downstreamFold, identities);
+ return String.join(",", identities);
+ }
+
+ /**
+ * Resolve the chain from its last filter to its first, collecting
identities, and return the
+ * case-folding context that a filter placed in front of the chain would
run in.
+ */
+ private static FoldContext walkCharFilters(
+ String filterList, FoldContext downstreamFold, Deque<String>
identities) {
+ FoldContext fold = downstreamFold;
if (Strings.isNullOrEmpty(filterList)) {
- return "";
+ return fold;
}
- StringBuilder sb = new StringBuilder();
String[] filters = filterList.split(",\\s*");
// DO NOT sort - filter order is semantically significant
- for (int i = 0; i < filters.length; i++) {
- String filter = filters[i].trim();
- if (i > 0) {
- sb.append(",");
+ for (int i = filters.length - 1; i >= 0; --i) {
+ String filterName = filters[i].trim();
+ String filter = resolveComponentIdentity(filterName,
IndexPolicyTypeEnum.CHAR_FILTER, fold);
+ if (Strings.isNullOrEmpty(filter)) {
+ continue;
}
+ identities.addFirst(filter);
+ fold = foldContextBefore(filterName, fold);
+ }
+ return fold;
+ }
- if (IndexPolicy.BUILTIN_CHAR_FILTERS.contains(filter)) {
- sb.append(filter);
- } else {
- sb.append(resolveComponentIdentity(filter,
IndexPolicyTypeEnum.CHAR_FILTER));
+ /**
+ * Context for the filter that runs before this one: a case fold starts a
fresh context, a
+ * char_replace filter adds the bytes it rewrites, and any other filter
ends the context.
+ */
+ private static FoldContext foldContextBefore(String filterName,
FoldContext fold) {
+ FoldContext caseFold = caseFoldingCharFilterContext(filterName);
+ if (caseFold != null) {
+ return caseFold;
+ }
+ if (fold == null) {
+ return null;
+ }
+ boolean[] sourceBytes = charReplaceSourceBytes(filterName);
+ if (sourceBytes == null) {
+ return null;
+ }
+ fold.block(sourceBytes);
+ return fold;
+ }
+
+ /** Bytes a named char_replace filter rewrites, or null for any other
filter. */
+ private static boolean[] charReplaceSourceBytes(String filterName) {
+ IndexPolicy policy = findPolicy(filterName,
IndexPolicyTypeEnum.CHAR_FILTER);
+ if (policy == null || policy.isInvalid() || policy.getProperties() ==
null) {
+ return null;
+ }
+ Map<String, String> properties = policy.getProperties();
+ String type = normalizeBuiltinComponentName(
+ properties.get(IndexPolicy.PROP_TYPE),
IndexPolicyTypeEnum.CHAR_FILTER);
+ String pattern = properties.get("pattern");
+ if (!"char_replace".equals(type) || pattern == null) {
+ return null;
+ }
+ // Replacing the single replacement byte with itself leaves the stream
unchanged.
+ String replacement = properties.getOrDefault("replacement", " ");
+ int replacementByte = replacement.length() == 1 &&
replacement.charAt(0) < 128 ? replacement.charAt(0) : -1;
+ boolean[] sourceBytes = new boolean[256];
+ for (int i = 0; i < pattern.length(); ++i) {
+ char patternByte = pattern.charAt(i);
+ if (patternByte < sourceBytes.length && patternByte !=
replacementByte) {
+ sourceBytes[patternByte] = true;
}
}
- return sb.toString();
+ return sourceBytes;
+ }
+
+ /** The named policy when one exists with the expected type, or null. */
+ private static IndexPolicy findPolicy(String name, IndexPolicyTypeEnum
expectedType) {
+ if (Strings.isNullOrEmpty(name)) {
+ return null;
+ }
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ IndexPolicy policy =
env.getIndexPolicyMgr().getPolicyByName(name);
+ if (policy != null && policy.getType() == expectedType) {
+ return policy;
+ }
+ }
+ } catch (RuntimeException e) {
+ // Treat lookup failures as an unknown policy.
+ }
+ return null;
+ }
+
+ /** Fold context started by a named or built-in case-folding char filter,
or null for any other filter. */
+ private static FoldContext caseFoldingCharFilterContext(String name) {
+ if (Strings.isNullOrEmpty(name)) {
+ return null;
+ }
+
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ IndexPolicy policy =
env.getIndexPolicyMgr().getPolicyByName(name);
+ if (policy != null && policy.getType() ==
IndexPolicyTypeEnum.CHAR_FILTER) {
+ if (policy.isInvalid()) {
+ return null;
+ }
+ Map<String, String> properties = policy.getProperties();
+ if (properties != null && !properties.isEmpty()) {
+ String type = normalizeBuiltinComponentName(
+ properties.get(IndexPolicy.PROP_TYPE),
IndexPolicyTypeEnum.CHAR_FILTER);
+ return "icu_normalizer".equals(type) ?
icuNormalizerFoldContext(properties) : null;
+ }
+ }
+ }
+ } catch (RuntimeException e) {
+ // Fall through to built-in resolution.
+ }
+
+ return "icu_normalizer".equals(normalizeBuiltinComponentName(name,
IndexPolicyTypeEnum.CHAR_FILTER))
+ ? FoldContext.unfiltered() : null;
+ }
+
+ /**
+ * Fold context of an icu_normalizer component: the default nfkc_cf form
folds case over every
+ * code point, or only inside a parsable non-empty unicode_set_filter.
Null for other forms.
+ */
+ private static FoldContext icuNormalizerFoldContext(Map<String, String>
properties) {
+ if (!"nfkc_cf".equals(icuNormalizerName(properties))) {
+ return null;
+ }
+ String filter = properties.get("unicode_set_filter");
+ if (filter == null || filter.isEmpty()) {
+ return FoldContext.unfiltered();
+ }
+ try {
+ UnicodeSet unicodeSet = new UnicodeSet(filter);
+ return unicodeSet.isEmpty() ? FoldContext.unfiltered() : new
FoldContext(unicodeSet.freeze());
+ } catch (IllegalArgumentException e) {
+ return null;
+ }
+ }
+
+ /** Whether an icu_normalizer component leaves ASCII letters as they are.
*/
+ private static boolean isAsciiCaseTransparentIcuNormalizer(Map<String,
String> properties) {
+ String name = icuNormalizerName(properties);
+ return "nfc".equals(name) || "nfd".equals(name) || "nfkc".equals(name)
|| "nfkd".equals(name);
+ }
+
+ private static String icuNormalizerName(Map<String, String> properties) {
+ return properties.getOrDefault("name",
"nfkc_cf").trim().toLowerCase(Locale.ROOT);
+ }
+
+ /** The outer char filter runs before everything else, so it takes the
analyzer's fold context. */
+ private static String appendOuterCharFilterIdentity(
+ String analyzerIdentity, Map<String, String> properties,
FoldContext fold) {
+ String type =
properties.get(InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE);
+ String pattern =
properties.get(InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN);
+ if (!"char_replace".equals(type) || Strings.isNullOrEmpty(pattern)) {
+ return analyzerIdentity;
+ }
+ String replacement = properties.getOrDefault(
+
InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT, " ");
+ String canonicalPattern = canonicalizeCharReplacePattern(pattern,
replacement, fold);
+ if (canonicalPattern.isEmpty()) {
+ return analyzerIdentity;
+ }
+ return analyzerIdentity + "|outer_char_filter=char_replace:"
+ + canonicalPattern.length() + ":" + canonicalPattern + ":"
+ + replacement.length() + ":" + replacement + ";";
+ }
+
+ /**
+ * Canonicalize the ASCII pattern to the BE filter's byte set.
+ * Order, duplicate bytes, and replacements of a byte with itself do not
change the stream.
+ */
+ private static String canonicalizeCharReplacePattern(
+ String pattern, String replacement, FoldContext fold) {
+ if (replacement.length() != 1) {
+ return pattern;
+ }
+ char replacementByte = replacement.charAt(0);
Review Comment:
[P1] Canonicalize lower-to-upper replacements that a proven downstream fold
immediately erases. `a->A` before an unfiltered lowercase token filter emits
exactly the same bytes and offsets as no char filter, but this branch only
removes the opposite `A->a` form because it requires a lowercase replacement.
Two named `keyword -> lowercase` aliases can therefore get different identities
and pass CREATE/ALTER duplicate checks. Compare the downstream folded source
and replacement bytes in both directions, and add the reverse-case DDL coverage.
##########
be/src/storage/index/inverted/token_filter/pinyin_filter.cpp:
##########
@@ -214,29 +245,94 @@ bool PinyinFilter::readTerm(Token* token) {
return false;
}
-bool PinyinFilter::processCurrentToken() {
- processed_candidate_ = true;
+bool PinyinFilter::prepareCurrentSource(std::vector<UChar32>&
source_codepoints) {
+ size_t source_start = 0;
+ size_t source_end = current_token_text_.size();
+ if (config_->trimWhitespace) {
+ source_start = current_token_text_.find_first_not_of(" \t\n\r");
+ if (source_start == std::string::npos) {
+ return false;
+ }
+ source_end = current_token_text_.find_last_not_of(" \t\n\r") + 1;
+ }
+ current_source_ = current_token_text_.substr(source_start, source_end -
source_start);
- if (!has_current_token_) {
+ if (current_source_.empty()) {
return false;
}
- current_source_ = current_token_text_;
+ if (config_->ignorePinyinOffset) {
+ convertToCodepoints(current_source_, source_codepoints);
+ return !source_codepoints.empty();
+ }
- // Apply trimming if configured
- if (config_->trimWhitespace) {
- current_source_ = trim(current_source_);
+ current_runes_ = convertToRunes(current_source_, source_codepoints);
+
+ std::vector<RuneInfo> original_runes;
+ if (!current_source_byte_offsets_.empty()) {
+ std::vector<UChar32> original_codepoints;
+ original_runes = convertToRunes(current_token_text_,
original_codepoints);
+ if (current_source_byte_offsets_.size() != original_runes.size() + 1) {
+ current_source_byte_offsets_.clear();
+ current_source_byte_end_offsets_.clear();
+ }
+ }
+ if (!current_source_byte_offsets_.empty()) {
+ DORIS_CHECK(current_source_byte_end_offsets_.empty() ||
+ current_source_byte_end_offsets_.size() ==
original_runes.size());
+ const auto start_rune = std::ranges::lower_bound(
+ original_runes, static_cast<int32_t>(source_start), {},
&RuneInfo::byte_start);
+ const auto end_rune = std::ranges::lower_bound(
+ original_runes, static_cast<int32_t>(source_end), {},
&RuneInfo::byte_start);
+ const auto start_index = static_cast<size_t>(start_rune -
original_runes.begin());
+ const auto end_index = static_cast<size_t>(end_rune -
original_runes.begin());
+ DORIS_CHECK_EQ(end_index - start_index, current_runes_.size());
+ const int32_t token_start_offset = current_start_offset_;
+ current_start_offset_ += current_source_byte_offsets_[start_index];
+ current_end_offset_ =
+ token_start_offset + (current_source_byte_end_offsets_.empty()
+ ?
current_source_byte_offsets_[end_index]
+ :
current_source_byte_end_offsets_[end_index - 1]);
+ for (size_t i = 0; i < current_runes_.size(); ++i) {
+ current_runes_[i].byte_start =
current_source_byte_offsets_[start_index + i] -
+
current_source_byte_offsets_[start_index];
+ current_runes_[i].byte_end =
+ (current_source_byte_end_offsets_.empty()
+ ? current_source_byte_offsets_[start_index + i +
1]
+ : current_source_byte_end_offsets_[start_index +
i]) -
+ current_source_byte_offsets_[start_index];
+ }
+ } else if (has_current_conservative_source_span_) {
+ DORIS_CHECK_GE(current_conservative_source_start_, 0);
+ DORIS_CHECK_GE(current_conservative_source_end_,
current_conservative_source_start_);
+ const int32_t token_start_offset = current_start_offset_;
+ current_start_offset_ = token_start_offset +
current_conservative_source_start_;
+ current_end_offset_ = token_start_offset +
current_conservative_source_end_;
+ const int32_t source_length =
+ current_conservative_source_end_ -
current_conservative_source_start_;
+ for (auto& rune : current_runes_) {
+ rune.byte_start = 0;
+ rune.byte_end = source_length;
+ }
+ } else {
+ current_start_offset_ += static_cast<int32_t>(source_start);
+ current_end_offset_ =
+ current_start_offset_ + static_cast<int32_t>(source_end -
source_start);
}
- if (current_source_.empty()) {
+ return !source_codepoints.empty();
+}
+
+bool PinyinFilter::processCurrentToken() {
+ processed_candidate_ = true;
+
+ if (!has_current_token_) {
return false;
}
- // Convert to Unicode codepoints for processing
+ // Convert to Unicode codepoints for processing.
std::vector<UChar32> source_codepoints;
- convertToRunes(current_source_, source_codepoints);
-
- if (source_codepoints.empty()) {
+ if (!prepareCurrentSource(source_codepoints)) {
return false;
}
Review Comment:
[P1] Avoid the new per-rune index vector when Pinyin offsets are ignored.
The default configuration still pushes one `int` per ASCII alphanumeric rune
here, but `setTokenAttributes()` uses candidate subranges only under
`!ignorePinyinOffset`; otherwise every candidate gets the whole input span.
With the supported `empty -> pinyin` chain, a 100 MiB ASCII token therefore
adds about 400 MiB of pointless peak index storage. Keep the exact index vector
only for offset-aware mode and cover a large default-mode token while
processing is live.
##########
be/src/storage/index/inverted/token_filter/lower_case_filter.h:
##########
@@ -131,16 +166,40 @@ class LowerCaseFilter : public DorisTokenFilter {
static_cast<int32_t>(status), u_errorName(status));
}
Review Comment:
[P2] Gate these provenance-only rune counts on
`_source_byte_offsets_enabled`. Ordinary analyzer pipelines leave that flag
false, so upstream maps are empty and `_rune_count_changed` cannot affect any
consumer, yet each transformed non-ASCII token now gets two additional full
UTF-8 scans. `ASCIIFoldingFilter` has the same unconditional input/output
counts. Skip both paths when provenance is disabled and add a seam showing
disabled mode does no counting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
