airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4075516650
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -116,6 +220,28 @@ private static String resolveAnalyzerIdentity(String
analyzerName, String defaul
}
}
+ /** Whether BE builds the built-in normalizer for this name; an exact
legacy policy shadows it. */
+ private static boolean isBuiltinNormalizerBinding(String name) {
+ if (!IndexPolicy.BUILTIN_NORMALIZERS.contains(name)) {
+ return false;
+ }
+ try {
+ Env env = Env.getCurrentEnv();
+ return env == null || env.getIndexPolicyMgr() == null
+ || env.getIndexPolicyMgr().getPolicyByExactName(name) ==
null;
+ } catch (RuntimeException e) {
+ return true;
+ }
+ }
+
+ /**
+ * BE builds a built-in normalizer as the keyword tokenizer plus the
built-in token filter of
+ * the same name, so it shares the identity of that custom pipeline.
+ */
+ private static String builtinNormalizerIdentity(String name) {
+ return IndexPolicyTypeEnum.NORMALIZER.name() + ":" +
IndexPolicy.PROP_TOKEN_FILTER + "=" + name + ";";
Review Comment:
Confirmed and fixed in 79715d0b1b2.
A NORMALIZER identity is now built as the equivalent keyword analyzer: the
identity type becomes ANALYZER and `tokenizer=keyword` is injected, so
`NORMALIZER(token_filter=lowercase)` and `ANALYZER(tokenizer=keyword,
token_filter=lowercase)` collapse to one identity and both the CREATE and the
ALTER duplicate fence catch the pair. `builtinNormalizerIdentity()` goes
through the same path, so the built-in `lowercase` normalizer aliases with them
too.
The ASCII-fold context is unaffected by the rewrite: the NORMALIZER branch
of `foldsAsciiCaseAfterCharFilters()` and the case-transparent-tokenizer branch
taken for `keyword` both evaluate the same token-filter chain.
Test: `testNormalizerIdentityMatchesEquivalentKeywordAnalyzer`, with a
non-keyword tokenizer as the negative.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -150,47 +282,562 @@ private static String
buildIdentityFromPolicyProperties(IndexPolicyTypeEnum type
* Resolve a component (tokenizer) to its identity.
*/
private static String resolveComponentIdentity(String name,
IndexPolicyTypeEnum expectedType) {
+ return resolveComponentIdentity(name, expectedType, null);
+ }
+
+ /** {@code fold} is the case-folding context of a char filter, or null
without a downstream fold. */
+ private static String resolveComponentIdentity(
+ String name, IndexPolicyTypeEnum expectedType, FoldContext fold) {
if (Strings.isNullOrEmpty(name)) {
return "";
}
- // Check if it's a built-in component
- if (expectedType == IndexPolicyTypeEnum.TOKENIZER
- && IndexPolicy.BUILTIN_TOKENIZERS.contains(name)) {
- return name;
+ // Existing named policies take precedence over built-ins for upgrade
compatibility.
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ IndexPolicy policy =
env.getIndexPolicyMgr().getPolicyByName(name);
+ if (policy != null && policy.getType() == expectedType) {
+ if (policy.isInvalid()) {
+ return "invalid-policy:" + policy.getId() + ":" +
policy.getName();
+ }
+ Map<String, String> props = policy.getProperties();
+ if (props != null && !props.isEmpty()) {
+ TreeMap<String, String> sortedProps = new
TreeMap<>(props);
+ String type = sortedProps.get(IndexPolicy.PROP_TYPE);
+ String normalizedType =
normalizeBuiltinComponentName(type, expectedType);
+ if (normalizedType != null) {
+ if ("empty".equals(normalizedType)) {
+ return "";
+ }
+ sortedProps.put(IndexPolicy.PROP_TYPE,
normalizedType);
+ canonicalizeEffectiveComponentProperties(
+ sortedProps, normalizedType, expectedType);
+ if (sortedProps.size() == 1) {
+ return normalizedType;
+ }
+ }
+ if (expectedType == IndexPolicyTypeEnum.TOKENIZER
+ &&
"ngram".equals(sortedProps.get(IndexPolicy.PROP_TYPE))) {
+ // This setting only limits policy creation; it
does not change emitted tokens.
+ sortedProps.remove(PROP_MAX_NGRAM_DIFF);
+ }
+ if (expectedType == IndexPolicyTypeEnum.CHAR_FILTER
Review Comment:
Confirmed and fixed in 79715d0b1b2.
When a `char_replace` policy's canonical pattern and replacement equal what
`CharReplaceCharFilterFactory` applies to a bare reference (`,._` and a single
space), both properties are now dropped, so the identity degenerates to the
bare `char_replace` identity. `charReplaceSourceBytes()` also answers for a
bare built-in reference now instead of returning null, so the fold context is
the same on both sides. A non-default pattern or replacement is still recorded
verbatim.
Test: `testExplicitCharReplaceDefaultsMatchBuiltinReference`, which keeps a
non-default pattern/replacement pair as the negative.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -206,17 +853,15 @@ private static String resolveTokenFilterIdentity(String
filterList) {
String[] filters = filterList.split(",\\s*");
// DO NOT sort - filter order is semantically significant
- for (int i = 0; i < filters.length; i++) {
- String filter = filters[i].trim();
- if (i > 0) {
- sb.append(",");
+ for (String filterName : filters) {
Review Comment:
Confirmed and fixed in 79715d0b1b2.
`resolveTokenFilterIdentity()` now skips a filter whose resolved identity
equals the immediately preceding one and is on an explicit idempotent list,
currently `{lowercase}`. Only adjacent repeats collapse and the order is never
changed, so `lowercase, asciifolding, lowercase` still keeps all three entries.
Test: `testAdjacentDuplicateLowercaseFiltersCollapse`, with a repeated
non-idempotent filter as the negative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
