airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4093595271
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -116,6 +319,30 @@ private static String resolveAnalyzerIdentity(String
analyzerName, String defaul
}
}
+ /** Whether BE builds the built-in normalizer for this name; an exact
legacy policy shadows it. */
+ private static boolean isBuiltinNormalizerBinding(String name) {
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ return env.getIndexPolicyMgr().getTopLevelBuiltin(
+ name, IndexPolicy.BUILTIN_NORMALIZERS) != null;
+ }
+ } catch (RuntimeException e) {
+ // Fall through to the name-only answer.
+ }
+ return IndexPolicy.BUILTIN_NORMALIZERS.contains(
+ Strings.nullToEmpty(name).trim().toLowerCase(Locale.ROOT));
+ }
+
+ /**
+ * BE builds a built-in normalizer as the keyword tokenizer plus the
built-in token filter of
+ * the canonical name, so it shares the identity of that custom pipeline.
+ */
+ private static String builtinNormalizerIdentity(String name) {
+ return
buildIdentityFromPolicyProperties(IndexPolicyTypeEnum.NORMALIZER,
+ Map.of(IndexPolicy.PROP_TOKEN_FILTER,
name.trim().toLowerCase(Locale.ROOT)));
Review Comment:
We verified the mechanism, and we are treating its trigger as out of scope
for this PR, for the same reason as
https://github.com/apache/doris/pull/67918#discussion_r4079903845.
What is true: `builtinNormalizerIdentity()` builds `keyword +
token_filter=lowercase` through `resolveComponentIdentity()`, which lets an
existing policy take precedence over the built-in (`getPolicyByName()`: exact
spelling first, then the normalized name). BE's
`build_builtin_normalizer("lowercase")` constructs the built-in lowercase
filter directly and never consults policies. So a replayed TOKEN_FILTER policy
whose normalized name is `lowercase` (your `LOWERCASE` = `asciifolding`) would
give `normalizer=lowercase` an ASCII-folding identity on FE while BE runs
lowercase, and a named ASCII-folding normalizer on the same column would then
be refused as a duplicate.
What it takes: such a policy cannot be created on this base. Since #59117
(`4f83889c83a`, 2026-02-02) `createIndexPolicy()` normalizes the name before
comparing it with `BUILTIN_TOKEN_FILTERS`, so both `lowercase` and `LOWERCASE`
fail with `conflicts with built-in token filter name`; before #59117 that
comparison was case-sensitive (`BUILTIN_TOKEN_FILTERS.contains(policyName)`),
which is the only window in which a `LOWERCASE` token filter could have been
persisted. The state therefore needs a cluster that, before #59117, created a
token filter named like a built-in one in a different case, gave it another
type, still replays it today, and then defines a named ASCII-folding normalizer
next to the built-in one on the same column. As in the linked thread, we are
not carrying compatibility for pre-#59117 case-colliding policy names in this
PR; the user-facing effect here would be a rejected CREATE/ALTER with the
duplicate-index message, not wrong query results.
If that legacy state is ever brought into scope, the change is contained:
encode the built-in normalizer identity literally as keyword plus the built-in
lowercase filter instead of resolving the component through policies, with the
CREATE/ALTER case you describe. We are leaving it out here and resolving this
thread; if you see a way to reach this state on a current version, we will look
again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
