github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4000703276
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -152,35 +153,49 @@ private static String resolveComponentIdentity(String
name, IndexPolicyTypeEnum
return "";
}
- // Check if it's a built-in component
- if (expectedType == IndexPolicyTypeEnum.TOKENIZER
- && IndexPolicy.BUILTIN_TOKENIZERS.contains(name)) {
- return name;
- }
-
- // For custom component, get its properties
+ // Existing named policies take precedence over built-ins for upgrade
compatibility.
try {
Env env = Env.getCurrentEnv();
- if (env == null || env.getIndexPolicyMgr() == null) {
- return name;
- }
-
- IndexPolicy policy = env.getIndexPolicyMgr().getPolicyByName(name);
- if (policy == null || policy.getType() != expectedType) {
- return name;
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ IndexPolicy policy =
env.getIndexPolicyMgr().getPolicyByName(name);
+ if (policy != null && policy.getType() == expectedType) {
+ Map<String, String> props = policy.getProperties();
+ if (props != null && !props.isEmpty()) {
+ TreeMap<String, String> sortedProps = new
TreeMap<>(props);
+ String type = sortedProps.get(IndexPolicy.PROP_TYPE);
+ String normalizedType =
normalizeBuiltinComponentName(type, expectedType);
+ if (normalizedType != null) {
+ if (sortedProps.size() == 1) {
+ return normalizedType;
+ }
+ sortedProps.put(IndexPolicy.PROP_TYPE,
normalizedType);
+ }
+ return sortedProps.toString();
+ }
+ }
}
+ } catch (RuntimeException e) {
+ // Fall through to built-in resolution or the original name.
+ }
- Map<String, String> props = policy.getProperties();
- if (props == null || props.isEmpty()) {
- return name;
- }
+ String normalizedName = normalizeBuiltinComponentName(name,
expectedType);
+ return normalizedName == null ? name : normalizedName;
Review Comment:
[P1] Unify legacy and custom IK identities
This branch canonicalizes tokenizer identities only inside a custom
analyzer. A bare analyzer using `ik_smart` resolves to
`ANALYZER:tokenizer=ik_smart;`, while the equivalent legacy `parser=ik,
parser_mode=ik_smart` returns `ik` above; BE builds the same IK mode,
dictionary, lowercase behavior, and unfiltered token stream for both.
Consequently both CREATE TABLE and ALTER duplicate checks accept two equivalent
inverted indexes on one column. The same applies to `ik_max_word`, and an
omitted legacy mode defaults to smart. Please canonicalize these cross-family
representations to one semantic identity and cover both duplicate paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
