airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4073746459
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java:
##########
@@ -337,23 +339,48 @@ private static void
checkInvertedIndexProperties(Map<String, String> properties,
// dict_compression now silently ignores by V2/V3 inverted index
}
- // Normalize analyzer and normalizer names to lowercase for
case-insensitive matching
+ // Canonicalize built-ins while retaining the exact spelling of a
resolved legacy policy.
normalizeInvertedIndexProperties(properties);
}
/**
- * Normalize analyzer and normalizer names in index properties to
lowercase.
- * This ensures case-insensitive matching between table creation and query
time.
+ * Canonicalize analyzer and normalizer names in index properties. Legacy
metadata may contain
+ * case-distinct policy names, so a resolved custom policy must keep its
exact stored name.
*/
private static void normalizeInvertedIndexProperties(Map<String, String>
properties) {
+ resolvePolicyNames(properties);
AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(
properties,
- INVERTED_INDEX_ANALYZER_NAME_KEY,
- INVERTED_INDEX_NORMALIZER_NAME_KEY,
INVERTED_INDEX_PARSER_KEY,
INVERTED_INDEX_PARSER_KEY_ALIAS);
}
+ /** Store analyzer and normalizer names in the spelling BE dispatches on.
*/
+ public static void resolvePolicyNames(Map<String, String> properties) {
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_ANALYZER_NAME_KEY);
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_NORMALIZER_NAME_KEY);
+ }
+
+ private static void normalizeResolvedPolicyName(Map<String, String>
properties, String key) {
+ String name = properties.get(key);
+ if (name == null || name.isEmpty()) {
+ return;
+ }
+ properties.put(key, resolveAnalyzerName(name));
+ }
+
+ /** Resolve built-in names and retain the stored spelling of custom
policies. */
+ public static String resolveAnalyzerName(String name) {
+ String trimmedName = name.trim();
+ // Match the BE writer's case-sensitive built-in dispatch before
policy lookup.
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(trimmedName)
+ || IndexPolicy.BUILTIN_NORMALIZERS.contains(trimmedName)) {
+ return trimmedName;
Review Comment:
Fixed in b0924c2b7f4. Reproduced first: with only a replayed `LOWERCASE`
policy, `normalizer=LowerCase` passed validation as the built-in but was stored
(and resolved for MATCH) as `LOWERCASE`; `analyzer=Ik` with a normalized legacy
`IK` policy was stored as `IK` the same way.
`IndexPolicyMgr` now has one top-level resolver matching BE's order
(canonical built-in analyzer, then exact policy, then built-in by normalized
name; normalizers skip the first step), and
`validateAnalyzerExists`/`validateNormalizerExists`, the stored-name
normalization and the MATCH lookup all use it, so the stored spelling always
binds what validation accepted. The analyzer key only considers built-in
analyzers and the normalizer key only built-in normalizers.
`testMixedCaseBuiltinSpellingsStoreBuiltinDespiteNormalizedLegacyPolicies` and
`testMixedCaseBuiltinNormalizerIgnoresNormalizedLegacyPolicyInDdl` cover both
kinds with replayed collisions.
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java:
##########
@@ -508,11 +602,10 @@ public void dropIndexPolicy(boolean isIfExists, String
indexPolicyName,
* tables, and indexes. In large-scale clusters with many tables, this can
be slow.
* Consider maintaining a reverse index (analyzer -> tables) if this
becomes a bottleneck.
*
- * @param analyzerName the analyzer name to check
+ * @param analyzer the analyzer policy to check
Review Comment:
Fixed in b0924c2b7f4. Reproduced first: with a replayed policy `IK` and an
index stored as canonical `analyzer=ik`, `DROP ANALYZER IK` failed with "the
analyzer IK is used by index: idx_1" although the index binds the built-in.
The index dependency check now goes through `indexBindsPolicyLocked`, which
applies the same top-level resolver: a name that resolves to a built-in never
counts as a reference to a policy. Nested references between policies keep
their existing lookup. `testDropDependencyFollowsTopLevelBuiltinPrecedence`
covers `IK` versus built-in `ik` and `LOWERCASE` versus built-in `lowercase`,
and keeps DROP blocked when the index stores the exact `IK` name or an exact
legacy `lowercase` policy exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
