airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4073756073
##########
be/src/storage/index/inverted/token_filter/lower_case_filter.h:
##########
@@ -131,16 +166,40 @@ class LowerCaseFilter : public DorisTokenFilter {
static_cast<int32_t>(status), u_errorName(status));
}
Review Comment:
Fixed in 787e0ab4f3f. Reproduced first: with provenance disabled,
`LowerCaseFilter` on U+0130 and `ASCIIFoldingFilter` on U+00C6 still computed
both rune counts and set `_rune_count_changed`.
Both filters now count runes only when `_source_byte_offsets_enabled` is
set; with it disabled the upstream maps are empty anyway, so the delegated
getters return the same empty result.
`TestCaseAndFoldingFiltersCountRunesOnlyForOffsets` asserts the flag stays
false when disabled and is set when enabled, for both filters.
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java:
##########
@@ -93,14 +149,13 @@ public List<IndexPolicy> getCopiedIndexPolicies() {
public void validateAnalyzerExists(String analyzerName) throws
DdlException {
String normalizedName = normalizeKey(analyzerName);
- // Built-in analyzers are stored in lowercase, so use normalized name
for comparison
- if (IndexPolicy.BUILTIN_ANALYZERS.contains(normalizedName)) {
- return;
- }
-
readLock();
Review Comment:
Fixed in b0924c2b7f4. Reproduced first: with an exact legacy TOKENIZER named
`ik`, `analyzer=ik` failed validation with "Policy 'ik' is not an analyzer"
(and ALTER failed the same way), although BE's analyzer factory intercepts
canonical `ik` before any policy lookup.
`validateAnalyzerExists` now uses the shared top-level resolver, so a
canonical lower-case built-in analyzer name wins over an exact legacy policy,
while exact-first behavior is kept for non-canonical spellings and for
normalizers (an exact `English` tokenizer and an exact `lowercase` analyzer are
still rejected).
`testCanonicalBuiltinAnalyzerWinsValidationOverExactLegacyPolicy` and
`testCanonicalBuiltinAnalyzerIgnoresExactLegacyPolicyInDdl` cover CREATE/ALTER
with the replayed collision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
