airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4073748451
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -72,7 +177,7 @@ private static String resolveAnalyzerIdentity(String
analyzerName, String defaul
// Check if it's a built-in normalizer
if (IndexPolicy.BUILTIN_NORMALIZERS.contains(analyzerName)) {
Review Comment:
Fixed in b0924c2b7f4. Reproduced first: an exact replayed normalizer named
`lowercase` backed by `asciifolding` was given the built-in
keyword-plus-lowercase identity, so CREATE and ALTER compared it as the wrong
pipeline.
`resolveAnalyzerIdentity` and `customAnalyzerFoldContext` now check for an
exact policy of that name before applying the built-in normalizer identity
(only the exact spelling, matching BE's `find_top_level_policy_locked`), and
use that policy's pipeline when it exists. A `LOWERCASE` policy that only
matches after normalization still does not shadow the built-in.
`testExactLegacyLowercaseNormalizerShadowsBuiltinIdentity`,
`testCreateTableUsesExactLegacyLowercaseNormalizerIdentity` and
`testAddInvertedIndexUsesExactLegacyLowercaseNormalizerIdentity` cover
identity, CREATE and ALTER.
##########
be/src/storage/index/inverted/tokenizer/pinyin/pinyin_tokenizer.cpp:
##########
@@ -48,6 +48,9 @@
PinyinTokenizer::PinyinTokenizer(std::shared_ptr<doris::segment_v2::PinyinConfig
void PinyinTokenizer::reset() {
DorisTokenizer::reset();
+ has_current_span_ = false;
+ ascii_buff_rune_starts_.clear();
Review Comment:
Fixed in 787e0ab4f3f. Reproduced first: with default settings
(`ignore_pinyin_offset=true`) a 96 KiB ASCII value still left 262,144 entries
of capacity in the two per-letter vectors, and after an offset-aware large
value `reset()` kept 1 MiB.
`PinyinTokenizer` now records per-letter source ranges only when offsets are
tracked (with ignored offsets `next()` replaces every candidate range by the
whole input span, so letter positions stand in for them in `parseBuff`), and
`reset()` releases oversized capacity with the shared high-water helper.
`TestPinyinTokenizerCollectsSourceScratchOnlyForOffsets` covers the default
mode (no collection), the tracked mode, and a large -> small reset.
##########
be/src/runtime/index_policy/index_policy_mgr.cpp:
##########
@@ -142,50 +205,61 @@ AnalyzerPtr IndexPolicyMgr::get_policy_by_name(const
std::string& name) {
AnalyzerPtr IndexPolicyMgr::get_analyzer_by_name(const std::string& name) {
std::shared_lock lock(_mutex);
const std::string normalized_name = normalize_name(name);
- auto name_it = _name_to_id.find(normalized_name);
- if (name_it == _name_to_id.end()) {
- if (is_builtin_normalizer(normalized_name)) {
- return build_builtin_normalizer(name);
+ bool builtin_normalizer = false;
+ const auto* index_policy = find_top_level_policy_locked(name,
&builtin_normalizer);
+ if (index_policy == nullptr) {
+ if (builtin_normalizer) {
+ return build_builtin_normalizer(normalized_name);
}
throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
name: " + name);
}
- auto policy_it = _policys.find(name_it->second);
- if (policy_it == _policys.end()) {
- throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
id: " + name);
- }
- if (policy_it->second.type == TIndexPolicyType::ANALYZER) {
- return build_analyzer_provider_from_config(
- build_analyzer_config_from_policy(policy_it->second),
{})
+ if (index_policy->type == TIndexPolicyType::ANALYZER) {
+ return
build_analyzer_provider_from_config(build_analyzer_config_from_policy(*index_policy),
+ {})
->get_analyzer();
}
- if (policy_it->second.type == TIndexPolicyType::NORMALIZER) {
- return build_normalizer_from_policy(policy_it->second);
+ if (index_policy->type == TIndexPolicyType::NORMALIZER) {
+ return build_normalizer_from_policy(*index_policy);
}
throw Exception(ErrorCode::INVALID_ARGUMENT, "Analyzer policy not found: "
+ name);
}
AnalyzerProviderPtr IndexPolicyMgr::get_analyzer_provider_by_name(
- const std::string& name, const std::map<std::string, std::string>&
outer_char_filter_map) {
+ const std::string& name, const std::map<std::string, std::string>&
outer_char_filter_map,
+ std::string* resolved_name, std::string* legacy_name) {
std::shared_lock lock(_mutex);
+ if (resolved_name != nullptr) {
+ *resolved_name = name;
+ }
+ if (legacy_name != nullptr) {
+ legacy_name->clear();
+ }
const std::string normalized_name = normalize_name(name);
- auto name_it = _name_to_id.find(normalized_name);
- if (name_it == _name_to_id.end()) {
- if (is_builtin_normalizer(normalized_name)) {
- return
std::make_shared<SingleAnalyzerProvider>(build_builtin_normalizer(name));
+ bool builtin_normalizer = false;
+ const auto* index_policy = find_top_level_policy_locked(name,
&builtin_normalizer);
+ if (index_policy == nullptr) {
+ if (builtin_normalizer) {
+ return std::make_shared<SingleAnalyzerProvider>(
+ build_builtin_normalizer(normalized_name));
}
throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
name: " + name);
}
- auto policy_it = _policys.find(name_it->second);
- if (policy_it == _policys.end()) {
- throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with
id: " + name);
+ if (resolved_name != nullptr) {
Review Comment:
Fixed in 787e0ab4f3f. Reproduced first: with a replayed exact `LOWERCASE`
normalizer, `get_analyzer_provider_by_name("LOWERCASE", …)` returned
`legacy_name = "lowercase"`, i.e. it advertised the built-in normalizer's name
as a compatibility alias of the custom policy.
The alias is now withheld for built-in normalizer names as well as built-in
analyzer names, so reader selection can never pair a canonical built-in
`lowercase` index with the custom pipeline.
`ExactCustomNormalizerDoesNotPublishBuiltinAlias` covers the exact-custom
collision (resolved name stays `LOWERCASE`, alias empty).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
