github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4068510696


##########
be/src/storage/index/inverted/char_filter/icu_normalizer_char_filter.cpp:
##########
@@ -59,33 +63,97 @@ void ICUNormalizerCharFilter::fill() {
     input.resize(_reader->size());
     _reader->readCopy(input.data(), 0, static_cast<int32_t>(input.size()));
     normalize_text(input, _buf);
+    build_source_byte_offset_runs();
     _transformed_input.init(_buf.data(), static_cast<int32_t>(_buf.size()), 
false);
 }
 
 void ICUNormalizerCharFilter::normalize_text(const std::string& input, 
std::string& output) {
     output.clear();
+    _edits.reset();
     if (input.empty()) {
         return;
     }
 
     UErrorCode status = U_ZERO_ERROR;
-    icu::UnicodeString src16 = icu::UnicodeString::fromUTF8(input);
-    UNormalizationCheckResult quick_result = _normalizer->quickCheck(src16, 
status);
-    if (U_SUCCESS(status) && quick_result == UNORM_YES) {
+    icu::StringByteSink<std::string> sink(&output);
+    _normalizer->normalizeUTF8(0, icu::StringPiece(input), sink, &_edits, 
status);
+    if (U_FAILURE(status)) {
+        LOG(WARNING) << "ICU normalize failed: " << u_errorName(status) << ", 
using original text";
         output = input;
+        _edits.reset();
+        _edits.addUnchanged(static_cast<int32_t>(input.size()));
         return;
     }
+}
 
-    icu::UnicodeString result16;
-    status = U_ZERO_ERROR;
-    _normalizer->normalize(src16, result16, status);
+void ICUNormalizerCharFilter::build_source_byte_offset_runs() {
+    _offset_correction_runs.clear();
+    UErrorCode status = U_ZERO_ERROR;
+    auto iterator = _edits.getFineChangesIterator();
+    while (iterator.next(status)) {
+        if (U_FAILURE(status)) {
+            _offset_correction_runs.clear();
+            return;
+        }
+
+        const int32_t source_start = iterator.sourceIndex();
+        const int32_t destination_start = iterator.destinationIndex();
+        const int32_t source_length = iterator.oldLength();
+        const int32_t destination_length = iterator.newLength();
+        if (!_offset_correction_runs.empty()) {
+            auto& previous = _offset_correction_runs.back();
+            const int64_t previous_source_end =
+                    static_cast<int64_t>(previous.source_start) +
+                    static_cast<int64_t>(previous.source_length) * 
previous.repeat_count;
+            const int64_t previous_destination_end =
+                    static_cast<int64_t>(previous.destination_start) +
+                    static_cast<int64_t>(previous.destination_length) * 
previous.repeat_count;
+            if (previous.source_length == source_length &&
+                previous.destination_length == destination_length &&
+                previous_source_end == source_start &&
+                previous_destination_end == destination_start) {
+                ++previous.repeat_count;
+                continue;
+            }
+        }
+        _offset_correction_runs.push_back(

Review Comment:
   [P1] Bound correction storage for alternating ICU edits. This loop coalesces 
only adjacent changed spans. Under the default `nfkc_cf`, repeating ASCII `a` 
followed by full-width `A` leaves a one-byte unchanged gap between every 3-to-1 
edit, so every four input bytes append another 20-byte `OffsetCorrectionRun`. A 
100 MiB analyzed value therefore needs about 500 MiB for this vector alone, 
before capacity slack, input/output buffers, ICU `Edits`, and tokenizer state; 
analyzed values are not capped by `ignore_above`. The sparse test exercises 
only one edit, so it misses this path. Use a monotonic ICU iterator or another 
representation bounded under alternating edits, and add dense-change allocation 
coverage. This is distinct from the earlier dense byte table because the 
replacement run format still degenerates per separated change.



##########
be/src/runtime/index_policy/index_policy_mgr.cpp:
##########
@@ -142,50 +186,58 @@ AnalyzerPtr IndexPolicyMgr::get_policy_by_name(const 
std::string& name) {
 AnalyzerPtr IndexPolicyMgr::get_analyzer_by_name(const std::string& name) {
     std::shared_lock lock(_mutex);
     const std::string normalized_name = normalize_name(name);
-    auto name_it = _name_to_id.find(normalized_name);
-    if (name_it == _name_to_id.end()) {
+    const auto* index_policy = find_policy_by_name_locked(name);
+    if (index_policy == nullptr) {
         if (is_builtin_normalizer(normalized_name)) {
             return build_builtin_normalizer(name);
         }
         throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with 
name: " + name);
     }
-    auto policy_it = _policys.find(name_it->second);
-    if (policy_it == _policys.end()) {
-        throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with 
id: " + name);
-    }
-    if (policy_it->second.type == TIndexPolicyType::ANALYZER) {
-        return build_analyzer_provider_from_config(
-                       build_analyzer_config_from_policy(policy_it->second), 
{})
+    if (index_policy->type == TIndexPolicyType::ANALYZER) {
+        return 
build_analyzer_provider_from_config(build_analyzer_config_from_policy(*index_policy),
+                                                   {})
                 ->get_analyzer();
     }
-    if (policy_it->second.type == TIndexPolicyType::NORMALIZER) {
-        return build_normalizer_from_policy(policy_it->second);
+    if (index_policy->type == TIndexPolicyType::NORMALIZER) {
+        return build_normalizer_from_policy(*index_policy);
     }
     throw Exception(ErrorCode::INVALID_ARGUMENT, "Analyzer policy not found: " 
+ name);
 }
 
 AnalyzerProviderPtr IndexPolicyMgr::get_analyzer_provider_by_name(
-        const std::string& name, const std::map<std::string, std::string>& 
outer_char_filter_map) {
+        const std::string& name, const std::map<std::string, std::string>& 
outer_char_filter_map,
+        std::string* resolved_name, std::string* legacy_name) {
     std::shared_lock lock(_mutex);
+    if (resolved_name != nullptr) {
+        *resolved_name = name;
+    }
+    if (legacy_name != nullptr) {
+        legacy_name->clear();
+    }
     const std::string normalized_name = normalize_name(name);
-    auto name_it = _name_to_id.find(normalized_name);
-    if (name_it == _name_to_id.end()) {
+    const auto* index_policy = find_policy_by_name_locked(name);

Review Comment:
   [P1] Keep canonical built-in normalizer precedence consistent with FE. FE 
deliberately accepts lowercase `lowercase` as the built-in when replay has only 
an exact `LOWERCASE` policy, even if that legacy policy is another family; only 
an exact spelling takes policy precedence. Here `find_policy_by_name_locked()` 
performs normalized fallback first, so `normalizer=lowercase` binds the legacy 
policy and this path throws `Analyzer policy not found` instead of building the 
built-in. Resolve top-level names as exact policy -> canonical built-in 
normalizer -> normalized fallback (while retaining the intended nested 
compatibility behavior), and cover replayed `LOWERCASE` token-filter policy 
plus a lowercase-normalizer index/query.



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -150,49 +216,430 @@ private static String 
buildIdentityFromPolicyProperties(IndexPolicyTypeEnum type
      * Resolve a component (tokenizer) to its identity.
      */
     private static String resolveComponentIdentity(String name, 
IndexPolicyTypeEnum expectedType) {
+        return resolveComponentIdentity(name, expectedType, false);
+    }
+
+    private static String resolveComponentIdentity(
+            String name, IndexPolicyTypeEnum expectedType, boolean 
lowercaseDownstream) {
         if (Strings.isNullOrEmpty(name)) {
             return "";
         }
 
-        // Check if it's a built-in component
-        if (expectedType == IndexPolicyTypeEnum.TOKENIZER
-                && IndexPolicy.BUILTIN_TOKENIZERS.contains(name)) {
-            return name;
+        // Existing named policies take precedence over built-ins for upgrade 
compatibility.
+        try {
+            Env env = Env.getCurrentEnv();
+            if (env != null && env.getIndexPolicyMgr() != null) {
+                IndexPolicy policy = 
env.getIndexPolicyMgr().getPolicyByName(name);
+                if (policy != null && policy.getType() == expectedType) {
+                    if (policy.isInvalid()) {
+                        return "invalid-policy:" + policy.getId() + ":" + 
policy.getName();
+                    }
+                    Map<String, String> props = policy.getProperties();
+                    if (props != null && !props.isEmpty()) {
+                        TreeMap<String, String> sortedProps = new 
TreeMap<>(props);
+                        String type = sortedProps.get(IndexPolicy.PROP_TYPE);
+                        String normalizedType = 
normalizeBuiltinComponentName(type, expectedType);
+                        if (normalizedType != null) {
+                            if ("empty".equals(normalizedType)) {
+                                return "";
+                            }
+                            sortedProps.put(IndexPolicy.PROP_TYPE, 
normalizedType);
+                            canonicalizeEffectiveComponentProperties(
+                                    sortedProps, normalizedType, expectedType);
+                            if (sortedProps.size() == 1) {
+                                return normalizedType;
+                            }
+                        }
+                        if (expectedType == IndexPolicyTypeEnum.TOKENIZER
+                                && 
"ngram".equals(sortedProps.get(IndexPolicy.PROP_TYPE))) {
+                            // This setting only limits policy creation; it 
does not change emitted tokens.
+                            sortedProps.remove(PROP_MAX_NGRAM_DIFF);
+                        }
+                        if (expectedType == IndexPolicyTypeEnum.CHAR_FILTER
+                                && 
"char_replace".equals(sortedProps.get(IndexPolicy.PROP_TYPE))) {
+                            String replacement = 
sortedProps.getOrDefault("replacement", " ");
+                            String pattern = canonicalizeCharReplacePattern(
+                                    sortedProps.get("pattern"), replacement, 
lowercaseDownstream);
+                            if (pattern.isEmpty()) {
+                                return "";
+                            }
+                            sortedProps.put("pattern", pattern);
+                            sortedProps.put("replacement", replacement);
+                        }
+                        if (normalizedType != null && sortedProps.size() == 1) 
{
+                            return normalizedType;
+                        }
+                        return sortedProps.toString();
+                    }
+                }
+            }
+        } catch (RuntimeException e) {
+            // Fall through to built-in resolution or the original name.
+        }
+
+        String normalizedName = normalizeBuiltinComponentName(name, 
expectedType);
+        return "empty".equals(normalizedName) ? "" : normalizedName == null ? 
name : normalizedName;
+    }
+
+    private static void canonicalizeEffectiveComponentProperties(
+            TreeMap<String, String> properties, String type, 
IndexPolicyTypeEnum expectedType) {
+        if ("pinyin".equals(type)) {
+            removeBooleanDefaults(properties, true,
+                    "keep_first_letter", "keep_full_pinyin", 
"keep_none_chinese",
+                    "keep_none_chinese_together", 
"keep_none_chinese_in_first_letter",
+                    "lowercase", "trim_whitespace", "ignore_pinyin_offset",
+                    "none_chinese_pinyin_tokenize");
+            removeBooleanDefaults(properties, false,
+                    "keep_separate_first_letter", "keep_joined_full_pinyin", 
"keep_original",
+                    "keep_none_chinese_in_joined_full_pinyin", 
"remove_duplicated_term",
+                    "fixed_pinyin_offset", "keep_separate_chinese");
+            removeIntegerDefault(properties, "limit_first_letter_length", 16);
+            canonicalizePinyinDependencies(properties);
+            return;
         }
 
-        // For custom component, get its properties
+        if (expectedType == IndexPolicyTypeEnum.TOKEN_FILTER) {
+            if ("asciifolding".equals(type)) {
+                removeBooleanDefaults(properties, false, "preserve_original");
+            } else if ("word_delimiter".equals(type)) {
+                removeBooleanDefaults(properties, true, "generate_word_parts", 
"generate_number_parts",
+                        "split_on_case_change", "split_on_numerics", 
"stem_english_possessive");
+                removeBooleanDefaults(properties, false, "catenate_words", 
"catenate_numbers",
+                        "catenate_all", "preserve_original");
+                canonicalizeWordSet(properties, "protected_words");
+                canonicalizeTypeTable(properties);
+            } else if ("icu_normalizer".equals(type)) {
+                canonicalizeIcuNormalizerDefaults(properties, false);
+            }
+            return;
+        }
+
+        if (expectedType == IndexPolicyTypeEnum.CHAR_FILTER) {
+            if ("icu_normalizer".equals(type)) {
+                canonicalizeIcuNormalizerDefaults(properties, true);
+            }
+            return;
+        }
+
+        if (expectedType != IndexPolicyTypeEnum.TOKENIZER) {
+            return;
+        }
+        switch (type) {
+            case "ngram":
+            case "edge_ngram":
+                removeIntegerDefault(properties, "min_gram", 1);
+                removeIntegerDefault(properties, "max_gram", 2);
+                canonicalizeWordSet(properties, "token_chars");
+                canonicalizeCustomTokenChars(properties);
+                break;
+            case "standard":
+                removeIntegerDefault(properties, "max_token_length", 255);
+                break;
+            case "char_group":
+                removeIntegerDefault(properties, "max_token_length", 255);
+                canonicalizeTokenizeOnChars(properties);
+                break;
+            case "keyword":
+                removeIntegerDefault(properties, "buffer_size", 256);

Review Comment:
   [P1] Remove keyword `buffer_size` from the semantic identity. This branch 
keeps non-default values, but BE only range-checks and stores the setting: 
`KeywordTokenizer::reset()` and `next()` never read `_buffer_size`, and every 
valid value emits the same single UTF-8-safe prefix capped by constant 8192. 
Keyword policies using 256 and 512 therefore produce identical terms, 
positions, offsets, and provenance while differently named analyzer aliases can 
bypass both CREATE and ALTER duplicate rejection. Ignore every valid value here 
(or reject the ineffective setting), and add identity plus both DDL-path cases 
for 256 versus 512.



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -226,26 +671,115 @@ private static String resolveTokenFilterIdentity(String 
filterList) {
      * IMPORTANT: Order is preserved because filter order is semantically 
significant.
      */
     private static String resolveCharFilterIdentity(String filterList) {
+        return resolveCharFilterIdentity(filterList, false);
+    }
+
+    private static String resolveCharFilterIdentity(String filterList, boolean 
lowercaseDownstream) {
         if (Strings.isNullOrEmpty(filterList)) {
             return "";
         }
 
-        StringBuilder sb = new StringBuilder();
+        ArrayDeque<String> identities = new ArrayDeque<>();
         String[] filters = filterList.split(",\\s*");
         // DO NOT sort - filter order is semantically significant
 
-        for (int i = 0; i < filters.length; i++) {
-            String filter = filters[i].trim();
-            if (i > 0) {
-                sb.append(",");
+        for (int i = filters.length - 1; i >= 0; --i) {
+            String filterName = filters[i].trim();
+            String filter = resolveComponentIdentity(
+                    filterName, IndexPolicyTypeEnum.CHAR_FILTER, 
lowercaseDownstream);
+            if (Strings.isNullOrEmpty(filter)) {
+                continue;
             }
+            identities.addFirst(filter);
+            lowercaseDownstream = isCaseFoldingCharFilter(filterName);
+        }
+        return String.join(",", identities);
+    }
 
-            if (IndexPolicy.BUILTIN_CHAR_FILTERS.contains(filter)) {
-                sb.append(filter);
-            } else {
-                sb.append(resolveComponentIdentity(filter, 
IndexPolicyTypeEnum.CHAR_FILTER));
+    private static boolean isCaseFoldingCharFilter(String name) {
+        if (Strings.isNullOrEmpty(name)) {
+            return false;
+        }
+
+        try {
+            Env env = Env.getCurrentEnv();
+            if (env != null && env.getIndexPolicyMgr() != null) {
+                IndexPolicy policy = 
env.getIndexPolicyMgr().getPolicyByName(name);
+                if (policy != null && policy.getType() == 
IndexPolicyTypeEnum.CHAR_FILTER) {
+                    if (policy.isInvalid()) {
+                        return false;
+                    }
+                    Map<String, String> properties = policy.getProperties();
+                    if (properties != null && !properties.isEmpty()) {
+                        String type = normalizeBuiltinComponentName(
+                                properties.get(IndexPolicy.PROP_TYPE), 
IndexPolicyTypeEnum.CHAR_FILTER);
+                        String normalizer = properties.getOrDefault("name", 
"nfkc_cf").trim();
+                        String unicodeSet = 
properties.getOrDefault("unicode_set_filter", "").trim();
+                        return "icu_normalizer".equals(type)
+                                && "nfkc_cf".equalsIgnoreCase(normalizer)
+                                && unicodeSet.isEmpty();
+                    }
+                }
+            }
+        } catch (RuntimeException e) {
+            // Fall through to built-in resolution.
+        }
+
+        return "icu_normalizer".equals(
+                normalizeBuiltinComponentName(name, 
IndexPolicyTypeEnum.CHAR_FILTER));
+    }
+
+    private static String appendOuterCharFilterIdentity(
+            String analyzerIdentity, Map<String, String> properties) {
+        String type = 
properties.get(InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE);
+        String pattern = 
properties.get(InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN);
+        if (!"char_replace".equals(type) || Strings.isNullOrEmpty(pattern)) {
+            return analyzerIdentity;
+        }
+        String replacement = properties.getOrDefault(
+                
InvertedIndexProperties.INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT, " ");
+        String canonicalPattern = canonicalizeCharReplacePattern(
+                pattern, replacement, 
isDefaultLowercaseBuiltinIkIdentity(analyzerIdentity));

Review Comment:
   [P1] Derive outer-filter redundancy from the effective custom pipeline, not 
only built-in IK. Two differently named analyzers with 
`tokenizer=keyword,token_filter=lowercase` have the same base identity. Giving 
one index outer `char_replace` `A->a` is behavior-neutral because BE applies 
that reader before the lowercase token filter, yet this call passes false for 
the custom identity and appends a suffix only to that index. The distinct alias 
names also bypass the same-selector fence, so CREATE and ALTER admit duplicate 
runtime-equivalent indexes. Include downstream custom case-folding filters in 
this canonicalization and add equivalent-alias coverage with and without 
`A->a`, plus a no-lowercase negative case.



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -226,26 +671,115 @@ private static String resolveTokenFilterIdentity(String 
filterList) {
      * IMPORTANT: Order is preserved because filter order is semantically 
significant.
      */
     private static String resolveCharFilterIdentity(String filterList) {
+        return resolveCharFilterIdentity(filterList, false);
+    }
+
+    private static String resolveCharFilterIdentity(String filterList, boolean 
lowercaseDownstream) {
         if (Strings.isNullOrEmpty(filterList)) {
             return "";
         }
 
-        StringBuilder sb = new StringBuilder();
+        ArrayDeque<String> identities = new ArrayDeque<>();
         String[] filters = filterList.split(",\\s*");
         // DO NOT sort - filter order is semantically significant
 
-        for (int i = 0; i < filters.length; i++) {
-            String filter = filters[i].trim();
-            if (i > 0) {
-                sb.append(",");
+        for (int i = filters.length - 1; i >= 0; --i) {
+            String filterName = filters[i].trim();
+            String filter = resolveComponentIdentity(
+                    filterName, IndexPolicyTypeEnum.CHAR_FILTER, 
lowercaseDownstream);
+            if (Strings.isNullOrEmpty(filter)) {
+                continue;
             }
+            identities.addFirst(filter);
+            lowercaseDownstream = isCaseFoldingCharFilter(filterName);

Review Comment:
   [P1] Preserve a downstream case fold through non-interacting intermediate 
filters. For valid policies `lower_a={A->a}`, `x_to_y={x->y}`, and 
`fold={icu_normalizer}` (`nfkc_cf`), the chains `lower_a,x_to_y,fold` and 
`x_to_y,fold` are equivalent for every input: the middle filter cannot observe 
`A`/`a`, and the final fold maps both to `a`. This reverse walk sees `fold`, 
passes the flag to `x_to_y`, then resets it here because that filter is not 
itself a fold, so it incorrectly retains `lower_a` and CREATE/ALTER admit both 
indexes. Carry the downstream effect through transforms that cannot distinguish 
the candidate case pair (or compose the effective byte transforms) and add this 
intervening-filter duplicate case while retaining the interacting 
`lower_a,a_to_b` negative case.



##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java:
##########
@@ -393,16 +420,40 @@ public static boolean 
canHaveMultipleInvertedIndexes(DataType colType, List<Inde
         }
 
         Set<String> analyzerKeys = new HashSet<>();
+        Set<String> analyzerSelectors = new HashSet<>();
         for (IndexDefinition indexDef : indexDefs) {
-            String key = buildAnalyzerIdentity(indexDef.getProperties());
+            Map<String, String> properties = indexDef.getProperties();
+            String key = buildAnalyzerIdentity(properties);
             // HashSet.add() returns false if element already exists
             if (!analyzerKeys.add(key)) {
                 return false;
             }
+            String selector = getAnalyzerSelector(properties);
+            if (!INVERTED_INDEX_PARSER_IK.equals(selector) && 
!analyzerSelectors.add(selector)) {

Review Comment:
   [P1] Do not admit multiple IK identities unless MATCH syntax can select each 
one. This exception permits smart/max-word x lowercase-on/off indexes on the 
same column because their identities differ, but `USING ANALYZER ik` matches 
only the bare/default max-word+lowercase identity and carries neither mode nor 
lowercase; an unqualified query just returns the first analyzed index. At least 
two legal IK indexes are therefore unreachable, and queries can silently use 
another IK configuration. Either make the selector carry the full effective IK 
identity or reject additional IK identities that cannot be disambiguated, then 
test selection of every accepted variant.



##########
be/src/storage/index/inverted/tokenizer/tokenizer.h:
##########
@@ -39,12 +45,148 @@ class DorisTokenizer : public Tokenizer, public 
DorisTokenStream {
 
     using Tokenizer::reset;
     // Only use the parameterless reset method
-    void reset() override { _in = _in_pending; };
+    void reset() override {
+        _in = _in_pending;
+        _source_byte_offsets.clear();
+        _source_byte_end_offsets.clear();
+        release_oversized_scratch(_source_byte_offsets);
+        release_oversized_scratch(_source_byte_end_offsets);
+    };
+
+    std::span<const int32_t> get_source_byte_offsets() const override {
+        return _source_byte_offsets_enabled ? std::span<const int32_t> 
{_source_byte_offsets}
+                                            : std::span<const int32_t> {};
+    }
+
+    std::span<const int32_t> get_source_byte_end_offsets() const override {
+        return _source_byte_offsets_enabled ? std::span<const int32_t> 
{_source_byte_end_offsets}
+                                            : std::span<const int32_t> {};
+    }
+
+    void set_source_byte_offsets_enabled(bool enabled) override {
+        _source_byte_offsets_enabled = enabled;
+    }
+
+    size_t source_byte_offsets_capacity_for_test() const {
+        return _source_byte_offsets.capacity() + 
_source_byte_end_offsets.capacity();
+    }
 
 protected:
+    int32_t correct_source_offset(int32_t offset) const {
+        const auto* char_filter = dynamic_cast<const 
DorisCharFilter*>(_in.get());
+        return char_filter == nullptr ? offset : 
char_filter->correct_offset(offset);
+    }
+
+    void set_source_byte_offsets(std::string_view term, int32_t source_start) {
+        set_source_byte_offsets(term, term, source_start);
+    }
+
+    void set_source_byte_offsets(std::string_view term, std::string_view 
source,
+                                 int32_t source_start) {
+        _source_byte_offsets.clear();
+        _source_byte_end_offsets.clear();
+        if (!_source_byte_offsets_enabled) {
+            return;
+        }
+
+        const auto* char_filter = dynamic_cast<const 
DorisCharFilter*>(_in.get());
+        const int32_t corrected_start =
+                char_filter == nullptr ? source_start : 
char_filter->correct_offset(source_start);
+        std::vector<int32_t> source_offsets {0};

Review Comment:
   [P2] Reuse the ordinary provenance capacity within a row. Offset-aware 
Pinyin enables this helper on its tokenizer, but every emitted token grows a 
fresh local `source_offsets` vector and the common strictly-increasing branch 
then move-assigns it over `_source_byte_offsets`; the following token repeats 
the allocation and discards the capacity that the new reset policy was designed 
to retain. A standard analyzer therefore performs vector allocation/growth per 
word even across identical reusable rows, and the reset-capacity test cannot 
observe it. Fill reusable tokenizer-owned scratch directly (and apply the same 
rule to IK's return-vector path), while keeping the existing oversized-reset 
high-water behavior.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to