github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4080251317
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -226,26 +925,400 @@ private static String resolveTokenFilterIdentity(String
filterList) {
* IMPORTANT: Order is preserved because filter order is semantically
significant.
*/
private static String resolveCharFilterIdentity(String filterList) {
+ return resolveCharFilterIdentity(filterList, null);
+ }
+
+ private static String resolveCharFilterIdentity(String filterList,
FoldContext downstreamFold) {
+ ArrayDeque<String> identities = new ArrayDeque<>();
+ walkCharFilters(filterList, downstreamFold, identities);
+ return String.join(",", identities);
+ }
+
+ /**
+ * Resolve the chain from its last filter to its first, collecting
identities, and return the
+ * case-folding context that a filter placed in front of the chain would
run in.
+ */
+ private static FoldContext walkCharFilters(
+ String filterList, FoldContext downstreamFold, Deque<String>
identities) {
+ FoldContext fold = downstreamFold;
if (Strings.isNullOrEmpty(filterList)) {
- return "";
+ return fold;
}
- StringBuilder sb = new StringBuilder();
String[] filters = filterList.split(",\\s*");
// DO NOT sort - filter order is semantically significant
- for (int i = 0; i < filters.length; i++) {
- String filter = filters[i].trim();
- if (i > 0) {
- sb.append(",");
+ for (int i = filters.length - 1; i >= 0; --i) {
+ String filterName = filters[i].trim();
+ String filter = resolveComponentIdentity(filterName,
IndexPolicyTypeEnum.CHAR_FILTER, fold);
+ if (Strings.isNullOrEmpty(filter)) {
+ continue;
}
+ identities.addFirst(filter);
Review Comment:
[P1] Collapse an adjacent identical char_replace identity here as well. BE
applies this filter as an independent one-byte substitution, so applying the
same mapping twice is idempotent even when the replacement byte is in the
pattern; it is length-preserving and the second reader delegates the same
offset correction. Keeping both identities lets keyword + char_replace and
keyword + char_replace,char_replace aliases pass both CREATE and ALTER
duplicate fences. This is distinct from the earlier single-filter
pattern/default canonicalization. Preserve order and fold context, and add
bare/named repeat coverage with different or separated filters as negatives.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -17,22 +17,89 @@
package org.apache.doris.analysis.invertedindex;
+import org.apache.doris.analysis.InvertedIndexProperties;
import org.apache.doris.catalog.Env;
import org.apache.doris.indexpolicy.IndexPolicy;
import org.apache.doris.indexpolicy.IndexPolicyTypeEnum;
import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableSet;
+import com.ibm.icu.lang.UCharacter;
+import com.ibm.icu.text.UnicodeSet;
import org.apache.logging.log4j.Logger;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Deque;
+import java.util.List;
+import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.TreeMap;
+import java.util.TreeSet;
+import java.util.regex.Pattern;
public final class AnalyzerIdentityBuilder {
private static final String PROP_MAX_NGRAM_DIFF = "max_ngram_diff";
+ private static final String KEYWORD_TOKENIZER = "keyword";
+ private static final String CHAR_REPLACE_FILTER = "char_replace";
+ private static final String PROP_PATTERN = "pattern";
+ private static final String PROP_REPLACEMENT = "replacement";
+ // Defaults CharReplaceCharFilterFactory applies to a bare built-in
reference.
+ private static final String CHAR_REPLACE_DEFAULT_PATTERN = ",._";
+ private static final String CHAR_REPLACE_DEFAULT_REPLACEMENT = " ";
+ // Token filters that emit the same terms, offsets and provenance when
applied twice in a row.
+ private static final Set<String> IDEMPOTENT_TOKEN_FILTERS =
ImmutableSet.of("lowercase", "asciifolding");
Review Comment:
[P1] Extend this proven-idempotent set to the canonical default
icu_normalizer and word_delimiter filters. Bare ICU uses unfiltered nfkc_cf, so
the second pass leaves normalized terms, positions, offsets, and delegated
provenance unchanged. Bare WordDelimiter emits maximal default subwords that
the identical second pass treats as single words, preserving position
increments, token offsets, and exact or conservative provenance. Keeping either
repeat gives one-pass and two-pass aliases different identities, so equivalent
indexes pass both CREATE and ALTER fences. This is distinct from the existing
lowercase/asciifolding threads. Cover both identities and DDL paths, while
retaining configured variants as negatives unless separately proven.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
