airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4004155515
##########
be/src/storage/index/inverted/tokenizer/keyword/keyword_tokenizer.h:
##########
@@ -45,6 +45,8 @@ class KeywordTokenizer : public DorisTokenizer {
int32_t length = std::min(_char_length, MAX_TOKEN_LENGTH_LIMIT);
std::string_view term(_char_buffer, length);
set(token, term);
+ token->setStartOffset(0);
+ token->setEndOffset(length);
Review Comment:
Fixed in 783f7657. Keyword and Standard tokenizers now publish
character-filter-corrected rune boundaries when Pinyin requests source offsets;
Pinyin consumes those boundaries for partial terms and reset paths. Coverage
includes ICU-normalized full-width input for both tokenizers.
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java:
##########
@@ -80,11 +81,44 @@ private void readUnlock() {
lock.readLock().unlock();
}
+ // Legacy metadata may contain names that collide after locale-independent
normalization.
+ // Policy IDs are allocated monotonically, so the higher ID reproduces the
latest definition.
+ // Callers must hold the write lock.
+ private void registerPolicyNameLocked(IndexPolicy indexPolicy) {
+ String normalizedName = normalizeKey(indexPolicy.getName());
+ IndexPolicy current = nameToIndexPolicy.get(normalizedName);
+ if (current == null || indexPolicy.getId() > current.getId()) {
+ nameToIndexPolicy.put(normalizedName, indexPolicy);
+ }
+ if (current != null && current.getId() != indexPolicy.getId()) {
+ LOG.warn("Index policies '{}' (id={}) and '{}' (id={}) have the
same normalized name; "
+ + "using the policy with the higher ID for name
lookup",
+ current.getName(), current.getId(), indexPolicy.getName(),
indexPolicy.getId());
+ }
+ }
+
+ private void unregisterPolicyNameLocked(IndexPolicy indexPolicy) {
+ String normalizedName = normalizeKey(indexPolicy.getName());
+ IndexPolicy current = nameToIndexPolicy.get(normalizedName);
+ if (current == null || current.getId() != indexPolicy.getId()) {
+ return;
+ }
+ nameToIndexPolicy.remove(normalizedName);
+ for (IndexPolicy remaining : idToIndexPolicy.values()) {
+ if (normalizedName.equals(normalizeKey(remaining.getName()))) {
+ registerPolicyNameLocked(remaining);
+ }
+ }
+ }
+
public List<IndexPolicy> getCopiedIndexPolicies() {
List<IndexPolicy> copiedPolicies = Lists.newArrayList();
readLock();
try {
- copiedPolicies.addAll(idToIndexPolicy.values());
+ // Only transmit the authoritative policy for each normalized
name. Legacy images may
+ // contain collisions, but sending both definitions would make BE
choose based on
+ // arrival order and repeatedly diverge from FE during
reconciliation.
+ copiedPolicies.addAll(nameToIndexPolicy.values());
Review Comment:
Fixed in 783f7657. FE and BE now retain exact trimmed-name bindings
alongside deterministic normalized fallback, and FE sends every legacy policy
during BE reconciliation. Journal/image and BE analyzer-term tests cover
case-distinct historical component policies in both arrival orders.
##########
be/src/storage/index/inverted/token_filter/pinyin_filter.cpp:
##########
@@ -234,7 +239,15 @@ bool PinyinFilter::processCurrentToken() {
// Convert to Unicode codepoints for processing
std::vector<UChar32> source_codepoints;
- convertToRunes(current_source_, source_codepoints);
+ current_runes_ = convertToRunes(current_source_, source_codepoints);
+
+ if (current_source_ == current_token_text_ &&
Review Comment:
Fixed in 783f7657. Pinyin now slices and rebases upstream rune boundaries
after trimming, preserving the leading source displacement for partial terms
and reset input. The keyword trim test verifies the expected spans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
