github-actions[bot] commented on code in PR #64667:
URL: https://github.com/apache/doris/pull/64667#discussion_r3628492959
##########
be/src/storage/index/inverted/analyzer/analyzer.cpp:
##########
@@ -107,6 +109,17 @@ AnalyzerPtr
InvertedIndexAnalyzer::create_builtin_analyzer(InvertedIndexParserTy
ik_analyzer->setMode(false);
}
analyzer = std::move(ik_analyzer);
+ } else if (parser_type == InvertedIndexParserType::PARSER_KUROMOJI) {
+ if (!config::enable_kuromoji_analyzer) {
+ throw Exception(ErrorCode::INVERTED_INDEX_ANALYZER_ERROR,
+ "kuromoji analyzer is disabled by default. Set "
+ "enable_kuromoji_analyzer=true in "
+ "be.conf (or via the BE config HTTP API) to enable
it.");
+ }
Review Comment:
[Major] Default the mode from the effective analyzer before calling the
strict Kuromoji parser. On the supported slow path where a CTE/projection loses
`Index` metadata, `USING ANALYZER kuromoji` is retained, but `MatchPredicate`
derives the mode from an empty property map and sends `coarse_grained`; the BE
recognizes the builtin analyzer and throws here. Derive `search` from the
effective Kuromoji selection and add a metadata-loss slow-path test.
##########
fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java:
##########
@@ -81,13 +85,24 @@ public static String getInvertedIndexParserMode(Map<String,
String> properties)
return INVERTED_INDEX_PARSER_COARSE_GRANULARITY;
}
String mode = properties.get(INVERTED_INDEX_PARSER_MODE_KEY);
+ if (mode != null) {
+ return mode;
+ }
String parser = properties.get(INVERTED_INDEX_PARSER_KEY);
if (parser == null) {
parser = properties.get(INVERTED_INDEX_PARSER_KEY_ALIAS);
}
- return mode != null ? mode :
- INVERTED_INDEX_PARSER_IK.equals(parser) ?
INVERTED_INDEX_PARSER_SMART :
- INVERTED_INDEX_PARSER_COARSE_GRANULARITY;
+ if (INVERTED_INDEX_PARSER_IK.equals(parser)) {
+ return INVERTED_INDEX_PARSER_SMART;
+ }
Review Comment:
[Major] Persist the Kuromoji default instead of leaving it dependent on the
running FE version. `Index` stores the user property map without `parser_mode`;
after failover to a pre-change FE, that FE derives `coarse_grained` and sends
it to a new BE, whose Kuromoji mode parser rejects it. Materialize
`parser_mode=search` when accepting an omitted mode (including the parser alias
and builtin analyzer form), or gate the feature until all FEs are upgraded, and
add a catalog serialization/failover test.
##########
be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.cpp:
##########
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.h"
+
+#include <algorithm>
+#include <string_view>
+
+#include "common/exception.h"
+#include "storage/index/inverted/analyzer/kuromoji/kuromoji_normalize.h"
+
+namespace doris::segment_v2 {
+
+namespace {
+// Returns the idx-th comma-separated field of an IPADIC feature string
+// (0=POS1 ... 6=base form, 7=reading, 8=pronunciation), or empty.
+std::string_view feature_field(std::string_view feat, int idx) {
+ int cur = 0;
+ std::size_t start = 0;
+ for (std::size_t i = 0; i <= feat.size(); ++i) {
+ if (i == feat.size() || feat[i] == ',') {
+ if (cur == idx) {
+ return feat.substr(start, i - start);
+ }
+ ++cur;
+ start = i + 1;
+ }
+ }
+ return {};
+}
+
+// Part-of-speech (POS1) classes dropped for full-text search. A coarse subset
of
+// Lucene/OpenSearch's ja stoptags: particles, auxiliary verbs, conjunctions,
+// symbols, fillers. (Full stoptags fidelity is a later refinement.)
+bool is_stop_pos(std::string_view pos1) {
+ return pos1 == "\xE5\x8A\xA9\xE8\xA9\x9E" || // 助詞
(particle)
+ pos1 == "\xE5\x8A\xA9\xE5\x8B\x95\xE8\xA9\x9E" || // 助動詞
(auxiliary verb)
+ pos1 == "\xE6\x8E\xA5\xE7\xB6\x9A\xE8\xA9\x9E" || // 接続詞
(conjunction)
+ pos1 == "\xE8\xA8\x98\xE5\x8F\xB7" || // 記号
(symbol)
+ pos1 == "\xE3\x83\x95\xE3\x82\xA3\xE3\x83\xA9\xE3\x83\xBC"; // フィラー
(filler)
+}
+
+void ascii_lower(std::string& s) {
+ for (char& c : s) {
+ if (c >= 'A' && c <= 'Z') {
+ c = static_cast<char>(c - 'A' + 'a');
+ }
+ }
+}
+} // namespace
+
+KuromojiTokenizer::KuromojiTokenizer(KuromojiMode mode, bool lower_case, bool
own_reader,
+ const
inverted_index::kuromoji::KuromojiDictionary* dict)
+ : mode_(mode), dict_(dict) {
+ this->lowercase = lower_case;
+ this->ownReader = own_reader;
+}
+
+void KuromojiTokenizer::reset(lucene::util::Reader* reader) {
+ this->input = reader;
+ buffer_index_ = 0;
+ data_length_ = 0;
+ tokens_text_.clear();
+
+ // Read the entire input. readCopy returns the count read, or <= 0 at EOF.
+ std::string text;
+ char buf[4096];
+ int32_t n = 0;
+ while ((n = reader->readCopy(buf, 0, static_cast<int32_t>(sizeof(buf)))) >
0) {
+ text.append(buf, n);
+ }
+
+ if (dict_ == nullptr) {
+ throw doris::Exception(doris::ErrorCode::INVERTED_INDEX_ANALYZER_ERROR,
+ "kuromoji tokenizer requires a loaded
dictionary");
+ }
+
+ // Viterbi morphological segmentation, then OpenSearch-default-style
filtering:
+ // drop stop part-of-speech (particles/auxiliaries/...), emit the
dictionary
+ // base form for conjugated words, and lowercase embedded ASCII.
+ inverted_index::kuromoji::KuromojiViterbi viterbi(*dict_, mode_);
+ std::vector<inverted_index::kuromoji::KuromojiMorpheme> morphemes;
+ viterbi.segment(text, &morphemes);
+ tokens_text_.reserve(morphemes.size());
+ for (const auto& m : morphemes) {
+ const std::string_view feat =
+ m.known ? dict_->feature(dict_->word(m.word_id))
+ :
dict_->unknown_feature(dict_->unknown_word(m.word_id));
+ if (is_stop_pos(feature_field(feat, 0))) {
+ continue; // part-of-speech stop filtering
Review Comment:
[Major] Preserve the correct positions for filtered morphemes. `reset()`
removes particles, punctuation, and spaces through this one branch, while
`next()` calls `setNoCopy`, which resets every retained increment to 1; `寿司が好き`
is therefore indexed with adjacent `寿司`/`好き` postings and can falsely match
`MATCH_PHRASE '寿司好き'`. Split the semantics: carry position gaps for linguistic
POS stops, discard punctuation/space without a gap, and restore the chosen
increment after `setNoCopy`. Then make every active phrase-family path consume
`TermInfo.position` (including the legacy `PhraseQuery` matchers and
`PhraseEdgeQuery`), or positive queries containing the same removed particle
will fail. Add phrase/prefix/edge tests around particles, punctuation, and
whitespace.
##########
be/src/tools/CMakeLists.txt:
##########
@@ -116,3 +116,51 @@ if (BUILD_INDEX_TOOL)
)
endif()
endif()
+
+# Offline generator for the Japanese kuromoji dictionary. Compiles the UTF-8
+# mecab-ipadic source into the four dict/kuromoji/*.bin files that the BE
install
+# rule ships. EXCLUDE_FROM_ALL and not installed: it is a maintenance tool, not
+# part of doris_be. Regenerate with `ninja kuromoji_dict`.
+add_executable(kuromoji_build_dict EXCLUDE_FROM_ALL
+ kuromoji_build_dict.cpp
+)
+
+target_include_directories(kuromoji_build_dict PRIVATE
${PROJECT_SOURCE_DIR}/..)
+
+pch_reuse(kuromoji_build_dict)
+
+set_target_properties(kuromoji_build_dict PROPERTIES ENABLE_EXPORTS 1)
+
+if (COMPILER_CLANG)
+ target_compile_options(kuromoji_build_dict PRIVATE
+ -Wno-implicit-int-conversion
+ -Wno-shorten-64-to-32
+ )
+endif()
+
+target_link_libraries(kuromoji_build_dict
+ ${DORIS_LINK_LIBS}
+)
+
+# `ninja kuromoji_dict` runs the tool over the staged mecab-ipadic source and
+# (re)writes dict/kuromoji/*.bin. Point KUROMOJI_IPADIC_SRC elsewhere if the
+# source is not under the thirdparty share directory.
+set(KUROMOJI_IPADIC_SRC "${THIRDPARTY_DIR}/share/mecab-ipadic-2.7.0-20250920"
+ CACHE PATH "UTF-8 mecab-ipadic source directory used to generate the
kuromoji dictionary")
+set(KUROMOJI_DICT_OUT "${BASE_DIR}/dict/kuromoji")
+add_custom_command(
+ OUTPUT "${KUROMOJI_DICT_OUT}/system.bin" "${KUROMOJI_DICT_OUT}/matrix.bin"
+ "${KUROMOJI_DICT_OUT}/chardef.bin"
"${KUROMOJI_DICT_OUT}/unkdict.bin"
+ COMMAND ${CMAKE_COMMAND} -E make_directory "${KUROMOJI_DICT_OUT}"
+ COMMAND $<TARGET_FILE:kuromoji_build_dict> "${KUROMOJI_IPADIC_SRC}"
"${KUROMOJI_DICT_OUT}"
+ DEPENDS kuromoji_build_dict
Review Comment:
[Major] Track the IPADIC files that this command consumes. The edge depends
only on the converter executable, so after an initial generation, editing or
replacing a CSV/`matrix.def`/`char.def`/`unk.def` under the same
`KUROMOJI_IPADIC_SRC` leaves `ninja kuromoji_dict` up to date and packages
stale binaries. Add the complete source set (including file-addition handling)
to `DEPENDS`, and test that changing one input regenerates all four outputs.
##########
build.sh:
##########
@@ -465,6 +465,12 @@ if [[ ! -f
"${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ]]; then
fi
fi
+MECAB_IPADIC_DIR="${DORIS_THIRDPARTY}/installed/share/mecab-ipadic-2.7.0-20250920"
Review Comment:
[Major] Keep this BE-only staging out of unrelated build modes. The block
runs before the clean-only exit and ignores `BUILD_BE`, so `./build.sh --fe`,
`--cloud`, or even `--clean` invokes `build-thirdparty.sh ... mecab_ipadic`
when an existing cache lacks the new directory. That adds an unexpected
download/mutation and can break formerly offline FE/clean workflows. Gate
staging on the mode that actually configures the dictionary generator and
express the dependency from its real BE CMake target.
##########
be/src/tools/kuromoji_build_dict.cpp:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Offline tool: compile a UTF-8 mecab-ipadic source directory into the four
+// kuromoji .bin files consumed by KuromojiDictionary.
+// usage: kuromoji_build_dict <ipadic_src_dir> <out_dir>
+// Built on demand via `ninja kuromoji_dict`; never linked into doris_be.
+
+#include <cstdio>
+#include <filesystem>
+#include <fstream>
+#include <functional>
+#include <sstream>
+#include <string>
+#include <string_view>
+#include <system_error>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include
"storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.h"
+#include
"storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.h"
+
+namespace fs = std::filesystem;
+using namespace doris::segment_v2::inverted_index::kuromoji;
+using doris::Status;
+
+namespace {
+
+bool read_file(const std::string& path, std::string* out) {
+ std::ifstream in(path, std::ios::binary);
+ if (!in) {
+ std::fprintf(stderr, "cannot open %s\n", path.c_str());
+ return false;
+ }
+ std::ostringstream ss;
+ ss << in.rdbuf();
+ *out = ss.str();
+ return true;
+}
+
+void for_each_line(const std::string& content, const
std::function<void(std::string_view)>& fn) {
+ std::size_t i = 0;
+ while (i < content.size()) {
+ auto nl = content.find('\n', i);
+ if (nl == std::string::npos) {
+ nl = content.size();
+ }
+ std::string_view line(content.data() + i, nl - i);
+ if (!line.empty() && line.back() == '\r') {
+ line.remove_suffix(1);
+ }
+ if (!line.empty()) {
+ fn(line);
+ }
+ i = nl + 1;
+ }
+}
+
+} // namespace
+
+int main(int argc, char** argv) {
+ if (argc < 3) {
+ std::fprintf(stderr, "usage: %s <ipadic_src_dir> <out_dir>\n",
argv[0]);
+ return 2;
+ }
+ const std::string src = argv[1];
+ const std::string out = argv[2];
+ std::error_code ec;
+ fs::create_directories(out, ec);
+
+ // --- system dictionary: group all *.csv lexicon rows by surface
(homographs) ---
+ std::unordered_map<std::string, std::vector<BuilderWord>> by_surface;
+ std::size_t lexicon_rows = 0;
+ for (const auto& entry : fs::directory_iterator(src)) {
Review Comment:
[Major] Make dictionary generation independent of filesystem enumeration
order. `directory_iterator` is unordered, and records are appended to each
surface's homograph vector in that order; `write_system` sorts surface keys but
never sorts the words within a surface. Cross-file homographs therefore change
entry/feature order and the generated `system.bin` bytes across filesystems,
and cost ties can make that order affect selected features. Sort the CSV paths
and each homograph by a complete stable key, then compare generated bytes under
permuted input order.
##########
be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.cpp:
##########
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.h"
+
+#include <algorithm>
+#include <string_view>
+
+#include "common/exception.h"
+#include "storage/index/inverted/analyzer/kuromoji/kuromoji_normalize.h"
+
+namespace doris::segment_v2 {
+
+namespace {
+// Returns the idx-th comma-separated field of an IPADIC feature string
+// (0=POS1 ... 6=base form, 7=reading, 8=pronunciation), or empty.
+std::string_view feature_field(std::string_view feat, int idx) {
+ int cur = 0;
+ std::size_t start = 0;
+ for (std::size_t i = 0; i <= feat.size(); ++i) {
+ if (i == feat.size() || feat[i] == ',') {
+ if (cur == idx) {
+ return feat.substr(start, i - start);
+ }
+ ++cur;
+ start = i + 1;
+ }
+ }
+ return {};
+}
+
+// Part-of-speech (POS1) classes dropped for full-text search. A coarse subset
of
+// Lucene/OpenSearch's ja stoptags: particles, auxiliary verbs, conjunctions,
+// symbols, fillers. (Full stoptags fidelity is a later refinement.)
+bool is_stop_pos(std::string_view pos1) {
+ return pos1 == "\xE5\x8A\xA9\xE8\xA9\x9E" || // 助詞
(particle)
+ pos1 == "\xE5\x8A\xA9\xE5\x8B\x95\xE8\xA9\x9E" || // 助動詞
(auxiliary verb)
+ pos1 == "\xE6\x8E\xA5\xE7\xB6\x9A\xE8\xA9\x9E" || // 接続詞
(conjunction)
+ pos1 == "\xE8\xA8\x98\xE5\x8F\xB7" || // 記号
(symbol)
+ pos1 == "\xE3\x83\x95\xE3\x82\xA3\xE3\x83\xA9\xE3\x83\xBC"; // フィラー
(filler)
+}
+
+void ascii_lower(std::string& s) {
+ for (char& c : s) {
+ if (c >= 'A' && c <= 'Z') {
+ c = static_cast<char>(c - 'A' + 'a');
+ }
+ }
+}
+} // namespace
+
+KuromojiTokenizer::KuromojiTokenizer(KuromojiMode mode, bool lower_case, bool
own_reader,
+ const
inverted_index::kuromoji::KuromojiDictionary* dict)
+ : mode_(mode), dict_(dict) {
+ this->lowercase = lower_case;
+ this->ownReader = own_reader;
+}
+
+void KuromojiTokenizer::reset(lucene::util::Reader* reader) {
+ this->input = reader;
+ buffer_index_ = 0;
+ data_length_ = 0;
+ tokens_text_.clear();
+
+ // Read the entire input. readCopy returns the count read, or <= 0 at EOF.
+ std::string text;
+ char buf[4096];
+ int32_t n = 0;
+ while ((n = reader->readCopy(buf, 0, static_cast<int32_t>(sizeof(buf)))) >
0) {
+ text.append(buf, n);
+ }
+
+ if (dict_ == nullptr) {
+ throw doris::Exception(doris::ErrorCode::INVERTED_INDEX_ANALYZER_ERROR,
+ "kuromoji tokenizer requires a loaded
dictionary");
+ }
+
+ // Viterbi morphological segmentation, then OpenSearch-default-style
filtering:
+ // drop stop part-of-speech (particles/auxiliaries/...), emit the
dictionary
+ // base form for conjugated words, and lowercase embedded ASCII.
+ inverted_index::kuromoji::KuromojiViterbi viterbi(*dict_, mode_);
+ std::vector<inverted_index::kuromoji::KuromojiMorpheme> morphemes;
+ viterbi.segment(text, &morphemes);
+ tokens_text_.reserve(morphemes.size());
+ for (const auto& m : morphemes) {
+ const std::string_view feat =
+ m.known ? dict_->feature(dict_->word(m.word_id))
+ :
dict_->unknown_feature(dict_->unknown_word(m.word_id));
+ if (is_stop_pos(feature_field(feat, 0))) {
Review Comment:
[Major] Do not use the dictionary POS alone to decide whether punctuation is
searchable. In the pinned IPADIC source, unknown characters in the `SYMBOL`
category use the feature `名詞,サ変接続`, so an uncatalogued symbol such as U+2603
survives this `記号` check and is emitted/indexed. The reference Japanese
tokenizer discards punctuation independently of its later POS filter. Preserve
the character category/surface needed to apply that behavior and add an
unknown-symbol TOKENIZE/index-query test.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java:
##########
@@ -124,7 +125,8 @@ public static void checkInvertedIndexParser(String
indexColName, PrimitiveType c
|| parser.equals(INVERTED_INDEX_PARSER_CHINESE)
|| parser.equals(INVERTED_INDEX_PARSER_ICU)
||
parser.equals(INVERTED_INDEX_PARSER_BASIC)
- ||
parser.equals(INVERTED_INDEX_PARSER_IK))) {
+ ||
parser.equals(INVERTED_INDEX_PARSER_IK)
Review Comment:
[Major] Gate Kuromoji until every target BE can interpret it. A pre-change
BE maps the persisted `kuromoji` string to `PARSER_UNKNOWN`;
`should_analyzer()` then returns false and the writer silently creates an
`INDEX_UNTOKENIZED` keyword field. New BEs build morpheme terms from the same
metadata, so writes during a rolling upgrade can commit permanently
incompatible index segments. Reject this DDL until all target BEs advertise
Kuromoji readiness (including config/dictionary), and make unknown analyzed
parsers fail rather than silently becoming keyword indexes.
##########
be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.cpp:
##########
@@ -0,0 +1,266 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.h"
+
+#include <algorithm>
+#include <cstddef>
+#include <limits>
+
+namespace doris::segment_v2::inverted_index::kuromoji {
+
+namespace {
+
+constexpr int64_t KMJ_INF = std::numeric_limits<int64_t>::max() / 4;
+constexpr uint32_t MAX_UNKNOWN_GROUP_CHARS = 1024;
+
+// Search/Extended-mode compound-decomposition penalties, matching Lucene's
+// JapaneseTokenizer. Lengths are counted in code points. A token longer than
the
+// length threshold is penalized so the minimum-cost path prefers its shorter
+// parts: all-kanji runs over KANJI_LENGTH chars, other runs over OTHER_LENGTH.
+constexpr uint32_t SEARCH_MODE_KANJI_LENGTH = 2;
+constexpr int64_t SEARCH_MODE_KANJI_PENALTY = 3000;
+constexpr uint32_t SEARCH_MODE_OTHER_LENGTH = 7;
+constexpr int64_t SEARCH_MODE_OTHER_PENALTY = 1700;
+
+struct DecodedCp {
+ char32_t cp;
+ uint32_t len;
+};
+
+// Decode one UTF-8 code point at text[pos]. Invalid/truncated -> single byte.
+DecodedCp decode_utf8(std::string_view text, std::size_t pos) {
+ auto b0 = static_cast<unsigned char>(text[pos]);
+ const std::size_t avail = text.size() - pos;
+ if (b0 < 0x80) {
+ return {b0, 1};
+ }
+ if ((b0 >> 5) == 0x6 && avail >= 2) {
+ auto b1 = static_cast<unsigned char>(text[pos + 1]);
+ return {static_cast<char32_t>(((b0 & 0x1FU) << 6) | (b1 & 0x3FU)), 2};
+ }
+ if ((b0 >> 4) == 0xE && avail >= 3) {
+ auto b1 = static_cast<unsigned char>(text[pos + 1]);
+ auto b2 = static_cast<unsigned char>(text[pos + 2]);
+ return {static_cast<char32_t>(((b0 & 0x0FU) << 12) | ((b1 & 0x3FU) <<
6) | (b2 & 0x3FU)),
+ 3};
+ }
+ if ((b0 >> 3) == 0x1E && avail >= 4) {
+ auto b1 = static_cast<unsigned char>(text[pos + 1]);
+ auto b2 = static_cast<unsigned char>(text[pos + 2]);
+ auto b3 = static_cast<unsigned char>(text[pos + 3]);
+ return {static_cast<char32_t>(((b0 & 0x07U) << 18) | ((b1 & 0x3FU) <<
12) |
+ ((b2 & 0x3FU) << 6) | (b3 & 0x3FU)),
+ 4};
+ }
+ return {b0, 1};
+}
+
+// Lucene JapaneseTokenizer's search-mode penalty for the token covering
+// [start, end) bytes: penalize long compounds so the Viterbi prefers their
+// shorter parts. Returns 0 for tokens at or under the length thresholds.
+int64_t compute_penalty(const KuromojiDictionary& dict, std::string_view text,
uint32_t start,
+ uint32_t end) {
+ uint32_t length = 0;
+ bool all_kanji = true;
+ for (uint32_t p = start; p < end;) {
+ const DecodedCp d = decode_utf8(text, p);
+ if (dict.char_category(d.cp) != CAT_KANJI) {
+ all_kanji = false;
+ }
+ p += d.len;
+ ++length;
+ }
+ if (length > SEARCH_MODE_KANJI_LENGTH) {
+ if (all_kanji) {
+ return static_cast<int64_t>(length - SEARCH_MODE_KANJI_LENGTH) *
+ SEARCH_MODE_KANJI_PENALTY;
+ }
+ if (length > SEARCH_MODE_OTHER_LENGTH) {
+ return static_cast<int64_t>(length - SEARCH_MODE_OTHER_LENGTH) *
+ SEARCH_MODE_OTHER_PENALTY;
+ }
+ }
+ return 0;
+}
+
+// A lattice node spanning [start, end) bytes of the input.
+struct VNode {
+ uint32_t start;
+ uint32_t end;
+ int16_t left_id;
+ int16_t right_id;
+ int16_t word_cost;
+ bool known;
+ uint32_t word_id;
+ int64_t total_cost;
+ int back; // previous node index, -1 if none
+};
+
+} // namespace
+
+void KuromojiViterbi::segment(std::string_view text,
std::vector<KuromojiMorpheme>* out) const {
+ out->clear();
+ const auto n = static_cast<uint32_t>(text.size());
+ if (n == 0) {
+ return;
+ }
+
+ std::vector<VNode> nodes;
+ std::vector<int32_t> end_head(n + 1, -1);
+ std::vector<int32_t> end_next;
+
+ // BOS (index 0): ends at position 0, context id 0, zero cost.
+ nodes.push_back(VNode {0, 0, 0, 0, 0, false, 0, 0, -1});
+ end_next.push_back(-1);
+ end_head[0] = 0;
+
+ // Add a node and relax it against all nodes ending at its start position.
+ auto add_node = [&](uint32_t s, uint32_t e, int16_t lid, int16_t rid,
int16_t wcost, bool known,
+ uint32_t wid) {
+ int64_t best = KMJ_INF;
+ int best_prev = -1;
+ for (int pe = end_head[s]; pe >= 0; pe = end_next[pe]) {
+ const VNode& pv = nodes[static_cast<std::size_t>(pe)];
+ if (pv.total_cost >= KMJ_INF) {
+ continue;
+ }
+ const int64_t c =
+ pv.total_cost +
_dict.connection_cost(static_cast<uint32_t>(pv.right_id),
+
static_cast<uint32_t>(lid));
+ if (c <= best) {
+ best = c;
+ best_prev = pe;
+ }
+ }
+ if (best_prev < 0) {
+ return;
+ }
+ // Search/Extended mode penalizes long compounds so shorter parts win.
+ const int64_t penalty =
+ _mode == KuromojiMode::Normal ? 0 : compute_penalty(_dict,
text, s, e);
+ const auto idx = static_cast<int>(nodes.size());
+ nodes.push_back(
+ VNode {s, e, lid, rid, wcost, known, wid, best + wcost +
penalty, best_prev});
+ end_next.push_back(end_head[e]);
+ end_head[e] = idx;
+ };
+
+ uint32_t pos = 0;
+ std::vector<KuromojiDictionary::PrefixMatch> matches;
+ while (pos < n) {
+ if (end_head[pos] < 0) {
+ pos += decode_utf8(text, pos).len; // unreachable boundary; skip
+ continue;
+ }
+ const DecodedCp d0 = decode_utf8(text, pos);
+ const auto before = nodes.size();
+
+ // System-dictionary words (common-prefix search).
+ _dict.common_prefix_search(text.data() + pos, n - pos, &matches);
+ bool any_known = false;
+ for (const auto& mt : matches) {
+ const WordIdRun run = _dict.run_for_value(mt.trie_value);
+ for (uint32_t k = 0; k < run.count; ++k) {
+ const uint32_t wid = run.entry_start + k;
+ const WordEntry& e = _dict.word(wid);
+ add_node(pos, pos + mt.length, e.left_id, e.right_id,
e.word_cost, true, wid);
+ any_known = true;
+ }
+ }
+
+ // Unknown words: when no known word starts here, or the category
forces it.
+ if (!any_known || _dict.is_invoke(d0.cp)) {
+ const uint8_t cat = _dict.char_category(d0.cp);
+ uint32_t group_len = d0.len;
+ if (_dict.is_group(d0.cp)) {
+ uint32_t p = pos + d0.len;
+ uint32_t chars = 1;
+ while (p < n && chars < MAX_UNKNOWN_GROUP_CHARS) {
Review Comment:
[Major] Avoid rescanning every suffix of a grouped unknown run. Because the
single-character node below keeps every boundary reachable, a long `ALPHA`/OOV
value scans up to 1,024 code points at every position, then `compute_penalty`
scans the group again for every unknown-dictionary entry while both single and
grouped nodes are retained. Tokenized fields do not apply `ignore_above`, so
ordinary long text can cause massive CPU and lattice-memory amplification.
Precompute same-category run ends/reuse penalties and add a bounded long-OOV
stress test.
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java:
##########
@@ -69,7 +69,7 @@ public class IndexPolicy implements Writable,
GsonPostProcessable {
"empty", "char_replace", "icu_normalizer");
public static final Set<String> BUILTIN_ANALYZERS = ImmutableSet.of(
- "none", "standard", "unicode", "english", "chinese", "icu",
"basic", "ik");
+ "none", "standard", "unicode", "english", "chinese", "icu",
"basic", "ik", "kuromoji");
Review Comment:
[Major] Preserve existing custom analyzers named `kuromoji`. Before this
change the name was not reserved, so a cluster could persist `CREATE INVERTED
INDEX ANALYZER kuromoji ...` and indexes with `analyzer=kuromoji`. After
upgrade, FE treats that reference as builtin and new BE dispatches it to IPADIC
before consulting the custom-policy manager, while old segments/BEs still use
the custom analyzer; this can mix incompatible terms, fail queries, or fail
writes while the builtin config is off. Detect and migrate/reject an existing
name collision, or preserve custom-policy precedence, and add an upgrade test.
##########
be/src/tools/kuromoji_build_dict.cpp:
##########
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Offline tool: compile a UTF-8 mecab-ipadic source directory into the four
+// kuromoji .bin files consumed by KuromojiDictionary.
+// usage: kuromoji_build_dict <ipadic_src_dir> <out_dir>
+// Built on demand via `ninja kuromoji_dict`; never linked into doris_be.
+
+#include <cstdio>
+#include <filesystem>
+#include <fstream>
+#include <functional>
+#include <sstream>
+#include <string>
+#include <string_view>
+#include <system_error>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include
"storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.h"
+#include
"storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.h"
+
+namespace fs = std::filesystem;
+using namespace doris::segment_v2::inverted_index::kuromoji;
+using doris::Status;
+
+namespace {
+
+bool read_file(const std::string& path, std::string* out) {
+ std::ifstream in(path, std::ios::binary);
+ if (!in) {
+ std::fprintf(stderr, "cannot open %s\n", path.c_str());
+ return false;
+ }
+ std::ostringstream ss;
+ ss << in.rdbuf();
+ *out = ss.str();
+ return true;
+}
+
+void for_each_line(const std::string& content, const
std::function<void(std::string_view)>& fn) {
+ std::size_t i = 0;
+ while (i < content.size()) {
+ auto nl = content.find('\n', i);
+ if (nl == std::string::npos) {
+ nl = content.size();
+ }
+ std::string_view line(content.data() + i, nl - i);
+ if (!line.empty() && line.back() == '\r') {
+ line.remove_suffix(1);
+ }
+ if (!line.empty()) {
+ fn(line);
+ }
+ i = nl + 1;
+ }
+}
+
+} // namespace
+
+int main(int argc, char** argv) {
+ if (argc < 3) {
+ std::fprintf(stderr, "usage: %s <ipadic_src_dir> <out_dir>\n",
argv[0]);
+ return 2;
+ }
+ const std::string src = argv[1];
+ const std::string out = argv[2];
+ std::error_code ec;
+ fs::create_directories(out, ec);
+
+ // --- system dictionary: group all *.csv lexicon rows by surface
(homographs) ---
+ std::unordered_map<std::string, std::vector<BuilderWord>> by_surface;
+ std::size_t lexicon_rows = 0;
+ for (const auto& entry : fs::directory_iterator(src)) {
+ if (!entry.is_regular_file() || entry.path().extension() != ".csv") {
+ continue;
+ }
+ std::string content;
+ if (!read_file(entry.path().string(), &content)) {
+ return 1;
+ }
+ for_each_line(content, [&](std::string_view line) {
+ std::string surface;
+ BuilderWord w;
+ if (parse_lexicon_line(line, &surface, &w).ok()) {
Review Comment:
[Major] Fail the dictionary build when source records cannot be compiled.
This loop silently drops every `parse_lexicon_line` error; `parse_char_def` and
`parse_unk_def` similarly skip malformed rows while returning OK, and
`parse_matrix_def` leaves missing coordinates as zero. A truncated or custom
IPADIC source can therefore produce packageable, structurally valid binaries
with missing words/categories or wrong transition costs. Propagate file/line
errors, require complete/unique matrix and category coverage, and add
malformed-input tool tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
