airborne12 opened a new pull request, #70550: URL: https://github.com/apache/doris/pull/70550
### What problem does this PR solve? Issue Number: #64646 Related PR: https://github.com/apache/doris/pull/64667; Kuromoji portion of https://github.com/apache/doris/pull/67451 (https://github.com/apache/doris/issues/67448). Problem Summary: Backport the Japanese Kuromoji morphological analyzer to `branch-4.1`. This includes dictionary generation and packaging, parser registration, FE property validation, and the original unit and regression tests. The analyzer remains disabled by default through `enable_kuromoji_analyzer`; its default mode is `search` when enabled. Include the follow-up macOS ARM64 dictionary-converter allocator repair from #67451. The independent SNII CRC repair from that PR is outside this backport's selected scope. The release branch lacks the upstream unit-test allocator workaround, so allocator stubs are added with a private converter-only guard. Source feature squash: `11171329d6d0b56bf5d45b4f212aa311b92da946`. Selected follow-up: `383be27d963c6bac3595e3528510348a65da0551` (manually adapted Kuromoji subset). Candidate commits: - `66e75a7b6f8e471a027c19cc1144de4ecb9bdd78` branch-4.1: [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer #64667 - `8f2370e0b4f409cbb242af8cd55835c2581354fd` [fix](build) Adapt Kuromoji dictionary tool and tests for branch-4.1 - `0c6a9ab5ed0562bd66035110850287e85285f4ad` [chore](be) Adapt Kuromoji code to branch static checks - `058a1d419658a302f658b45afd413961bca423f6` [fix](build) Scope gperftools stubs to the Kuromoji dictionary tool ### Release note Add the optional Kuromoji Japanese analyzer with `normal`, `search`, and `extended` modes for inverted indexes and `TOKENIZE`. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test (dictionary byte comparison and static checks below) - Behavior changed: - [x] Yes. Add an optional Japanese analyzer; existing parser behavior is retained. - Does this need documentation? - [x] Yes. Upstream documentation: https://github.com/apache/doris-website/pull/3946 ### Validation All commands ran in this branch's isolated worktree with JDK 17 and Linux ASAN builds. | Check | Result | | --- | --- | | `./build.sh --be --fe` | Passed. Dictionary converter and all four packaged dictionary files generated successfully. | | `./run-be-ut.sh --run --filter="Kuromoji*.*:DartsSmokeTest.*"` | 39/39 passed, including all 39 original source tests and the three real-dictionary tests; no skips. | | `./run-fe-ut.sh --run org.apache.doris.analysis.InvertedIndexKuromojiValidationTest,org.apache.doris.analysis.InvertedIndexPropertiesTest` | 33/33 passed, including all six source-added methods; no skips. | | `bash run-regression-test.sh --run -d inverted_index_p0,variant_p0/predefine -s test_japanese_analyzer,test_properties,test_variant_field_pattern_invalid_inverted_index -parallel 1` | 3/3 suites passed, zero failures/fatal scripts/skips, against the independently built local FE/BE. | | clang-format 16 and JDK 17 Maven Checkstyle | Passed for the complete candidate merged with the target branch. | | clang-tidy | Changed-line checks passed for changed C++ files. One unchanged baseline `be/src/core/types.h:577` unmatched `NOLINTEND` diagnostic was reproduced with the identical target header and separately recorded. | | English descriptions | Passed across the complete PR diff. Ten exact Japanese input/expected-data literals were individually reviewed; comments and diagnostics are English. | The four generated dictionaries are byte-identical before and after the C++ static-check adaptations. BE unit and cluster logs contain no ASAN, UBSan, or LeakSanitizer reports. No sanitizer suppression was added for this change. Three narrow static-check suppressions retain the Viterbi traversal, dictionary generation order, and shared dictionary-test fixture in one function each. The final tool-only compile definition has identical Linux preprocessed converter output with and without the definition. macOS ARM64 execution was not available on this Linux host. CI results are pending. ### Backport review Concrete `git range-diff` mapping against the original feature squash: ```text 1: 11171329d6d ! 1: 66e75a7b6f8 [feature](inverted-index) Add Japanese (Kuromoji) morphological analyzer (#64667) -: ----------- > 2: 8f2370e0b4f [fix](build) Adapt Kuromoji dictionary tool and tests for branch-4.1 -: ----------- > 3: 0c6a9ab5ed0 [chore](be) Adapt Kuromoji code to branch static checks -: ----------- > 4: 058a1d41965 [fix](build) Scope gperftools stubs to the Kuromoji dictionary tool ``` Differences from the source: - Preserve the target's BE unit-test link configuration and existing FE property tests while adding all source tests. - Omit duplicate `InvertedIndexProperties` constants/default-mode hunks and the matching utility alias already present from #68550; retain the target analyzer-validation API. - Keep target context around configuration declarations, third-party dependency lists, and test placement. These context differences do not change the added feature. - Apply the selected allocator repair, scope its stubs to the dictionary converter, and retain dictionary generation/install guards. - Translate descriptive comments and diagnostic labels into English while preserving Japanese data. Resolve the real-dictionary analyzer test through the same dictionary source path as its other tests. - Use equivalent C++ static-check syntax and remove one trailing blank license line. Original feature behavior and dictionary bytes are retained. Original-source hunk audit (a whole-file row is used only when the committed file is byte-identical to the source): | Source file / original hunk | Result | Evidence / adaptation | | --- | --- | --- | | `.clang-format-ignore` | Ported(verbatim) | Target: `.clang-format-ignore`. | | `.gitignore`<br>`@@ -154,6 +154,9 @@ docker/runtime/be/resource/apache-doris/` | Ported | Original added behavior retained with target context. | | `.licenserc.yaml`<br>`@@ -112,6 +112,7 @@ header:` | Ported | Original added behavior retained with target context. | | `LICENSE.txt`<br>`@@ -369,6 +369,14 @@ be/src/olap/skiplist.h : BSD-style license` | Ported | Original added behavior retained with target context. | | `NOTICE.txt` | Ported(verbatim) | Target: `NOTICE.txt`. | | `be/CMakeLists.txt`<br>`@@ -374,6 +374,26 @@ install(DIRECTORY` | Adapted | Retain dictionary generation/packaging; add the selected #67451 allocator fix and scope stubs to the converter with a private compile definition. | | `be/CMakeLists.txt`<br>`@@ -1099,6 +1119,49 @@ if (BUILD_META_TOOL OR BUILD_INDEX_TOOL)` | Adapted | Retain dictionary generation/packaging; add the selected #67451 allocator fix and scope stubs to the converter with a private compile definition. | | `be/dict/kuromoji/README.md` | Ported(verbatim) | Target: `be/dict/kuromoji/README.md`. | | `be/src/common/config.cpp`<br>`@@ -1420,6 +1420,8 @@ DEFINE_mInt64(snii_forced_spill_min_arena_bytes, "67108864");` | Ported | Original added behavior retained with target context. | | `be/src/common/config.h`<br>`@@ -1460,6 +1460,8 @@ DECLARE_mInt64(snii_forced_spill_min_arena_bytes);` | Ported | Original added behavior retained with target context. | | `be/src/exprs/function/function_tokenize.cpp`<br>`@@ -159,7 +159,7 @@ Status FunctionTokenize::execute_impl(FunctionContext* /*context*/, Block& block` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/analyzer/analyzer.cpp`<br>`@@ -39,6 +39,7 @@` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/analyzer/analyzer.cpp`<br>`@@ -88,7 +89,8 @@ bool InvertedIndexAnalyzer::is_builtin_analyzer(const std::string& analyzer_name` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/analyzer/analyzer.cpp`<br>`@@ -126,6 +128,22 @@ AnalyzerPtr InvertedIndexAnalyzer::create_builtin_analyzer(InvertedIndexParserTy` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/analyzer/kuromoji/KuromojiAnalyzer.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/KuromojiAnalyzer.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/KuromojiMode.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/KuromojiMode.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.cpp`<br>`@@ -0,0 +1,207 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/KuromojiTokenizer.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dict_format.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dict_format.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary.cpp`<br>`@@ -0,0 +1,325 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary.h`<br>`@@ -0,0 +1,140 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.cpp`<br>`@@ -0,0 +1,263 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_dictionary_builder.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.cpp`<br>`@@ -0,0 +1,301 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/dict/kuromoji_ipadic_parser.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize.h`. | | `be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.cpp`<br>`@@ -0,0 +1,301 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.h` | Ported(verbatim) | Target: `be/src/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi.h`. | | `be/src/storage/index/inverted/inverted_index_parser.cpp`<br>`@@ -39,6 +39,8 @@ std::string inverted_index_parser_type_to_string(InvertedIndexParserType parser_` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/inverted_index_parser.cpp`<br>`@@ -62,6 +64,8 @@ InvertedIndexParserType get_inverted_index_parser_type_from_string(const std::st` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/inverted_index_parser.cpp`<br>`@@ -89,8 +93,17 @@ std::string get_parser_mode_string_from_properties(` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/inverted_index_parser.h`<br>`@@ -44,7 +44,8 @@ enum class InvertedIndexParserType {` | Ported | Original added behavior retained with target context. | | `be/src/storage/index/inverted/inverted_index_parser.h`<br>`@@ -80,6 +81,10 @@ const std::string INVERTED_INDEX_PARSER_CHINESE = "chinese";` | Ported | Original added behavior retained with target context. | | `be/src/tools/kuromoji_build_dict.cpp`<br>`@@ -0,0 +1,202 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/CMakeLists.txt`<br>`@@ -161,6 +161,12 @@ if (TARGET phdr_cache_test_dso)` | Adapted | Add the source dictionary path definition while retaining target unit-test link settings. | | `be/test/storage/index/inverted/analyzer/kuromoji/darts_smoke_test.cpp`<br>`@@ -0,0 +1,70 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dict_format_test.cpp` | Ported(verbatim) | Target: `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dict_format_test.cpp`. | | `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dictionary_builder_test.cpp`<br>`@@ -0,0 +1,74 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_dictionary_test.cpp`<br>`@@ -0,0 +1,155 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_ipadic_parser_test.cpp`<br>`@@ -0,0 +1,109 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_normalize_test.cpp`<br>`@@ -0,0 +1,50 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_real_dict_test.cpp`<br>`@@ -0,0 +1,177 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. Resolve the analyzer dictionary through the same source path as the other real-dictionary tests. | | `be/test/storage/index/inverted/analyzer/kuromoji/kuromoji_viterbi_test.cpp`<br>`@@ -0,0 +1,256 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/storage/index/inverted/analyzer/kuromoji_analyzer_test.cpp` | Ported(verbatim) | Target: `be/test/storage/index/inverted/analyzer/kuromoji_analyzer_test.cpp`. | | `be/test/storage/index/inverted/analyzer/kuromoji_tokenizer_test.cpp`<br>`@@ -0,0 +1,39 @@` | Adapted | Preserve the source behavior and Japanese data; use English descriptions and target C++ static-check syntax where required. | | `be/test/storage/index/inverted/kuromoji_parser_type_test.cpp` | Ported(verbatim) | Target: `be/test/storage/index/inverted/kuromoji_parser_type_test.cpp`. | | `build.sh`<br>`@@ -606,6 +606,21 @@ update_submodule() {` | Ported | Original added behavior retained with target context. | | `dist/LICENSE-dist.txt`<br>`@@ -1532,3 +1532,5 @@ Other dependencies:` | Ported | Original added behavior retained with target context. | | `dist/licenses/LICENSE-darts-clone.txt` | Ported(verbatim) | Target: `dist/licenses/LICENSE-darts-clone.txt`. | | `dist/licenses/LICENSE-ipadic.txt`<br>`@@ -0,0 +1,71 @@` | Adapted | License text retained; remove only a trailing blank line required by the whitespace check. | | `fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java`<br>`@@ -35,11 +35,15 @@ public class InvertedIndexProperties {` | N-A | Already present from #68550 in target `InvertedIndexProperties`: `INVERTED_INDEX_PARSER_KUROMOJI`, its mode constants, and the parser/analyzer default of `search`. The target file has no new diff. | | `fe/fe-catalog/src/main/java/org/apache/doris/analysis/InvertedIndexProperties.java`<br>`@@ -81,13 +85,24 @@ public static String getInvertedIndexParserMode(Map<String, String> properties)` | N-A | Already present from #68550 in target `InvertedIndexProperties`: `INVERTED_INDEX_PARSER_KUROMOJI`, its mode constants, and the parser/analyzer default of `search`. The target file has no new diff. | | `fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java`<br>`@@ -54,6 +54,7 @@ public class InvertedIndexUtil {` | N-A | The Kuromoji parser alias already exists in the target from #68550; the remaining source validation changes are included below. | | `fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java`<br>`@@ -147,7 +148,8 @@ public static void checkInvertedIndexParser(String indexColName, PrimitiveType c` | Ported | Retain Kuromoji parser and mode validation with the existing target analyzer-validation API. | | `fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java`<br>`@@ -255,9 +257,10 @@ private static void checkInvertedIndexProperties(Map<String, String> properties,` | Ported | Retain Kuromoji parser and mode validation with the existing target analyzer-validation API. | | `fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java`<br>`@@ -271,8 +274,13 @@ private static void checkInvertedIndexProperties(Map<String, String> properties,` | Ported | Retain Kuromoji parser and mode validation with the existing target analyzer-validation API. | | `fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java`<br>`@@ -72,7 +72,7 @@ public class IndexPolicy implements Writable, GsonPostProcessable {` | Ported | Original added behavior retained with target context. | | `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexKuromojiValidationTest.java` | Ported(verbatim) | Target: `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexKuromojiValidationTest.java`. | | `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java`<br>`@@ -105,6 +105,40 @@ public void testGetParserModeDefaultForNonIk() {` | Adapted | Add all four source Kuromoji mode tests alongside the existing target tests. | | `regression-test/suites/inverted_index_p0/analyzer/test_japanese_analyzer.groovy`<br>`@@ -0,0 +1,114 @@` | Adapted | Preserve the complete test behavior and Japanese inputs; translate descriptive comments into English. | | `regression-test/suites/inverted_index_p0/test_properties.groovy`<br>`@@ -84,7 +84,7 @@ suite("test_properties", "p0"){` | Ported | Original added behavior retained with target context. | | `regression-test/suites/variant_p0/predefine/test_variant_field_pattern_invalid_inverted_index.groovy` | Ported(verbatim) | Target: `regression-test/suites/variant_p0/predefine/test_variant_field_pattern_invalid_inverted_index.groovy`. | | `thirdparty/build-thirdparty.sh`<br>`@@ -2282,6 +2282,18 @@ build_icu() {` | Ported | Original added behavior retained with target context. | | `thirdparty/build-thirdparty.sh`<br>`@@ -2568,6 +2580,7 @@ if [[ "${#packages[@]}" -eq 0 ]]; then` | Ported | Original added behavior retained with target context. | | `thirdparty/build-thirdparty.sh`<br>`@@ -2675,6 +2688,7 @@ cleanup_package_source() {` | Ported | Original added behavior retained with target context. | | `thirdparty/vars.sh`<br>`@@ -563,6 +563,12 @@ ICU_NAME=release-69-1.tar.gz` | Ported | Original added behavior retained with target context. | | `thirdparty/vars.sh`<br>`@@ -590,6 +596,7 @@ LANCE_C_MD5SUM="15ef7cd20a2e1606384251cb2d41d42f"` | Ported | Original added behavior retained with target context. | Selected Kuromoji follow-up: #67451 (`383be27d963c6bac3595e3528510348a65da0551`). | Source file / original hunk | Result | Evidence / adaptation | | --- | --- | --- | | `be/CMakeLists.txt`<br>`@@ -867,7 +867,8 @@ if ("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG" OR "${CMAKE_BUILD_TYPE}" STREQUAL "RE` | N-A | This source hunk only updates a comment in the unit-test allocator fallback, which the target does not have. | | `be/CMakeLists.txt`<br>`@@ -1100,7 +1101,25 @@ endif()` | Adapted | Apply the macOS ARM64 system-allocator override only to kuromoji_build_dict; retain dictionary generation and install guards. Use a private tool-only definition for the stubs. | | `be/CMakeLists.txt`<br>`@@ -1109,7 +1128,7 @@ if (NOT MAKE_TEST)` | Adapted | Apply the macOS ARM64 system-allocator override only to kuromoji_build_dict; retain dictionary generation and install guards. Use a private tool-only definition for the stubs. | | `be/src/storage/index/snii/encoding/crc32c.cpp`<br>`@@ -34,10 +34,15 @@` | N-A | Independent SNII CRC repair explicitly excluded by the user; no SNII file is changed. | | `be/test/testutil/gperftools_stubs.cpp`<br>`@@ -15,13 +15,14 @@` | Adapted | The target lacks the upstream unit-test allocator fallback. Add its four required stub definitions with a private KUROMOJI_DICT_TOOL guard; keep target unit-test linking unchanged. | ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
