airborne12 opened a new pull request, #67918:
URL: https://github.com/apache/doris/pull/67918
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Doris already contains IK segmentation support, but custom inverted-index
analyzers cannot select it as a tokenizer or compose it with token filters.
This PR exposes `ik_smart` and `ik_max_word` as built-in custom-analyzer
tokenizers. It adapts the existing IK tokenizer to the custom token-stream
interface, preserves byte offsets, registers both modes in FE and BE, and
verifies composition with the pinyin token filter.
### Release note
Add `ik_smart` and `ik_max_word` tokenizer types for custom inverted-index
analyzers.
### Check List (For Author)
- Test
- [ ] Regression test (coverage added; execution is pending CI)
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [ ] No.
- [x] Yes. Custom analyzers can use IK smart or maximum-word
segmentation and compose it with token filters.
- Does this need documentation?
- [ ] No.
- [x] Yes. The two new built-in tokenizer names should be added to the
custom analyzer documentation.
Validation:
- `./build.sh --be -j48` (ASAN)
- `./run-be-ut.sh --run
--filter='AnalysisFactoryMgrTest.*:IKTokenizerTest.*:PinyinFilterTest.*' -j48`
(71 tests passed)
- `./run-fe-ut.sh --run org.apache.doris.indexpolicy.PolicyValidatorTests`
(18 tests passed)
- clang-tidy on all changed C++ files with the ASAN unit-test compilation
database
- C++ format and build-hygiene checks
The Chinese literals in the tests are tokenizer input and expected token
data, not code descriptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
