airborne12 opened a new pull request, #70268:
URL: https://github.com/apache/doris/pull/70268
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70267 (T8), whose commits show in this diff until
they merge; review only this PR's own commits: `53b3cdc2ec0` [improvement](be)
Match the ik dictionaries through a flat trie. It stays a draft until its
dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70267
Problem Summary:
The ik tokenizer walked dictionaries of DictSegment nodes,
each a map or array of child pointers, reached them through
Dictionary::getSingleton() for every character, asked a chain of range tests
for each character's type, regularized every word, kept its candidate hits in
a std::list and its results in a std::queue rebuilt for each value, and built
a std::string for every word it handed out. The tokens do not change.
- dic/FlatDict.h: the dictionaries as breadth-first flat tries of 12-byte
nodes, children sorted (scanned up to 8, bisected above), with a table of
0x10000 entries for the root's children. match() and isWord() keep the hit
semantics of the old walk. DictSegment is now a load-time builder: each
loader fills one, turns it into a FlatDict and frees it, and a load is
still
published only after it succeeds, so a failed load can be tried again. The
FlatDict constructor lives in DictSegment.cpp, since a new source file
would
move the Storage unity batches. dic/Hit.h holds (const FlatDict*, node).
- core/CharacterUtil: identifyCharType reads a table of 0x10000 entries built
once for the BMP, the block rules above it; a character's type is taken
once, on the regularized character; regularizeInPlace returns a word with
no
0xE3 or 0xEF byte and no ASCII capital untouched, and regularizeString
wraps
it with its old signature.
- core/IKSegmenter, AnalyzeContext, CJKSegmenter, CN_QuantifierSegmenter: the
Dictionary is taken once by reference instead of through getSingleton() on
every character; the four segmenters are members called in the old order;
an idle quantifier segmenter only releases its lock on a character that is
not Chinese, and the surrogate pair segmenter only runs on surrogate pairs;
CJKSegmenter's hits are a vector compacted in place in the same order; the
results are one vector reused across windows and values; words are handed
out as views, and Lexeme keeps no string.
- IKTokenizer: the words of a value lie back to back in one buffer reused for
every value and are lowercased there, in lenient and strict mode alike; a
value that does not decode still throws at reset().
Kept as they decide tokens: charEnd is the current index after matchWithHit
and the index plus one after matchInMainDict, which the quantifier path reads
as its next index; the 4096-byte windows with BUFF_EXHAUST_CRITICAL; the
255-byte cut at a character boundary; lowercase always on for dictionary
matching, so "Hello ＡＢＣ" with lower_case=false still gives hello and ABC.
An IKTokenizer or IKSegmenter now needs a loaded dictionary when it is made,
not at its first Chinese character; the factory and the builtin preset load
it first.
### Release note
The ik parser tokenizes about 1.8 times as fast, and its dictionaries take
about a tenth of the memory.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- BE UT under ASAN: the analysis, index and search suites with the T5 to
T9 ones, 4213 of 4220 tests pass and the 7 skipped are the opt-in ones;
the code point scan passes with ANALYSIS_GOLDEN_FULL=1. New:
IKFlatDictTest (FlatDict against DictSegment for every word, prefix and
suffix of main.dic, quantifier.dic, stopword.dic and
extra_stopword.dic,
every step and length compared; 20k random strings against a trie with
supplementary-plane words and nodes of more than 8 children; no
DictSegment left after Dictionary::initial; a tokenizer keeps working
with the singleton gone, so the per-character path does not reach it)
and IKCharacterUtilTest (the type table against the block rules on
every
rune up to 0x1FFFFF; regularizeString against the old implementation on
300k random byte strings). ik_anayzer_test gains "Hello ＡＢＣ" (hello,
ABC without lowercase; hello, abc with it), a character on the
4096-byte
boundary at offsets 4088 to 4100 for é, 中 and 😀, and 你好\xE4\xB8
throwing. IndexFileWriterTest.CacheEvictionTest failed once in a run of
the wider suite list and passes on its own and in the later runs: it
depends on what earlier tests leave in a shared searcher cache.
- The analysis golden files are unchanged, and the benchmark's digest
mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows,
ik_smart and ik_max_word among them.
- The new code was also compared with the old ik code directly, outside
the BE build: 28,659,978 values in 8 modes (ik_smart and ik_max_word,
lower_case on and off, strict and lenient), through kept and new
streams: every dictionary line alone and joined, random strings with
invalid bytes, every offset around the 4096-byte boundary and all six
corpora in full. 523M tokens and 211,200 identical exceptions, 0
differences; an ASAN/UBSan run of 7.5M values found none, and no
sanitizer report. In that harness, measured as thread CPU time with old
and new taking turns on each 128 KiB chunk (the host load reached about
410, which made wall time useless; old against old gave 0.97 to 1.00),
the new code is 1.71 to 1.92 times as fast on every corpus in both
modes, and the loaded dictionaries take 6.0 MB instead of 64.2 MB.
- Throughput of the benchmark binary of this change with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each corpus and their geometric mean, reuse/ and
row/, at a host load of 50 to 135. Against the T1 baseline: ik_max_word
1.84 and 1.74 (zh_wiki 1.58 and 1.54), ik_smart 1.80 and 1.70 (zh_wiki
1.50 and 1.42), no corpus below 1.42. Against T8: ik_max_word 1.93 and
1.76, ik_smart 1.89 and 1.75.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed:
- [x] No.
- [ ] Yes.
Every token of ik_smart and ik_max_word is the same as before
(above).
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
