airborne12 opened a new pull request, #70263:
URL: https://github.com/apache/doris/pull/70263
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70262 (T3), whose commits show in this diff until
they merge; review only this PR's own commits: `d90ab96380b` [refactor](be)
Move every analysis component onto the CLucene-free stream interface. It stays
a draft until its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70262
Problem Summary:
The SNII text analysis is moving off CLucene. The two previous
changes added the TextAnalyzer interface and moved every consumer onto it,
with
the analyzers still CLucene ones behind LuceneBackedTextAnalyzer. This change
moves the components themselves, all in one step because the two sets of base
classes cannot be mixed in one chain: the 11 registered tokenizers (keyword,
empty, standard, ngram, edge_ngram, char_group, basic, icu, pinyin, ik_smart,
ik_max_word) and kuromoji, the 6 token filters, the 3 char filters, their
factories and the analyzers built from them. english, standard, unicode and
chinese are CLucene analyzers, not Doris components, and stay behind
LuceneBackedTextAnalyzer until their own tasks.
- TokenizerFactory::create() const returns a TextTokenizer,
TokenFilterFactory::create() const takes and returns a TextTokenStream, and
CharFilterFactory::create() const returns a TextCharFilter, a text-to-text
function run in reset() instead of a CLucene Reader.
- ChainAnalyzer is char filters (CharFilteredStream), a tokenizer and token
filters. A custom analyzer and a custom normalizer are ChainAnalyzers, and
create_builtin_analyzer() builds the basic, icu, ik and kuromoji parsers as
preset chains with the same lowercasing and dictionaries as before. The
index
char filter is a CharFilteredAnalyzer around whatever the index analyzes
with;
create_reader() is gone and filter_characters() runs the native
char_replace.
- Components that make one token at a time derive from PerTokenTokenizer and
PerTokenFilter and keep their state machines. Terms that stay valid until
the
next value, views of it or per-value storage, go into the batch without a
copy.
- The source byte offsets added with #67918 are removed: nothing outside the
analysis layer read them. Their only effect on terms, strict handling of
invalid UTF-8 below a pinyin filter with ignore_pinyin_offset=false, stays
as
set_strict_utf8(): icu, ngram and edge_ngram throw, ik folds rune by rune.
PositionCapability had no reader either and goes.
- Every component reports what it keeps from one value to the next
(reserved_bytes()) and releases buffers above 64 KiB at the next value.
basic,
ik, pinyin and kuromoji kept a long value's buffers until the stream died,
which was harmless with a stream per value and is not with the reused
stream
of the SNII writer.
- The IK core reads the value through a TextCursor (the same 4096-byte
window and
read semantics), throws doris::Exception instead of CLuceneError, and
decodes
runes with analysis/common/rune.h, jieba's rules byte for byte, instead of
the
CLucene submodule's Unicode.hpp. StandardTokenizerImpl reads through the
same
cursor. The ik tokenizer factory builds its Configuration once instead of
for
every stream.
- Deleted: analysis/reader.h (ReaderPtr moves to clucene/lucene_reader.h for
the
V1/V2 writer), token_stream.h, DorisTokenizer, DorisTokenFilter,
DorisCharFilter, LuceneGramTokenizer, and the BasicAnalyzer, ICUAnalyzer,
IKAnalyzer and KuromojiAnalyzer classes. match.cpp had CLucene's _CLFINALLY
only through the old headers; it uses Defer.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- ./run-be-ut.sh --run with T3's suites plus the component, query engine,
kuromoji, ik and icu suites passes on ASAN: 4121 tests, 6 skipped.
- The new component_stream_test runs every tokenizer, filter and char
filter through the stream contract (kuromoji in kuromoji_viterbi_test,
on the dictionary that test builds); checks that after a 2 MiB value
and
a short one each keeps at most 1 MiB, exactly what its members hold
(read
with -fno-access-control); checks strict UTF-8 (icu, ngram and
edge_ngram
throw, ik folds rune by rune, a pinyin filter with ignore_pinyin_offset
false turns it on upstream, the other tokenizers ignore it); and checks
that a value NFKC_CF folds to nothing makes no token.
- analysis_golden.txt and ann_write_golden.txt are unchanged, and
ANALYSIS_GOLDEN_FULL=1 (every code point through every configuration)
passes. The benchmark's digest mode gives the same tokens as the T1
baseline binary in all 5,240,060 rows of the six corpora and fourteen
analyzers, kuromoji on the real dictionary included.
- Throughput with be/benchmark/analyzer_ab.py: five alternating rounds,
the median of the per-round ratios on each of six corpora and their
geometric mean; an A/A of the T3 binary stays within 0.98 to 1.03.
Against T3, reuse/ (one stream for all values) and row/ (a stream per
value): basic 2.49 and 2.01, custom basic+lowercase 2.48 and 2.00,
keyword+lowercase 1.88 and 1.54, standard+asciifolding+lowercase 1.57
and 1.24, standard+lowercase 1.76 and 1.51,
standard+word_delimiter+lowercase 2.10 and 1.70, icu 1.35 and 1.18,
ik_max_word 1.17 and 1.02, ik_smart 1.17 and 1.01, kuromoji 1.01 and
1.01. Against the T1 baseline binary: basic 1.82 and 1.73, custom basic
1.83 and 1.72, keyword 1.15 and 1.40, standard+asciifolding 1.23 and
1.09, standard 1.41 and 1.34, standard+word_delimiter 1.79 and 1.57,
icu 1.14 and 1.12, ik_max_word and ik_smart 0.99 and 0.99, kuromoji
0.99
and 0.99; the lowest single corpus is 0.96 (ik on agent_input).
- ./run-regression-test.sh --run -d inverted_index_p0 on an ASAN cluster
of this build with debug points: 249 of 252 suites pass. Two hit the
FE's limit of 100 index policies, which the earlier suites fill, and
test_drop_index_with_format_v2 lists the index file as soon as the FE
alter job finishes, while the BE received the drop-index task 23 and 28
seconds later in two runs. All three pass on a fresh cluster, and the
BE
logged no sanitizer error.
- build-support/check-build-hygiene.sh passes with the CLucene exception
tables down from 29 + 46 to 13 + 12 entries; analyzer.cpp is the only
file of analysis/ left in them. build-support/run-clang-tidy.sh reports
no warning on the changed lines.
- The rewritten tests drop only assertions on deleted features: token
offsets and source byte offsets (keyword 6, ngram 2, icu tokenizer 8,
pinyin tokenizer 50, ik 2, lowercase 4, icu normalizer filter 7, icu
normalizer char filter 7 macros, pinyin filter offsets in 27 tests and
its provenance accessors in 10), the token type of lowercase (2,
TokenRef has none), and char_filter_test.cpp, whose 3 assertions tested
the reader methods of the deleted DorisCharFilter.
- Behavior changed:
- [x] No.
- [ ] Yes.
The terms and positions of every analyzer are unchanged (above). An
analyzer error now comes from the component as a doris::Exception instead of
through the CLuceneError conversion, with the same
INVERTED_INDEX_ANALYZER_ERROR code and a slightly different message.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
