airborne12 opened a new pull request, #70265:
URL: https://github.com/apache/doris/pull/70265
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70264 (T5), whose commits show in this diff until
they merge; review only this PR's own commits: `50041fb391b` [improvement](be)
Tokenize standard and unicode with a dense UAX#29 table. It stays a draft until
its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70264
Problem Summary:
The builtin standard and unicode parsers still ran CLucene's
standard95 StandardTokenizer behind LuceneBackedTextAnalyzer, and the custom
standard tokenizer ran the JFlex scanner StandardTokenizerImpl one code point
at a time through a buffer it refills and shifts. Both now run one flat table
of the same DFA over the value.
- tokenizer/standard/uax29_table.{h,cpp}: Uax29Table lays the scanner's ZZ_*
tables out at first use: 59 states by 29 classes with one byte per
transition, and a compact copy of the two-level character map, so that a
step is one load and no table is copied by hand. CLucene's standard95
scanner has the same tables entry for entry. longest_match() is the JFlex
longest match, over a window its caller defines.
- tokenizer/utf8_validator.h: the UTF-8 rule of the builtin parsers, ported
from CLucene's StringUtil::validate_utf8, which accepts overlong forms, and
the decoder of CLucene's scanner that goes with it.
- Uax29Tokenizer, the builtin standard and unicode parsers: a value strict
UTF-8 accepts (simdutf) passes, the lenient rule decides the others, and a
value it refuses has no token, as before. The window (a token is scanned
while it is shorter than 255 bytes), the drop of a longer token without a
gap, the ASCII lowercase of WORD tokens and the 33 stop words, also only
for WORD tokens, follow the old tokenizer. Lowercasing folds the whole
value
into a copy and scans the copy: A-Z and a-z are one class of the DFA and
every accepting state reached through a letter is WORD, so the tokens are
the same; Uax29TableTest checks both on the table. Stop words are looked up
as packed integers after a length test, not in an unordered_set.
- StandardTokenizer, the custom standard tokenizer: a valid UTF-8 value runs
through the same table with the scanner's window, max_token_length bytes
from each token start: a character that does not fit ends the token, and
one that does not fit an empty window ends the value, as the scanner's
refill did. Its terms are views of the value. Any other value still runs
through StandardTokenizerImpl, whose reaction to invalid bytes depends on
where its buffer refills; those terms are copied, since a refill moves the
buffer.
- analyzer.cpp builds the standard and unicode parsers on Uax29Tokenizer;
lucene_builtin_analyzer stays for chinese.
- storage/utils.cpp no longer opens ErrorCode for whatever follows it in a
unity batch: the new sources moved it into a batch with version_graph.cpp,
and gen_cpp/types.pb.h, first included there, then saw two LIMIT_REACH.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- BE UT under ASAN: the analysis, index and search suites with the T5 and
T6 ones, 4177 of 4184 tests pass; the 7 skipped are opt-in reports and
generators, the opt-in code point scan and the vector width check of
the
build without AVX2. The code point scan passes with
ANALYSIS_GOLDEN_FULL=1. AnalysisDifferential, which runs both standard
parsers beside standard95 and the custom tokenizer beside
StandardTokenizerImpl for max_token_length 1, 2, 3, 4, 5, 16, 255, 256,
1000 and 16383, on the golden families and on random lines, passes.
New tests: Uax29TableTest (the table against ZZ_* entry by entry, the
class of every code point, every first character accepted, the proof
that lowercasing the whole value changes only WORD tokens),
Utf8ValidatorTest (CLucene's validate_utf8 on every value of up to
three
bytes and on random bytes, the decoder against CLucene's
string_to_wstring on every accepted sequence), Uax29TokenizerTest
(standard95 beside it on random values for every lower_case and stop
words setting, the 255-byte window and drop, overlong and invalid
values, the 33 stop words with their prefixes, suffixes and case
variants, batches around 256 tokens, a guard page) and
CustomStandardTokenizerTest (the scanner beside it on random values for
the same ten max_token_length values, the cut at the window, the end of
the value at a character longer than the window, the copied terms of an
invalid value with max_token_length=8, the views of a valid one).
LuceneBackedTextAnalyzerTest now wraps CLucene's standard analyzer
itself instead of the standard parser.
- The analysis golden files are unchanged, and the benchmark's digest
mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows
of
the six corpora and fourteen analyzers, the builtin standard parser and
three custom standard chains among them.
- Throughput of the benchmark binary of this change, with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each of six corpora and their geometric mean.
Against the T1 baseline binary, reuse/ (one stream for all values) and
row/ (a stream per value): the builtin standard parser 1.82 and 1.79,
no
corpus below 1.52 (prose_en); unicode is the same analyzer. The custom
standard chains, whose lowercase filter is still per token: standard +
lowercase 1.84 and 1.78, + asciifolding + lowercase 1.51 and 1.33,
+ word_delimiter + lowercase 2.06 and 1.78. Against T5: standard 2.13
and 2.06 (it no longer goes through the CLucene adapter), the custom
chains 1.13 to 1.32.
- check-build-hygiene.sh passes with the CLucene exception tables
unchanged (analyzer.cpp keeps CLucene for chinese); clang-format 16
(run_clang_format.py) leaves the changed files as they are;
run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed:
- [x] No.
- [ ] Yes.
Every token of standard, unicode and the custom standard tokenizer
is the same as before (above).
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
