airborne12 opened a new pull request, #70264:
URL: https://github.com/apache/doris/pull/70264
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70263 (T4), whose commits show in this diff until
they merge; review only this PR's own commits: `23d1e3674b6` [improvement](be)
Tokenize english and basic with a vectorized byte classifier. It stays a draft
until its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70263
Problem Summary:
The english parser still ran CLucene's SimpleTokenizer<char>
behind LuceneBackedTextAnalyzer, and the basic tokenizer walked the value
byte
by byte into a token list. Both now classify the value 64 bytes at a time and
cut it at the edges of the letter-and-digit bits.
- tokenizer/byte_classifier.h: classify_block() turns 64 bytes into a bitmap
of ASCII letters and digits, a bitmap of the bytes that can start a
multi-byte character and, when lowercasing, a copy with A-Z folded. It is a
plain loop the compiler vectorizes on every target, the byte masks turned
into bits by simd::bytes32_mask_to_bits32_mask as the Doris filters turn
theirs. classify_tail() copies the last bytes through a stack buffer, so
nothing past the value is read.
- simd::bytes32_mask_to_bits32_mask gets a NEON branch; aarch64 took its
scalar loop. It compiles for aarch64 but has not run: no aarch64 host or
emulator was at hand, so BitsTest still has to pass on an ARM machine.
- EnglishTokenizer replaces SimpleAnalyzer<char> for the english parser and
an
unknown one: the runs of ASCII letters and digits, cut every 255 bytes,
lowercased into a copy that keeps the offsets of the value, or views of the
value without lower_case. The chain has no stop words: SimpleAnalyzer<char>
never read the ones set_stopwords() gave it.
- BasicTokenizer runs the same block loop and decodes a character only at a
byte that can start one, the only bytes the scalar loop made a token of
besides the runs; a run is cut to 255 bytes and the rest dropped, as
before.
It lowercases its own tokens, so the builtin basic parser drops its
lowercase filter, which changed nothing else in them: the CJK characters it
keeps have no case. With extra_chars it keeps the scalar loop.
- normalize() lowercases the text of a builtin parser with the same ASCII
function the tokenizers use. The std::tolower it replaces changed only A-Z
as well: the BE never calls setlocale.
- rowid_read_ahead.cpp includes storage/iterators.h for StorageReadOptions;
it
compiled only through another file of its unity batch, and the new source
moved it to a batch without that file.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- BE UT under ASAN: the analysis suites with the new ones, 3396 of 3402
tests pass; the 6 skipped are the opt-in code point scan, four gram
report tests and the vector width check, which runs in the build
without AVX2 below. The wider set of analysis, index and search suites:
4152 of 4159 pass, 7 skipped. The code point scan passes with
ANALYSIS_GOLDEN_FULL=1. New tests:
ByteClassifierTest (every byte value at every position, random blocks,
every tail length, a guard page after the value), EnglishTokenizerTest
(CLucene's SimpleAnalyzer<char> run beside it on random bytes, block
edges, the 255-byte cuts, batches around 256 tokens, views without
lowercase, a guard page), BasicTokenizerTest (the same against the
scalar loop), BitsTest for the mask conversion, and ComponentStreamTest
for the buffers both release after a long value. After the last change,
push_chinese_char forced inline, the tokenizer, component and golden
suites ran again: 60 pass, 2 skipped.
- The analysis golden files are unchanged, and the benchmark's digest
mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows
of
the six corpora and fourteen analyzers.
- -Rpass=loop-vectorize with the RELEASE flags: the classification loop
is vectorized at width 32 with AVX2 and at width 16 with SSE4.2 alone,
in all four instantiations (a block or a tail, with or without the
lowered copy).
- A second ASAN UT build with USE_AVX2=0 in its own CMAKE_BUILD_DIR
(USE_AVX2=0 in its CMakeCache.txt) runs the same 3402 tests: 3397 pass,
ByteClassifierTest.BuiltWithTheExpectedVectorWidth among them, which
asserts that __AVX2__ is not defined there.
- Throughput of the benchmark binary of this commit, with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each of six corpora and their geometric mean.
Against the T1 baseline binary, reuse/ (one stream for all values) and
row/ (a stream per value): english 3.78 and 3.12, no corpus below 1.80
(row/http_request); basic 7.42 and 7.74, no corpus below 4.27
(reuse/ja_wiki); custom basic+lowercase 2.40 and 2.37. Against T4:
english 4.81 and 3.91, basic 4.05 and 4.45, custom basic+lowercase 1.31
and 1.38, and standard, which this change does not touch, 1.00 and
0.99. push_chinese_char is forced inline: out of line, its call for
every lead byte made basic 5-8% slower on ja_wiki and zh_wiki. A binary
with the same machine code for both tokenizers at other alignments
differs by up to 8% on one corpus of row/english (geomean 0.96), so a
difference of that size is code layout.
- check-build-hygiene.sh passes with the CLucene exception tables
unchanged; clang-format 16 (run_clang_format.py) leaves the changed
files as they are; run-clang-tidy.sh reports nothing on the changed
lines.
- Behavior changed:
- [x] No.
- [ ] Yes.
Every token of english and basic is the same as before (above).
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
