airborne12 opened a new pull request, #69054:
URL: https://github.com/apache/doris/pull/69054
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
`SELECT tokenize('Apache Doris', '"analyzer"="STANDARD"')` fails with
`Policy not found with name: STANDARD`, while the equivalent `parser=STANDARD`
succeeds. Built-in analyzer dispatch compares names case-sensitively, so
uppercase and mixed-case spellings reach policy lookup.
Pass the analyzer configuration into policy lookup and resolve a built-in
case variant only when no exact policy exists. Keep this decision under the
policy manager's read lock and preserve parser mode, lowercase and stopword
settings. Canonical lowercase built-ins keep their existing precedence; exact
legacy policies keep their binding, including validation errors for incorrect
policy types. Stored index selection and provider dispatch are unchanged.
Add unit tests for uppercase and mixed-case built-ins, analyzer options,
exact-name policy collisions, incorrect policy types, removal followed by
built-in fallback, and ordinary custom-policy lookup.
### Release note
Fix uppercase and mixed-case built-in analyzer names in `tokenize()` while
preserving exact legacy policy bindings.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Reproduced the original error with a failing unit test before the fix.
After the fix, all 67 tests in the four related suites passed under ASAN:
```sh
GLIBC_COMPATIBILITY=OFF bash run-be-ut.sh --run \
--filter='FunctionTokenizeTest.*:AnalyzerTest.*:InvertedIndexParserTest.*:IndexPolicyMgrTest.*'
-j 8
```
The local UT build disables Doris's additional glibc compatibility library
because the installed toolchain already supplies it; otherwise the test
executable fails to link with duplicate symbols. No production build
configuration is changed.
Full `build-support/check-format.sh` (clang-format 16),
`build-support/check-build-hygiene.sh`, and `git diff --check` passed.
Working-tree English-description scanning passed.
`build-support/run-clang-tidy.sh` was attempted but is blocked by the
pre-existing unmatched `NOLINTEND` in `be/src/core/types.h:576`.
- Behavior changed:
- [ ] No.
- [x] Yes. Built-in analyzer case variants work when there is no exact
policy; exact policy bindings remain authoritative.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
