airborne12 opened a new pull request, #67917:
URL: https://github.com/apache/doris/pull/67917
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Custom ngram tokenizers hard-code the allowed difference between `max_gram`
and `min_gram` to 1. This prevents valid wider ngram ranges while offering no
explicit override.
This PR adds the `max_ngram_diff` tokenizer property, keeps 1 as the
backward-compatible default, validates non-negative values in FE and BE, and
adds unit and regression coverage for a 1-to-8 tokenizer.
### Release note
Allow custom ngram tokenizers to configure the maximum difference between
`max_gram` and `min_gram` with `max_ngram_diff`.
### Check List (For Author)
- Test
- [ ] Regression test (coverage added; execution is pending CI)
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [ ] No.
- [x] Yes. Custom ngram tokenizers can opt into a wider gram-size range.
- Does this need documentation?
- [ ] No.
- [x] Yes. The new tokenizer property should be added to the custom
analyzer documentation.
Validation:
- `./build.sh --be -j48` (ASAN)
- `./run-be-ut.sh --run --filter='NGramTokenizerTest.*' -j48` (14 tests
passed)
- `./run-fe-ut.sh --run org.apache.doris.indexpolicy.PolicyValidatorTests`
(20 tests passed)
- `build-support/run-clang-tidy.sh --base origin/master --build-dir
be/build_ASAN`
- Full merge-tree clang-format and Checkstyle preflight
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
