JingsongLi opened a new pull request, #8074: URL: https://github.com/apache/paimon/pull/8074
## Summary This PR adds configurable tokenizer support for Tantivy full-text global indexes, including ngram tokenization for short fragment matching and Jieba tokenization for Chinese text. ## Changes - Add persisted Tantivy full-text tokenizer options for `default`, `ngram`, and `jieba`, including ngram gram bounds, prefix-only mode, and lower-case behavior. - Wire tokenizer options through Java global index creation, JNI writer/searcher APIs, Rust Tantivy analyzers, and metadata serialization. - Teach PyPaimon to read tokenizer metadata, register the ngram analyzer, and query Jieba-tokenized indexes through Python `jieba` when installed. - Extend Java/Python interoperability and tokenizer option coverage, including ngram and Jieba scenarios. - Document tokenizer configuration in global index, PyPaimon, CLI, Flink, Spark, and Tantivy README docs. ## Testing - `cargo check` in `paimon-tantivy/paimon-tantivy-jni/rust` - `mvn -pl paimon-tantivy/paimon-tantivy-index -am -Pfast-build -DfailIfNoTests=false -Dflink.forkCount=0 -Dtest=TantivyFullTextIndexOptionsTest test -q` - `PYTHONPATH=. /Users/lijingsong/.venvs/paimon-py311/bin/pytest pypaimon/tests/vector_search_filter_test.py::TantivyFullTextIndexOptionsTest -q` - `/Users/lijingsong/.venvs/paimon-py311/bin/flake8 --config=./dev/cfg.ini pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py pypaimon/tests/e2e/java_py_read_write_test.py pypaimon/tests/vector_search_filter_test.py` - `git diff --check origin/master..HEAD` ## Notes - PyPaimon Jieba query support requires the optional Python `jieba` package. It is not added as a hard dependency. - A full `./dev/lint-python.sh -i flake8` run was attempted, but the local process was killed by the OS (`Killed: 9`) before producing lint findings; the Python files changed by this PR pass flake8 directly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
