SteNicholas opened a new pull request, #424:
URL: https://github.com/apache/paimon-cpp/pull/424
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #401
<!-- What is the purpose of the change -->
Java `org.apache.paimon.predicate.FullTextSearch` is `(fieldName, query,
limit)` plus a nullable
`includeRowIds`: `query` is a JSON DSL string passed unchanged to the native
engine, `limit` must
be positive and results are always scored. The C++ `FullTextSearch` used a
different model
(`SearchType`, an optional `limit`, `with_score`, `min_score` and
`pre_filter`), so a query written
for Java could not run in C++, and the `with_score` × `limit` matrix could
return unscored or
unlimited results that Java never produces.
This PR aligns the C++ API with Java and takes option 2 of #401 for
`lucene-fts`: remove the old
model and translate the DSL.
- **`FullTextSearch`**: takes a field name, a JSON DSL query, a positive
`limit` and optional
`include_row_ids`, which replaces `pre_filter` with Java `includeRowIds`
semantics. `SearchType`,
`with_score`, `min_score` and `ReplacePreFilter` are removed.
- **`FullTextSearch::OffsetRange(from, to)`**: as Java
`FullTextSearch#offsetRange`, clips
`include_row_ids` to a shard range and shifts them to shard-local row ids.
`OffsetGlobalIndexReader` uses it.
- **`GlobalIndexReader::VisitFullTextSearch`** returns
`ScoredGlobalIndexResult`. `UnionGlobalIndexReader`
merges scored results as for vector search, and the readers that reject
full-text search (BTree,
bitmap, lazy-filtered BTree, primary-key sorted, Lumina, file index
wrapper) are updated.
- **`full-text` reader**: passes the query to the engine unchanged and drops
the unscored and
`min_score` paths.
- **`lucene-fts` reader**: translates `match`, `multi_match`, `match_phrase`
(alias `phrase`) and
`boolean` into Lucene term, boolean and phrase queries. Field names,
aliases, defaults and
validation follow `core/src/query.rs` and `core/src/index.rs` of
apache/paimon-full-text:
operators, boolean occurrences and column names are trimmed of Unicode
whitespace as Rust
`str::trim` does, boosts are validated after rounding to `f32`, duplicate
known fields are
rejected and unknown fields are ignored. `boost` queries and fuzzy
matching are rejected with
`NotImplemented`, and prefix, wildcard and unlimited Lucene searches are
removed.
`limit` applies to each index shard. Searches across multiple shards return
the union of the shard
candidates, which can exceed `limit`; a final global top-k is tracked in
#402.
### Tests
<!-- List UT and IT cases to verify this change -->
UT:
- `src/paimon/common/predicate/full_text_search_test.cpp`:
`TestOffsetRangeWithoutIncludeRowIds`,
`TestOffsetRangeWithIncludeRowIds` (including row ids beyond 32 bits).
- `src/paimon/common/global_index/offset_global_index_reader_test.cpp`:
`TestVisitFullTextSearchWithOffset`,
`TestVisitFullTextSearchConvertsIncludeRowIds`.
- `src/paimon/common/global_index/union_global_index_reader_test.cpp`:
`TestVisitFullTextSearchUnion`.
- `src/paimon/global_index/full_text/full_text_global_index_test.cpp`:
`TestWriteAndSearch`
(`multi_match` on the native `text` field), `TestIncludeRowIdsAndLimit`,
`TestInvalidRead`.
- `src/paimon/global_index/lucene/lucene_global_index_test.cpp`:
`TestSimple` and
`TestSimpleChinese` (DSL queries, boosts, Unicode whitespace, scores and
`include_row_ids`),
`TestInvalidQuery` (invalid JSON, unknown query types, duplicate fields,
invalid values, boost
queries and fuzzy matching).
IT, `test/inte/global_index_test.cpp`:
`TestLuceneWriteCommitScanReadIndexWithScore`,
`TestWriteCommitScanReadLuceneIndexWithPartition`,
`TestFullTextWriteCommitScanReadIndex`,
`TestWriteCommitScanReadFullTextIndexWithPartition`.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
Yes, public API changes in `include/paimon/`:
- `FullTextSearch` (`include/paimon/predicate/full_text_search.h`): the
constructor is now
`FullTextSearch(field_name, query, limit, include_row_ids =
std::nullopt)`. `SearchType`,
`search_type`, `pre_filter`, `with_score`, `min_score` and
`ReplacePreFilter` are removed, `limit`
is `int32_t` instead of `std::optional<int32_t>`, and `OffsetRange` is
added.
- `GlobalIndexReader::VisitFullTextSearch`
(`include/paimon/global_index/global_index_reader.h`)
returns `Result<std::shared_ptr<ScoredGlobalIndexResult>>`.
`lucene-fts` queries must be written in the JSON DSL. No storage format
change.
### Documentation
<!-- Does this change introduce a new feature -->
Yes. `docs/source/user_guide/global_index.rst` describes the JSON DSL
queries of the `full-text`
and `lucene-fts` indexes, how `lucene-fts` translates and validates them,
and the per-shard
`limit`.
### Generative AI tooling
Generated-by: Claude Code 2.1.292 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
