SteNicholas opened a new pull request, #431:
URL: https://github.com/apache/paimon-cpp/pull/431
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #402
<!-- What is the purpose of the change -->
`UnionGlobalIndexReader::VisitFullTextSearch` ORed shard results, so a
full-text search with
`limit` k over multiple index shards could return up to shards × k rows, and
it failed when two
shards returned the same row id. `OffsetGlobalIndexReader` shifted global
`include_row_ids` by the
shard offset without clipping them, so row ids before a shard became
negative local ids, row ids
past it were kept, and a shard with no included row was still searched.
This PR follows Java `DataEvolutionFullTextRead`, which asks each split for
`limit` rows, combines
them with `ScoredGlobalIndexResult#merge` and applies `#topK(limit)` last.
- **`UnionGlobalIndexReader::VisitFullTextSearch`**: merges shard results,
where a duplicate row id
keeps its first score, and keeps the `limit` highest-scoring rows. Scores
are compared as Java
`Double#compare` does (via `FieldsComparator::CompareFloatingPoint`), and
ties among merged rows
are broken by ascending row id. A null search or a non-positive `limit` is
rejected before any
shard is searched.
- **`OffsetGlobalIndexReader`**: takes the last row id of its shard and
clips `include_row_ids` to
the shard with `FullTextSearch::OffsetRange(offset, to)`, as Java
`OffsetGlobalIndexReader(wrapped, offset, to)` does. A shard whose include
set becomes empty is
not searched.
- **Overlapping index ranges**: as Java
`DataEvolutionFullTextScan#chooseIndexRanges`,
`GlobalIndexScanImpl` ranks a row covered by overlapping ranges of one
index type only in the
first of them in `(from, to)` order. Each `OffsetGlobalIndexReader` is
given the last row ranked
by earlier ranges (`skip_to`) and ranks only the rows after it, so the top
`limit` rows of each
shard include all of its rows that score above the global `limit`-th
score. Passing the last
skipped row instead of the first row to rank does not overflow when a
range ends at `INT64_MAX`.
A fully covered shard is not searched but still rejects a non-positive
`limit`.
As in Java, each shard's index engine decides which of its rows tied at its
cutoff score it
returns, so ascending row id order is guaranteed only among the rows the
shards return.
### Tests
<!-- List UT and IT cases to verify this change -->
UT:
- `src/paimon/common/global_index/union_global_index_reader_test.cpp`:
`TestVisitFullTextSearchUnion`,
`TestVisitFullTextSearchKeepsTopKAcrossReaders` (top-k and ties
by row id), `TestVisitFullTextSearchDuplicateRowIdKeepsFirstScore`,
`TestVisitFullTextSearchOrdersSpecialScoresAsJava` (NaN, infinity and
signed zeros),
`TestVisitFullTextSearchAllNullptr`, `TestVisitFullTextSearchErrors`.
- `src/paimon/common/global_index/offset_global_index_reader_test.cpp`:
`TestVisitFullTextSearchConvertsIncludeRowIds` (clipping to the shard, and
an empty include set
that is not searched), `TestVisitFullTextSearchRejectsNonPositiveLimit`,
`TestVisitFullTextSearchSkipsRows`,
`TestVisitFullTextSearchEndingAtLargestRowId`. The other
tests pass the shard end.
IT, `test/inte/global_index_test.cpp`:
- `TestWriteCommitScanReadLuceneIndexWithPartition` and
`TestWriteCommitScanReadFullTextIndexWithPartition`: global top-k for
every `limit` across two
shards, with and without `include_row_ids`.
- `TestLuceneSearchOverlappingIndexRanges`: index ranges `[0, 2]`, `[1, 1]`
and `[1, 3]`, where the
best match is covered only by the tail of `[1, 3]`, also with
`include_row_ids` and a
`RowRangeIndex` that prunes `[0, 2]`.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No signature change in `include/`. The documented behavior changes:
- `GlobalIndexReader::VisitFullTextSearch`
(`include/paimon/global_index/global_index_reader.h`)
returns at most `limit` rows across all index shards.
- `FullTextSearch::limit` (`include/paimon/predicate/full_text_search.h`) is
the maximum number of
rows returned; each index shard is asked for that many rows before the
merge.
No storage format change.
### Documentation
<!-- Does this change introduce a new feature -->
Yes. `docs/source/user_guide/global_index.rst` replaces the note that
searches across shards return
the union of shard candidates with the global top-k behavior, tie handling,
overlapping index
ranges and `include_row_ids` clipping.
### Generative AI tooling
Generated-by: Claude Code 2.1.296 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
