SteNicholas opened a new pull request, #448:
URL: https://github.com/apache/paimon-cpp/pull/448
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #403
<!-- What is the purpose of the change -->
Deletion vectors were applied only when rows were read, so a full-text
search on a data-evolution
table with `deletion-vectors.enabled` still ranked deleted rows. They could
take every top-k slot,
and the read then dropped them: a search with `limit` k returned fewer than
k rows, or none, even
when enough live rows matched.
The fix aligns with Java `GlobalIndexLiveRowFilter` and
`DataEvolutionFullTextRead#includeRowIds`:
the rows live in the snapshot the index is read from are computed once and
intersected into the
rows each index shard ranks, while BM25 statistics still come from the whole
index.
- **`GlobalIndexLiveRowFilter::LiveRows`**: as Java
`GlobalIndexLiveRowFilter#liveRows` does, reads
the data files of the scanned partitions in the pinned snapshot with
`ScanMode::ALL`, unions
their row id ranges, then subtracts the rows deleted by their deletion
vectors. Every range is
added before any deletion vector is subtracted, so a deleted row is not
added back by another
file covering the same rows, and only one deletion vector is held at a
time. It returns nullopt,
so every row counts as live, when deletion vectors or row tracking are
disabled. A deletion
vector that cannot be read is reported with its data file, snapshot and
table.
- **`DataFileMeta::NonNullRowIdRange`**: computes a file's row id range as
Java
`DataFileMeta#nonNullRowIdRange` does, and rejects a null or negative
first row id, a file with
no rows, or a range that would pass `INT64_MAX`, instead of overflowing.
As in Java,
`DataEvolutionSplitGenerator` takes its ranges from it too, which covers
the split planning that
live rows are read through.
- **`LiveRowGlobalIndexReader`**: wraps each index shard. A full-text search
ranks only the live
rows of the shard, intersected with the include row ids if given, and is
passed on unchanged
when the whole shard is live. Predicate and vector searches are delegated
unchanged.
- **`LiveRowGlobalIndexReader::CreateUnion`**: loads live rows on the
visiting thread before the
search fans out to the shards, so a load that reads manifests on the
shared executor never waits
for it from one of its own threads. A search without a positive `limit`,
or whose include row
ids miss the rows every shard ranks, neither reads live rows nor fails on
them.
- **`GlobalIndexScanImpl`**: for tables with deletion vectors, wraps the
shard readers of every
index and loads live rows at most once per scan, on its first full-text
search that ranks rows,
pinned to the snapshot the scan reads the index from. A failed load is
retried on the next
search.
- **`FileStoreScan::WithPartitionFilter`**: restricts the live-row scan to
the partitions the
global index scan reads.
Vector search still ranks deleted rows, as before; Java filters it too, and
the same live-row
helper can be reused for it and for the table-level read in a follow-up.
### Tests
<!-- List UT and IT cases to verify this change -->
UT:
- `src/paimon/core/global_index/live_row_global_index_reader_test.cpp`:
`TestSearchUnchangedWhenEveryRowIsLive`, `TestSearchOnlyLiveRowsOfShard`
(including a shard
ending at `INT64_MAX`), `TestIncludeRowIdsClippedToShard`,
`TestCreateUnionLoadsLiveRowsBeforeSearchingShards` (live rows loaded on
the visiting thread with
a one-thread executor), `TestCreateUnionRanksLiveRowsOfSearchRanges`
(overlapping shards),
`TestCacheLiveRows` (cached, retried after a failure, and nullopt when
every row is live),
`TestLiveRowsErrorOnlyFailsSearchesThatRankRows` (a failed load fails a
search that ranks rows,
but not one without a positive `limit` or whose include row ids miss every
searched shard) and
`TestOtherVisitsDoNotReadLiveRows`.
- `src/paimon/core/io/data_file_meta_test.cpp`: `TestNonNullRowIdRange` (a
range ending at
`INT64_MAX`, one past it, no rows, a negative and a null first row id).
- `src/paimon/core/table/source/split_generator_test.cpp`:
`TestDataEvolutionRowIdRangeAtInt64Max`.
IT:
- `test/inte/data_evolution_table_test.cpp`:
`DataEvolutionDvTableTest.TestGlobalIndexLiveRows`
(live rows with and without a partition filter, in the snapshot before the
deletion, and the
error when a deletion vector cannot be read).
- `test/inte/global_index_test.cpp`:
`TestLuceneSearchSkipsRowsDeletedByDeletionVectors` and
`TestFullTextSearchSkipsRowsDeletedByDeletionVectors` share
`CheckSearchSkipsDeletedRows`: across
two shards, live rows keep the scores they had before the deletion, every
`limit` returns the
top-k live rows, include row ids are intersected with the live rows, a
scan created before the
deletion still ranks every row, and reading the matched rows returns all
of them.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
No change in `include/`. For tables with deletion vectors and row tracking,
full-text searches
through readers created by `GlobalIndexScan` rank only live rows.
Data-evolution scans now reject a
data file whose row id range is invalid instead of overflowing. No storage
format change.
### Documentation
<!-- Does this change introduce a new feature -->
No new feature. `docs/source/user_guide/global_index.rst` describes how
deleted rows are excluded
from full-text ranking and shows passing one snapshot id to both the index
scan and the data scan.
### Generative AI tooling
Generated-by: Claude Code 2.1.303 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
