lszskye opened a new pull request, #65: URL: https://github.com/apache/paimon-cpp/pull/65
## feat: add global index infrastructure ### Changes #### `GlobalIndexResult` - Abstract base class for global index query results holding matched row ids. Supports `CreateIterator()` for row id traversal, `And()`/`Or()` for set operations, `AddOffset()` for row id shifting, `ToRanges()` for converting to sorted non-overlapping ranges, and `Serialize()`/`Deserialize()` for binary persistence. #### `ScoredGlobalIndexResult` - Extends `GlobalIndexResult` with per-row relevance scores. Adds `CreateScoredIterator()` returning `(row_id, score)` pairs ordered by ascending row id. #### `BitmapGlobalIndexResult` - `GlobalIndexResult` implementation backed by a lazily-initialized Roaring64 bitmap. The bitmap is constructed on-demand via a `BitmapSupplier` callback on first access to `GetBitmap()`, avoiding unnecessary computation during early stopping. Provides `FromRanges()` factory for creating results from row id ranges. #### `BitmapScoredGlobalIndexResult` - `ScoredGlobalIndexResult` implementation combining a Roaring64 bitmap with a parallel float score vector. Both bitmap and scores are ordered by ascending row id. Supports bitmap-level `And()`/`Or()` with score merging. #### `GlobalIndexScan` - Entry point for scanning global indexes on a table. Supports partition filtering via explicit partition maps or `Predicate`-based pruning. Creates `GlobalIndexReader` instances for specific fields with optional `RowRangeIndex` scoping. #### `GlobalIndexReader` - Abstract reader interface extending `FunctionVisitor<GlobalIndexResult>` for evaluating filter predicates against a global index. Adds `VisitVectorSearch()` for approximate vector similarity search and `VisitFullTextSearch()` for full-text queries. Exposes `IsThreadSafe()` and `GetIndexType()`. #### `GlobalIndexWriter` - Abstract interface for building global indexes from Arrow data batches. `AddBatch()` accepts a struct `ArrowArray` with relative row ids, and `Finish()` finalizes the build returning `GlobalIndexIOMeta` entries. #### `GlobalIndexer` - Abstract interface for creating `GlobalIndexReader` and `GlobalIndexWriter` instances from Arrow schemas and file I/O handlers. #### `GlobalIndexerFactory` - Factory for creating `GlobalIndexer` instances by index type identifier (e.g., "bitmap"). Appends a "-global" suffix for namespace separation from file-level indexes. #### `GlobalIndexWriteTask` - Static utility for building and writing a global index for a specific data split and field. Orchestrates `GlobalIndexer`, `GlobalIndexFileWriter`, and split reading into a single `WriteIndex()` call returning a `CommitMessage`. #### `IndexedSplit` - Extends `Split` with physical data split details, row ranges, and optional per-row scores for index-aware reading. #### `GlobalIndexIOMeta` - Struct holding metadata for a single global index file: file path, file size, and optional serialized metadata bytes. #### `GlobalIndexFileReader` - Abstract interface for reading global index files. Provides `GetInputStream()` to open a file by path. #### `GlobalIndexFileWriter` - Abstract interface for writing global index files. Provides `NewFileName()`, `NewOutputStream()`, `GetFileSize()`, and `ToPath()`. #### `CompleteIndexScoreBatchReader` - `BatchReader` wrapper that enriches output Arrow arrays with index score values. Locates the `_INDEX_SCORE` column in the read schema and overwrites it with float scores from the provided score vector, advancing a cursor across batches. #### `GlobalIndexUtils` - Static utility providing `CheckRelativeRowIds()` for validating that relative row id arrays match Arrow array lengths and expected sequential ordering. ### Tests - `BitmapGlobalIndexResultTest` - `BitmapScoredGlobalIndexResultTest` - `CompleteIndexScoreBatchReaderTest` - `GlobalIndexResultTest` - `GlobalIndexUtilsTest` - `GlobalIndexerFactoryTest` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
