gripleaf opened a new issue, #428: URL: https://github.com/apache/paimon-cpp/issues/428
### Search before asking - [x] I searched existing issues and found no issue covering this combination of early row-range pruning and decoded manifest caching. Related metadata-planning improvements are tracked in #337 and #375. ### Motivation Data-evolution scans with row ID ranges currently materialize manifest entries and file metadata before discarding files outside the requested ranges. Selective row-ID reads therefore pay allocation and deserialization costs for metadata they do not need. The existing manifest cache stores source-file bytes. Although cache hits avoid source I/O, repeated scans still decode the manifest format. This adds planning overhead for repeated point lookups, batched lookups, and row-ID range reads. ### Solution - Check the aligned Arrow row-count and first-row-ID columns before materializing manifest entries when data evolution, lazy decoding, and row-range filtering are enabled. Preserve version validation, conservatively retain unknown or overflowing ranges, and keep existing filters and Add/Delete merge semantics. - Cache complete, query-independent metadata as Arrow IPC through the existing shared `ObjectsFile<T>` path for manifest files, manifest lists, and index manifests. Warm reads can bypass source-format decoding, and different filters can reuse the same cache entry. - Keep the IPC buffer and its allocator alive for active readers. Charge the retained buffer capacity through `CacheValue::GetMemoryUsage()` while preserving the logical payload length. Document the corresponding accounting contract for external cache implementations. No persisted table format or protocol change is needed. A cold cache load decodes the complete file, so a cold bucket read may do more work than an uncached selective read. ### Anything else? Implementation: #411. Regression coverage includes cold and warm reads, lazy-decode and cache modes, row-range boundaries, schema evolution, version validation, Add/Delete merging, retained-capacity accounting, eviction, and allocator lifetime. Application-level performance measurements and their limitations are documented in the PR. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! Implementation is available in #411. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
