gripleaf opened a new pull request, #430: URL: https://github.com/apache/paimon-cpp/pull/430
### Purpose Linked issue: none. Reduce repeated file-metadata decoding on exact snapshot manifest-cache hits when a scan selects only a small subset of a bucket's partitions. **Scenario.** A partitioned, bucketed Paimon table serves repeated point reads. An application batches 50 requested cells, grouping cells for the same key. Each underlying lookup uses a key equality predicate and a fixed date-partition filter, allowing the planner to select one bucket. The bucket-scoped manifest cache, however, contains live files from many date partitions. Reusing that cache currently decodes the full file metadata before discarding entries outside the requested partition. For example, one sampled bucket contains 604 live entries across 84 partitions, but only six entries belong to the queried partition: ```text Before: deserialize 604 full entries -> apply partition filter -> keep 6 After: inspect 604 partition fields -> apply partition filter -> fully decode 6 ``` The change selects the exact retained snapshot and evaluates its partition filter before constructing `ManifestEntry` / `DataFileMeta` objects and decoding file statistics. It still traverses the serialized payload; it does not eliminate all cache-read work. Scans selecting most partitions are not expected to gain the same benefit. The shared cache retains the complete bucket contents. The filtered selection is request-local and is never written back as the cache value. Existing miss/rebuild behavior, snapshot retention, and subsequent scan filters are preserved. **Component measurements (not end-to-end query latency).** A standalone prototype harness compared full deserialization followed by partition filtering against selective decoding, using the same serialized cache payloads from four real buckets. Each bucket had 84 partitions; each comparison selected one partition. Each path ran 200 iterations per pair, with five pairs and alternating execution order, on the same host. The table reports mean elapsed time per iteration across those pairs. Both paths returned exactly equal selected entries. | Sample | Cached entries | Selected entries | Before (ms) | After (ms) | Time reduction | | --- | ---: | ---: | ---: | ---: | ---: | | A | 604 | 6 | 4.566 | 0.839 | 81.6% | | B | 616 | 6 | 4.663 | 0.849 | 81.8% | | C | 566 | 5 | 4.282 | 0.774 | 81.9% | | D | 604 | 5 | 4.593 | 0.827 | 82.0% | These measurements motivated the upstream port; they are not timings of a deployed upstream build. The port was validated by the tests below. Full cold/warm end-to-end performance and long-running concurrent behavior have not yet been benchmarked for this upstream build. ### Tests Built the community checkout and passed **51 tests from six suites**: ```sh cmake --build build --target paimon-core-test -j 8 ./build/debug/paimon-core-test --gtest_filter='*FileStoreScanTest.*:AppendBucketPruningTest.*:ManifestEntrySerializerTest.*' ``` New regression coverage: - `FileStoreScanTest.SelectCachedSnapshotWithoutPollutingOtherPartitions`: alternating partition filters, unfiltered reads, exact-snapshot misses, reduced retention, empty snapshots, and filter error propagation; checks that the complete serialized value remains reusable. - `FileStoreScanTest.RejectInvalidSnapshotSelectionHeader`: empty input, invalid magic, negative counts, and a truncated header. - `AppendOnlyFileStoreScanTest.SnapshotCachePreservesOtherPartitions`: creates a local partitioned table, populates the cache through a filtered scan, then queries another partition, an empty result, the original partition, and all partitions. Compares filtered plans with cache-disabled scans and checks the new counters. Existing scan, bucket-pruning, snapshot cache rebuild, and serialization tests also passed. `pre-commit run --files` passed for all eight changed files; `git diff --check` passed. The repository-wide test suite was not run. ### API and Format Adds two metric-name constants to `ScanMetrics`: - `snapshotCacheCandidateEntries`: entries inspected in the exact retained snapshot on the selective cache-hit path. - `snapshotCacheDecodedEntries`: entries fully decoded after partition filtering on that path. These are per-plan counters, reset to zero when the selective path is not used. Existing lazy-scan candidate counts retain their meaning. No existing method signatures, serialized cache format, table storage format, or protocols change. The new selection helper is internal. ### Documentation Updated `manifest_entry_cache.rst` with the selective-decoding flow, an example, shared-cache correctness guarantees, and the new metrics. Updated `metrics.rst` with the metric definitions. ### Generative AI tooling Generated-by: OpenAI Codex (codex-cli 0.162.1) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
