XiaoHongbo-Hope opened a new pull request, #1175:
URL: https://github.com/apache/paimon-rust/pull/1175
## Summary
Add opt-in `local-cache.shared=true` for cooperating reader processes on one
node. The existing process-local memory/disk cache remains the default.
This shares raw immutable file blocks, not parsed Parquet metadata or decoded
Arrow values. It can be enabled by native SDK callers passing catalog
options:
```text
local-cache.enabled=true
local-cache.dir=/node-local/paimon-cache
local-cache.shared=true
local-cache.max-size=20 GiB
local-cache.whitelist=meta,global-index,data
```
## Implementation
- Separate versioned directory, same-UID local filesystem only, with explicit
matching capacities for participating processes.
- Standard-library file locks coordinate publication and accounting. Striped
file download locks coalesce concurrent misses, including full/range reads.
Runtime contention is bounded; failure falls back to the original source.
- Atomic block publication reuses the existing checksummed cache encoding.
- Persistent recency and batched eviction bound encoded block capacity across
processes; interrupted accounting is recovered before another publication.
- A shared generation fences old readers after invalidation. Invalidation is
intentionally directory-wide, even when a single logical path is
invalidated.
- Process-local file-size caching is bypassed in this mode.
## Scope / draft review questions
This is a draft for design review, not a production rollout recommendation.
- Large-cache, multi-worker throughput and real OSS request savings still
need
measurement; the directory gate serializes local cache publication/hits.
- Directory-wide invalidation is conservative and can hurt write-heavy
workloads.
- Invalidation lock timeout / filesystem-error semantics need a further audit
before this is ready: the cache API currently has infallible invalidation,
so
a failed shared generation rotation cannot be reported to mutation callers.
Do not use this draft for mutable same-path replacements.
- Participants must initialize catalogs/readers inside workers, not reuse
inherited live runtimes/caches after `fork`.
- The existing parsed Parquet metadata cache (#929) has a separate
inherited-lock
hazard after fork. This feature does not claim to fix that issue.
- NFS / cross-node sharing is out of scope. Control files and one temporary
in-flight block are outside the encoded-block budget.
## Validation
Rust 1.94 (MSRV), Linux:
The focused cache suite passes **52 tests** on the proposed source, including
the separate-process tests below. Formatting and whitespace checks pass.
```sh
cargo +1.94.0 test -p paimon --lib io::cache \
--no-default-features --features storage-fs,storage-memory
cargo +1.94.0 fmt --all -- --check
git diff --check
```
Tests cover separate executable processes sharing one source download,
CachedFileReader range/full reads, process termination releasing locks,
cross-process invalidation fencing stale readers, capacity mismatch,
corruption,
and interrupted accounting recovery. No production tables or customer
programs
are modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
