hash-14 opened a new pull request, #18987:
URL: https://github.com/apache/pinot/pull/18987
## What
Adds opt-in **lazy segment loading** for OFFLINE tables: servers register a
metadata-only stub on ONLINE assignment instead of downloading the segment;
the first query that touches the segment materializes it from the deep store,
and a background sweeper evicts segments idle longer than a configurable TTL
back to stubs. Disk/memory then scale with the working set instead of total
assigned data. Entirely server-side — brokers, controllers and clients are
unchanged, and stubbed segments remain ONLINE and routable.
Resolves #18986
## Why
Servers currently must hold 100% of every hosted table locally at all times,
so fleets are sized by total data volume rather than working set. Workloads
with a long cold tail (historical/audit data, sparse per-tenant tables) pay
for disk and memory they rarely use. See the linked issue for the full design
discussion.
## How
- `pinot-spi`: new `LazyLoadConfig` (`enabled`, `idleEvictionSeconds`,
`deleteLocalOnEvict`) on `TableConfig`; `InstanceDataManagerConfig` gains
default methods for the three instance-level keys.
- `pinot-common`: ZK ser/de for `lazyLoadConfig`; new metrics
`LAZY_SEGMENT_COLD_LOADS`, `LAZY_SEGMENT_EVICTIONS`,
`LAZY_SEGMENT_LOAD_TIME_MS`, `LAZY_STUBBED_SEGMENT_COUNT`.
- `pinot-segment-local`: `SegmentDataManager.lastAccessTimeMs` stamped on
acquire (idle clock); table-config validation restricting the feature to
plain OFFLINE tables (rejected for REALTIME/upsert/dedup/dimension).
- `pinot-core` (`BaseTableDataManager`): stub registry; transparent
materialization on `acquireSegment(s)` (required and optional segments)
under the per-segment lock (concurrent cold queries dedupe into one
download; multi-segment queries materialize in parallel on a bounded,
idle-timeout pool, waiting at most
`lazy.materialize.timeout.seconds` — still-stubbed segments are reported
missing for that query while downloads continue in the background); acquire
retries the eviction-sweeper race so no false missing-segment errors; stubs
are registered only when the deep store has an authoritative copy and no
valid local copy exists (restart warms up from disk; permanent deep-store
failures still surface as Helix ERROR states); TTL eviction that never
touches segments with in-flight queries or without a deep-store URL; failed
materialization keeps the stub so the next query retries.
- `pinot-server`: `pinot.server.instance.lazy.*` config keys; single
`lazy-eviction-sweeper` daemon in `HelixInstanceDataManager`.
Feature is **off by default** and opt-in twice: instance-level
`pinot.server.instance.lazy.load.enabled=true` AND per-table
`lazyLoadConfig.enabled=true`. Tables without `lazyLoadConfig` follow the
existing eager path exactly (covered by a regression test).
## Testing
New `OfflineTableDataManagerLazyLoadTest` (TestNG, follows the
`BaseTableDataManagerTest` pattern with a `file://` deep store):
- stub on assignment + materialization on both acquire paths
- 8-thread concurrent cold acquire dedupes into a single download
- TTL eviction and re-materialization loop
- no eviction while a query is in flight
- eager load when a valid local copy exists (restart warm-up)
- instance kill switch forces eager loading
- non-lazy table regression (behavior identical to upstream)
This feature has also been running in production since June 2026 (Pinot 1.5.0
base, MinIO deep store, including a 10 GB / 135M-row scale test).
## Known limitations
Called out in the linked issue for design discussion (happy to address in
this PR or fast-follows per reviewer preference): stub sizes are not yet
included in table-size/quota reporting; materialization holds the per-segment
lock (slow downloads can delay Helix transitions on that segment); non-query
REST callers of `acquireSegment` materialize stubs as a side effect; reload
paths skip stubbed segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
