johnsolomonj opened a new pull request, #19255:
URL: https://github.com/apache/pinot/pull/19255
## Summary
Adds an opt-in, per-index-type storage size breakdown, addressing #19241.
- New `indexSizeStatsEnabled` table config flag: when set, each index's
on-disk size (keyed by `IndexType#getId()`) is persisted into
`metadata.properties` at segment-build time and refreshed after every reload.
- New `?includeIndexSizeStats=true` query parameter on the server's
table-size endpoint and the controller's `/tables/{table}/size` aggregation,
returning an `indexSizeBreakdown` map alongside the existing disk-size totals.
- Controller-side aggregation picks one server as the "donor" of index-size
data per segment (independent of which server's disk-size is used as the
representative), merging disjoint index types across segments.
## Five things worth calling out explicitly
1. **`indexSizeBreakdown` intentionally ignores the `columns=` filter** that
the metadata endpoints otherwise respect. It's a segment/index-level total, not
a per-column breakdown, so there's nothing for a column filter to narrow.
2. **Turning on `indexSizeStatsEnabled` changes the segment CRC**, because
it adds new keys to `metadata.properties`. This is expected — it's the same
category of change as any other metadata-affecting config flag — but is worth
flagging for anyone diffing segments across the flag toggle.
3. **The persisted sizes are not a pure build-time snapshot.** They're
written at segment-seal time, but `SegmentPreProcessor` also refreshes the
relevant keys after every reload's index handlers finish, so the values track
the current on-disk state rather than going stale after a reload changes which
indexes exist.
4. **`OPEN_STRUCT` child columns are not measured.** Their sizes aren't
attributed per-child in this first pass; this is a known gap, not an oversight.
5. **"Cold-tier" segments are covered because there's no cold-tier-specific
code path**, not because of special handling. Every tier ultimately lives on
some Pinot server as a normal loaded `IndexSegment` (the only tier storage type
is `pinot_server`), so a tiered segment goes through the exact same
write/reload/read logic as any other segment — this was independently
re-verified twice this cycle by grepping for tier-based branching in the
collection, reload, and read paths and finding none.
## Test plan
- [x] Unit tests across all touched modules (pinot-spi, pinot-segment-spi,
pinot-common, pinot-segment-local, pinot-controller, pinot-server,
pinot-sql-ddl) — 124 tests, 0 failures.
- [x] Two new integration tests
(`IndexSizeBreakdownOfflineIngestionIntegrationTest`,
`IndexSizeBreakdownRealtimeIngestionIntegrationTest`) against a real cluster +
Kafka.
- [x] 4 formatting/style/license gates (spotless, checkstyle,
license:format, license:check) clean on every touched module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
