johnsolomonj opened a new issue, #19241:
URL: https://github.com/apache/pinot/issues/19241
## What
Extend `GET /tables/{table}/size` and `GET /tables/{table}/metadata` to
include a per-index-type size breakdown — how much disk space each index type
(forward index, inverted index, bloom filter, range index, star-tree, text,
vector, dictionary) consumes per column and per replica.
## Problem
[#18185](https://github.com/apache/pinot/pull/18185) added tier-level
storage breakdown and per-column compression stats. However the total segment
size per tier is still a black box — operators cannot tell which index types
contribute most to storage cost, making it impossible to make informed
decisions about index configuration.
## Solution
At segment write time, collect per-index sizes and persist to
`metadata.properties`. The controller fans out to servers via the same bounded
fan-out path from [#18185](https://github.com/apache/pinot/pull/18185),
aggregates by index type, and exposes a new `indexSizeBreakdown` field on
existing API responses.
## Collection approach
`writeMetadata()` is moved out of `flushColIndexes()` and called explicitly
after `convertFormatIfNecessary()` in `seal()`:
```
1. flushColIndexes() ← index files written (V1 format)
2. computeDataCrc() ← CRC on V1 files (unchanged)
3. convertFormatIfNecessary() ← V1→V3: packs into columns.psf, writes
index_map, deletes V1 files
4. buildStarTreeV2IfNecessary()
5. writeMetadata() ← moved here; reads index sizes, persists to
metadata.properties
6. persistCreationMeta()
```
**V3 (default):** Read `v3/index_map` — exact sizes for all packed indexes.
For text/vector indexes:
- `storeInSegmentFile=true` — packed into `columns.psf`, appears in
`index_map` automatically
- `storeInSegmentFile=false` (default) — external directory, not in
`index_map`. Use existing `TextIndexUtils.hasTextIndex()` /
`VectorIndexUtils.hasVectorIndex()` + `FileUtils.sizeOfDirectory()` — already
used in V3 converter, no new dependencies
**V1/V2 (rare):** Stat individual files using
`IndexType.getFileExtensions(colMeta)`. `index_map` sizes equal V1 file sizes —
V3 converter reads V1 file sizes verbatim into `index_map`.
**Persisted to `metadata.properties`:**
```properties
column.message.indexSize.inverted_index = 12345678
column.message.indexSize.text_index = 34567890
column.user_id.indexSize.forward_index = 8901234
column.user_id.indexSize.vector_index = 56789012
```
**API response:**
```json
"indexSizeBreakdown": {
"forward_index": { "sizePerReplicaInBytes": 32000000000 },
"inverted_index": { "sizePerReplicaInBytes": 18000000000 },
"bloom_filter": { "sizePerReplicaInBytes": 2000000000 },
"star_tree": { "sizePerReplicaInBytes": 4000000000 },
"dictionary": { "sizePerReplicaInBytes": 1500000000 },
"text_index": { "sizePerReplicaInBytes": 8000000000 },
"vector_index": { "sizePerReplicaInBytes": 12000000000 }
}
```
## Flag
`tableIndexConfig.indexSizeStatsEnabled` (default `false`) gates collection
at seal time. When enabled, sizes are read from `v3/index_map` (or file stats
for V1/V2) and persisted to `metadata.properties`. `indexSizeBreakdown` is
included in API responses only when `?includeIndexSizeStats=true` query param
is passed — same pattern as `?includeColumnStats=true` in
[#18185](https://github.com/apache/pinot/pull/18185).
## Modules touched
| Module | Change |
|---|---|
| `pinot-segment-spi` | New `metadata.properties` key constants |
| `pinot-segment-local` | Move `writeMetadata()` after V3 conversion;
collect and persist per-index sizes |
| `pinot-common` | New `IndexSizeBreakdownInfo` DTO |
| `pinot-server` | Read from `ColumnMetadataImpl`, include in response |
| `pinot-controller` | Aggregate by index type, expose on existing APIs |
## Note on forward index and dictionary sizes
`#18185` already exposes
`forwardIndexAndDictionaryStorageSizePerReplicaInBytes` as part of compression
stats. The `forward_index` and `dictionary` entries in `indexSizeBreakdown`
complement this — compression stats combines them for ratio computation, while
`indexSizeBreakdown` breaks them out separately alongside all other index types
for a complete per-index-type cost picture.
## Out of scope
- Pinot Console UI changes
- Backfilling existing segments
## Implementation
`#TBD`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
