airborne12 opened a new pull request, #67977:
URL: https://github.com/apache/doris/pull/67977
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Doris reports the total inverted index size of a rowset or table (for
example `information_schema.rowsets.INDEX_DISK_SIZE` and `SHOW DATA`), but SQL
users cannot see how much space each index takes, or how that space splits
between the term dictionary, postings, positions and statistics. Deciding
whether to drop `support_phrase`, change a parser or switch the index storage
format therefore requires inspecting index files by hand on each backend.
This PR adds the table function `index_disk_usage`:
```sql
SELECT index_name, column_name, structure, storage_format,
SUM(total_bytes), SUM(dict_bytes), SUM(posting_bytes),
SUM(position_bytes), SUM(stats_bytes)
FROM index_disk_usage("database" = "db1", "table" = "t1")
GROUP BY 1, 2, 3, 4;
```
- Parameters: `database` and `table` (required), `partitions` and `indexes`
(comma-separated names), `level` (`tablet` by default, `rowset` or `segment`),
and `position_detail` (default `false`).
- Columns: `PARTITION_NAME`, `TABLET_ID`, `BACKEND_ID`, `ROWSET_ID`,
`SEGMENT_ID`, `INDEX_ID`, `INDEX_NAME`, `INDEX_TYPE`, `COLUMN_NAME`,
`INDEX_SUFFIX`, `STRUCTURE` (`TERM`/`BKD`/`ANN`/`CONTAINER`), `STORAGE_FORMAT`
(`V1`/`V2`/`V3`/`SNII`), `SEGMENT_COUNT`, `ROW_COUNT`, `TOTAL_BYTES`,
`DICT_BYTES`, `POSTING_BYTES`, `POSITION_BYTES`, `STATS_BYTES`, `OTHER_BYTES`,
`STATS_SOURCE`.
How it works:
- FE resolves the table, its formal partitions and base-index tablets, and
checks the `SHOW` privilege. Partition visible versions are read after
releasing the table lock, because cloud partitions may fetch them from
meta-service; cloud mode uses the batched snapshot version API. Each tablet is
sent to one queryable replica through a new `INDEX_DISK_USAGE` metadata scan
range.
- BE captures the rowsets visible at that version (syncing rowsets first in
cloud mode) and reads only index file metadata of each segment:
- V1/V2/V3: CLucene compound directory entries are attributed by extension
(`tis`/`tii` to dictionary, `frq` to postings, `prx` to positions, `nrm` to
statistics, everything else to other). BKD files form a `BKD` row with only
`TOTAL_BYTES`. The shared V2/V3 compound header is reported as a `CONTAINER`
row.
- SNII: the logical index directory and core metadata are read. The
dictionary region, sampled term index, dictionary block directory and BSBF
count as dictionary; the posting region as postings; core metadata and norms as
statistics; the null bitmap as other. SNII interleaves positions with postings,
so `POSITION_BYTES` is NULL unless `position_detail=true` decodes the posting
region to split them out. Docs-only indexes report 0 positions.
- Rows are aggregated to the requested level. Within a segment,
`TOTAL_BYTES` of all rows adds up to the index file sizes; unknown components
are NULL.
- `position_detail=true` is rejected when the request covers more tablets
than the session variable `index_disk_usage_position_detail_max_tablets`
(default 10000).
Collecting these statistics at write time and an FE-aggregated
`information_schema` table are left to follow-up PRs.
### Release note
Add the table function `index_disk_usage` to show the on-disk size of each
inverted or ANN index, broken down into dictionary, posting, position and
statistics bytes, at tablet, rowset or segment level.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `external_table_p0/tvf/test_index_disk_usage_tvf`: V2, V3 and SNII
tables with two rowsets each; component presence per structure, per-rowset
totals reconciled with `information_schema.rowsets.INDEX_DISK_SIZE`, all three
levels, SNII `position_detail`, index filtering and parameter errors. Passed on
a local single FE/BE cluster.
- [x] Unit Test
- BE: `IndexDiskUsage*` (V1/V2/V3/SNII collectors, docs-only and
positional indexes, `position_detail`, BKD, index filtering, missing files,
level aggregation) and `SniiSegmentReaderLogicalIndexesTest`: 22 passed.
- FE: `IndexDiskUsageTableValuedFunctionTest` (parameters,
privileges, partition/index resolution, visible version read outside the table
lock, `position_detail` tablet limit) and `IndexDiskUsageScanNodeTest` (replica
selection and grouping by backend): 20 passed.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
- The new table function needs a page in doris-website; not
submitted yet.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
