xiangfu0 opened a new pull request, #19511: URL: https://github.com/apache/pinot/pull/19511
## What `ColumnValueSegmentPruner` asks each segment for a column's `DataSource` and then reads nothing from it but the `DataSourceMetadata` — data type, min/max, partition function. Reaching that metadata through `getDataSource` forces a segment that builds its columns lazily to construct the whole `ColumnIndexContainer` — every index reader for the column — for a segment it is about to discard. This adds `IndexSegment#getDataSourceMetadata(String, Schema)`, which names what the caller actually wants and defaults to `getDataSource(column, schema).getDataSourceMetadata()`, so no existing implementation has to change. `ImmutableSegmentImpl` overrides it to answer from column metadata: `ImmutableDataSourceMetadata` already delegates to `ColumnMetadata` and holds no readers, so nothing needs building. A column the segment does not have still falls through to the data source, which is where the schema-driven default and virtual columns are created. The pruner keeps its per-segment data-source cache for mutable segments, whose metadata is not derivable without the data source. ## Why `ValueBasedSegmentPruner#prune` is a serial loop over **every segment the server holds**, on the query thread — it implements only the two-arg `prune`, so `SegmentPruner`'s three-arg default silently discards the `ExecutorService` (`BloomFilterSegmentPruner`, by contrast, does override it and prunes in parallel). Combined with lazy column materialization (#19477) that means the pruner materializes a column per held segment, and on a tiered-storage external table each materialization parses a Parquet footer. Measured on a production server holding 44,780 segments, this dominated a query that timed out at 300 s for a filter matching 1,839 segments — the pruner was doing tens of thousands of footer parses, in series, for segments it then discarded. The win is not limited to lazy mode: building an index container to read min/max is wasted work in any configuration. ## Tests `ImmutableSegmentImplTest#testDataSourceMetadataDoesNotMaterializeTheColumn` builds a lazy segment with a mocked `ColumnMaterializer`, reads the metadata, and asserts `verifyNoInteractions(materializer)` — then shows that asking for the data source *does* materialize, and that the two agree. It fails when the override is removed. 51 pruner tests in pinot-core and 35 segment tests in pinot-segment-local pass; spotless, checkstyle and license clean. ## Stack Part 11, based on #19486. Review only this part's own commit; the earlier parts account for the rest of the diff. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
