xiangfu0 opened a new pull request, #19595: URL: https://github.com/apache/pinot/pull/19595
## Summary Adds distinct `ARRAY_AGG` (`arrayAgg(col, 'type', true)`) as a star-tree aggregation function. A star-tree cell aggregates many rows into one stored value, so the aggregation must be associative/idempotent under merge. Distinct `arrayAgg` satisfies this (merge = set-union); the non-distinct variant would require an unbounded per-node multiset and is intentionally **not** supported on the star-tree. ## What's included - **New `ArrayAggDistinctValueAggregator`** (`pinot-segment-local`): stores the distinct set as a variable-length `BYTES` cell prefixed with a stable 1-byte element-type tag. The payload is byte-for-byte identical to `ObjectSerDeUtils.*_SET_SER_DE`, so the query-time function deserializes the cell directly. Wired into `ValueAggregatorFactory`. - **Query path** (`pinot-core`): the six single-value-storable distinct arrayAgg functions (`INT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/STRING`) read the pre-aggregated `BYTES` column in **both** `aggregate()` and `aggregateGroupBySV/MV`. `canUseStarTree()` returns `true` only for the distinct variants, so a stored set is never served to a non-distinct query. - **Build validation** (`BaseSingleTreeBuilder`): arrayAgg source columns must be single-value, dictionary-encoded and non-`BYTES`. ## Scope / limitations - Only distinct `arrayAgg`; only single-value, dictionary-encoded, non-`BYTES` source columns. - Ordering is not preserved (distinct is inherently unordered). - `BYTES`-typed source columns are excluded because a raw `BYTES` value is indistinguishable at query time from a serialized set. ## Testing - `ArrayAggDistinctValueAggregatorTest` — per-type serialize/deserialize round-trips, dedup, union, wire-format layout, null handling. - `ArrayAggStarTreeV2Test` — full star-tree build → read → merge vs. raw scan (via `BaseStarTreeV2Test`). - `ArrayAggStarTreeQueriesTest` — end-to-end SQL comparing `useStarTree` on vs. off for grouped and non-grouped distinct arrayAgg, asserting the star-tree is actually used for distinct and **not** used for non-distinct. - No regressions in the existing arrayAgg and value-aggregator suites. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
