Jackie-Jiang opened a new pull request, #20186: URL: https://github.com/apache/pinot/pull/20186
## Summary Every `AggregationFunction` now serializes and deserializes its own intermediate result, and `ObjectSerDeUtils` becomes a set of codecs. - `serializeIntermediateResult` returns an `AggregationFunction.SerializedResult` (`getType`, `getSize`, `getBytes`, `writeTo(ByteBuffer)`, `writeTo(PinotOutputStream)`). `SerializedBytes` wraps a byte array, and `SerializedValue` sizes the value through its `SizedObjectSerDe` and writes it on demand. - `deserializeIntermediateResult` decodes the function's own type values, and the result never references the source buffer. - The `ObjectSerDeUtils.ObjectType` enum and the generic serialize/deserialize dispatch are removed. The codecs move from pinot-core to `org.apache.pinot.segment.local.utils.ObjectSerDeUtils` (`ObjectSerDe` / `SizedObjectSerDe`) and replace `CustomSerDeUtils`. - Data blocks and data tables write intermediate results straight into their `PagedPinotOutputStream` with `getSize()` and `writeTo`, so large states such as `PERCENTILE` lists and distinct sets skip the full-size byte array. `PinotOutputStream` gains bulk primitive array writes, `PrimitiveArrays` provides bulk reads in place of `PrimitiveArrayOutput`, and `PERCENTILE` merges a serialized list straight into the merged list. - The `FUNNELCOUNT` bitmap, set and theta strategies size their steps up front and write them on demand. The `ARRAYAGG` distinct star-tree aggregator keeps primitive sets and writes them with the shared set codecs. - `EXPRMIN` / `EXPRMAX` intermediate results deserialize into a mergeable object, so these functions can use the grouping-sets base-aggregation derive. Fixes: - Reading a data block from a buffer at a non-zero offset (`DataBlockUtils` / `ZeroCopyDataBlockSerde`), which `EXPRMIN` / `EXPRMAX` intermediate results rely on. - `PagedPinotOutputStream.seek(0)` after seeking past the first page left the page start offset stale. - The on-heap star-tree builder read the `ARRAYAGG` distinct max cell size before the cells were serialized (`ValueAggregator#isMaxAggregatedValueByteSizeDerivedFromSerialization`). The wire and on-disk formats are unchanged. `IntermediateResultByteBufferSerDeTest` pins the type value of every function, `FunnelCountSerDeTest` pins the step framings byte for byte, `DataSketchesSerialCompatTest` reads datasketches 6.2.0 fixtures, and `ObjectSerDeUtilsTest` checks every codec against hand-written bytes. ## Backward incompatibility Java API only: - A custom `AggregationFunction` returns a `SerializedResult` from `serializeIntermediateResult` and decodes its own type values in `deserializeIntermediateResult`. - Code using `org.apache.pinot.core.common.ObjectSerDeUtils`, its `ObjectType` enum or `CustomSerDeUtils` moves to the codecs in `org.apache.pinot.segment.local.utils.ObjectSerDeUtils`. Mixed-version clusters are unaffected, since type values and payload bytes do not change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
