gortiz opened a new issue, #19805: URL: https://github.com/apache/pinot/issues/19805
Exact `PERCENTILE` keeps every input value of a group in a `DoubleArrayList`. With large groups this list easily passes 512 KB, which is half a G1 region with the default 1 MB regions. G1 allocates such an object as a humongous object, directly in old-gen regions. Humongous allocations raise GC pressure and can cause early concurrent cycles, and full GCs when the heap is fragmented. On the way from one stage to the next, each intermediate result is copied at full size more times than necessary: | Step | Full-size copies | Required? | |---|---|---| | `serializeIntermediateResult` builds a `byte[]` | 1 | No. The builder copies that array into its output stream right away. | | `DataBlockBuilder` / `BaseDataTableBuilder` write the bytes into the output | 1 | Yes | | The receiver deserializes each value into a new `DoubleArrayList` | 1 | No. `merge()` then copies it into the merged list and drops it. | | `merge()` appends into the merged list | (growth only) | Yes | So for every large group, the sender and the receiver each create one humongous object that lives for a few microseconds. The serializer and the deserializer also move the values one at a time (`putDouble` / `getDouble` in a loop). A bulk copy through a typed buffer view is several times faster. I propose: 1. Let `SerializedIntermediateResult` expose `getSize()` and `writeTo(DataOutput)`. The builders write through them, so a function can write its result straight into the destination stream. The default keeps today's `byte[]` behavior. 2. Add `AggregationFunction.mergeSerializedIntermediateResult(intermediateResult, CustomObject)`. The default deserializes and calls `merge()`. A function can override it to read the serialized value straight into the merged result. The MSE aggregate executors use it for `OBJECT` columns of serialized blocks. 3. Add bulk `writeInts` / `writeLongs` / `writeDoubles` to `PinotOutputStream`, with a paged override, and read the primitive lists of `ObjectSerDeUtils` in bulk. 4. Use all of this in `PercentileAggregationFunction`. The wire format does not change. I have a PR ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
