Jackie-Jiang opened a new pull request, #19317: URL: https://github.com/apache/pinot/pull/19317
## Summary `OffHeapSingleTreeBuilder` keeps the star-tree record file offsets in a `List<Long>`, which costs ~28 bytes of heap per record (24-byte boxed `Long` plus a 4-byte reference under compressed oops). The number of star-tree records can go into the hundreds of millions for large segments with a wide `dimensionsSplitOrder`; in one production incident the offsets list alone held ~700M boxed `Long`s (~19.6GB including the `ArrayList` backing array), OOMing the server during realtime segment conversion. This PR replaces the boxed list with a compact `RecordOffsets` structure: - Offsets are tracked as a prefix sum of the appended record lengths, so `appendRecord` passes the record length instead of computing the next offset from a read-back of the last element. - Start offsets are stored in an `IntArrayList` (4 bytes per record) until the record file grows beyond `Integer.MAX_VALUE`, and in a `LongArrayList` (8 bytes per record) afterwards. Offsets increase monotonically, so the switch happens at most once and needs no copy: reads pick the list by comparing the index against the int head's size. - The end offset (total file length) is a plain `long` field, removing the extra end-sentinel entry and the boxing/unboxing on every append and on every read in the star-node sort comparator. For the build above this reduces the offset bookkeeping from ~19.6GB to ~5.6GB of heap (4 bytes per record while the record file stays under 2GB), and removes the `Long` allocation churn from the build hot path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
