david-mollitor-db opened a new pull request, #58876: URL: https://github.com/apache/spark/pull/58876
### What changes were proposed in this pull request? `ArrowConverters.serializeBatch` serializes a single `ArrowRecordBatch` into a `byte[]` through a `ByteArrayOutputStream`. The stream was created with no initial capacity, so it starts at the JDK default of 32 bytes and grows by repeatedly doubling its backing array (`grow` + `Arrays.copyOf`) as the batch is written. This pre-sizes the `ByteArrayOutputStream` to the batch's body length (`ArrowRecordBatch.computeBodyLength()`, plus a small 1 KB margin for the IPC message header), clamped to the `Int` range: ```scala val estimatedSize = (batch.computeBodyLength() + 1024).min(Int.MaxValue).max(0L).toInt val out = new ByteArrayOutputStream(estimatedSize) ``` ### Why are the changes needed? The batch is written to the stream incrementally, not as one block: `MessageSerializer.serialize` writes the message metadata and then each of the batch's buffers (with alignment padding) individually, and the `WritableByteChannel` returned by `Channels.newChannel` further splits each write into chunks of at most 8 KB. So a fresh 32-byte `ByteArrayOutputStream` reallocates and copies its backing array on the order of `log2(batchSize / 32)` times per batch as it grows, producing that many short-lived intermediate arrays (GC pressure) plus the copies. `computeBodyLength()` returns the exact serialized body size and is cheap: it is O(number of buffers) integer arithmetic over the buffer layout (no scan of the payload, no allocation), which is negligible next to serializing the batch body. Pre-sizing lets the buffer be allocated once at the right size, eliminating the repeated grow-and-copy. This is an allocation micro-optimization; it is not expected to change throughput on its own. ### Does this PR introduce _any_ user-facing change? No. The serialized output is byte-for-byte identical. ### How was this patch tested? Existing `ArrowConvertersSuite` passes -- it round-trips batches through `serializeBatch` / `toDataFrame`, including schema, empty, and multi-batch cases. This is a behavior-preserving change to buffer sizing only, so no new tests were added. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
