sunchao opened a new issue, #7309: URL: https://github.com/apache/datafusion-comet/issues/7309
Broadcast serialization allocates a 1 MiB heap chunk and constructs a compression codec for every input batch. When the input contains many tiny Arrow batches, this repeats allocation and codec setup thousands of times even though each serialized stream is small. Size output chunks from the Arrow field and dictionary buffers, bounded between 4 KiB and 1 MiB, and reuse a compression codec within one serializer iterator while creating an independent output stream for each batch. Keep output framing and decoding unchanged. This is the shared serializer improvement discussed in #7258. It should land independently of the experimental native producer-batching setting because it applies to every broadcast input. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
