sunchao opened a new issue, #7257: URL: https://github.com/apache/datafusion-comet/issues/7257
### What is the problem the feature request solves? A broadcast build that reads many small local native shuffle blocks serializes an Arrow stream for each decoded batch. The driver collects all these streams before combining them, so tiny inputs multiply Arrow export, serialization, and stream-opening work. ### Describe the potential solution Add opt-in bounded grouping of small decoded batches before Arrow export and broadcast serialization. Start with direct local Comet native shuffle inputs and INT, BIGINT, BOOLEAN, STRING, and BINARY columns. Preserve AQE partition ranges and other consumers of a reused shuffle. Bound grouped inputs by rows, retained/copy bytes, and input count; pass large batches through without a merge copy. ### Willingness to contribute I can contribute the implementation, regression tests, and a reproducible producer benchmark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
