sunchao opened a new pull request, #7310: URL: https://github.com/apache/datafusion-comet/pull/7310
## Which issue does this PR close? Closes #7309. ## Rationale for this change Every broadcast input batch currently allocates a 1 MiB output chunk and constructs a compression codec. Thousands of tiny batches repeat both costs. This shared serializer improvement is independent of the opt-in native producer batching in #7258 and can land first. ## What changes are included in this PR? - Size output chunks from the Arrow field and dictionary buffers, clamped to 4 KiB–1 MiB. Serialization metadata can exceed the estimate; the output stream grows additional chunks as needed. - Lazily create one compression codec per serializer iterator, while keeping an independent compressed output stream for each batch. - Add a one-row, many-column regression whose compressed schema metadata exceeds the small buffer estimate and round-trips all values across chunks. The serialized stream format and broadcast assembly remain unchanged. No new configuration is introduced. ## How are these changes tested? - Both changed Scala files compiled against Spark 3.5.9 dependencies. - All 16 `UtilsSuite` tests passed, including dictionary serialization, sliced vectors, metadata-driven multi-chunk output, and the existing broadcast string-size cap. - The new test fails against the exact `main` serializer at `d7f4fcaa58`: the original 1 MiB allocation produces one output chunk. - JVM library tracing confirmed that these tests use Arrow Java and compression libraries without loading Comet's native library. - Spotless formatting and `git diff --check` passed. Fresh matched performance measurements are in progress; no measurements from the earlier implementation are claimed here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
