comphead opened a new pull request, #7193: URL: https://github.com/apache/datafusion-comet/pull/7193
## Which issue does this PR close? Closes #7192. ## Rationale for this change `coalesceBroadcastBatches` appends the whole broadcast into one `VectorSchemaRoot`. Arrow Java's `VectorAppender` grows a string or binary data buffer by doubling it and cannot go past 1 GiB, so a broadcast join fails with `OversizedAllocationException` once one such column of the build side holds more than that. Spark runs the same query. #7192 has the details and a repro. ## What changes are included in this PR? - `coalesceBroadcastBatches` writes out the batch it is building and starts a new one when the next input batch would take a string or binary vector, nested ones included, past 2^30 bytes of values. A broadcast below that is still coalesced into one batch. The readers already take a broadcast of several batches, which is what the uncoalesced fallback produces. - The check adds the two sizes as `Long`, because `VectorAppender` adds them as an `Int`. - An `OversizedAllocationException` from the appender now leaves the broadcast uncoalesced, as an `IllegalArgumentException` already does. That covers a single batch too large to append on its own. - The cap is a parameter, `maxValueBytes`, so the test can use a small one. ## How are these changes tested? - A new `UtilsSuite` test coalesces four batches, each with a string column and a struct of strings, under a 20-byte cap. Only the nested strings reach the cap, so it also checks that the split looks into children. It expects two coalesced batches of four rows each, with every value in order. - The repro from #7192 on Spark 4.0 with the standard Comet setup (`CometShuffleManager`, AQE on), built from `main` at 52ea7120b0 with this change. With 1,200,000 rows (1.12 GiB of strings) the join now returns the same 1,000 rows as Spark, and its 148 input batches are broadcast as 2 coalesced batches. With 1,000,000 rows (0.94 GiB) the broadcast is still one batch, as before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
