zhangfengcdt opened a new pull request, #7316: URL: https://github.com/apache/datafusion-comet/pull/7316
## Which issue does this PR close? Closes #7247. ## Rationale for this change A string scalar, from a scalar subquery or a literal, is expanded to the batch length by aggregates, the hash partitioner and string kernels. A 256 KiB string over the default 8192-row batch needs 2 GiB of value bytes, past Arrow's 32-bit `Utf8` offsets, so native execution panics with `offset overflow` or fails with `byte array offset overflow`. Spark evaluates such scalars row by row and succeeds. The expansion happens in DataFusion(aggregate argument evaluation, `concat`) as well as in Comet's shuffle partitioner, and a plan-time fallback cannot see a subquery's value. So the fix bounds the rows per batch instead. ## What changes are included in this PR? - The planner adds up the bytes of every string and binary scalar it plans, literals and scalar subquery results. The native plan is built on the first `executePlan`, after the JVM has registered the subqueries, so the result's size is known then. - `executePlan` lowers the session batch size so that total, expanded to the batch length, stays within 1 GiB. For the issue's string that is 4096 rows. The 1 GiB leaves room for kernels that append to the scalar, such as `concat`. - DataFusion's sources read the batch size when they run. The native `range` source captured it at planning, so it now reads it from the session as it makes each batch. - `ScanExec` and `ShuffleScanExec` receive batches the JVM sized, so they slice any batch wider than the session batch size. Slices share buffers. With the default configuration nothing is sliced and nothing changes. ## How are these changes tested? - New fixture `operators/large_scalar_string.sql` with the issue's four queries plus the literal over a native `range`. - Rust unit tests for the batch slicer, the row cap, and the range source reading a batch size lowered after planning. - `CometSqlFileTestSuite` for `operators/`, `scalar_subquery` and `shuffle` on Spark 4.1, and the core and operators crate test suites. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
