sunchao commented on code in PR #7316:
URL: https://github.com/apache/datafusion-comet/pull/7316#discussion_r4417628385
##########
native/core/src/execution/jni_api.rs:
##########
@@ -1555,6 +1573,17 @@ fn execute_plan(
exec_context.scans = scans;
exec_context.shuffle_scans = shuffle_scans;
+ // The task context created below copies the session config, so every
source and the
+ // JVM-fed leaves see the smaller batch size.
+ let batch_size = exec_context.session_ctx.copied_config().batch_size();
+ let rows = scalar_batch_rows(planner.scalar_bytes(), batch_size);
+ if rows < batch_size {
+ let state = exec_context.session_ctx.state_ref();
+ let mut state = state.write();
+ let config = state.config_mut();
+ *config = std::mem::take(config).with_batch_size(rows);
Review Comment:
I reproduced the filter case on `e9b8b0c3` with Spark 4.1.3. Summing
`length(concat(repeat('x', 262144), cast(id % 2 AS string)))` over a
one-partition 8,192-row range returned `2147491840` in Spark and in Comet
without the filter. Adding `WHERE id % 7 >= 0` kept all 8,192 rows and the
native `CometFilter` plan failed with `byte array offset overflow`.
There is also a case where setting the cap before planning would still be
insufficient:
```sql
SELECT sum(length(concat(repeat('x', 262144), cast(tile AS string))))
FROM (SELECT ntile(2) OVER (ORDER BY id) AS tile FROM range(0, 8192, 1, 1))
```
Spark returned `2147491840`. The native window emitted its whole partition
and the aggregate above it failed with the same overflow. The window contains
only integers, so the failure happens after the window, separate from the large
window payload tracked in #7270. Could the expanding operator enforce the cap
on the input it actually receives?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
