zhangfengcdt opened a new issue, #7689:
URL: https://github.com/apache/datafusion-comet/issues/7689
### Describe the bug
#7316 bounds the rows per batch that reach an operator expanding a large
string or binary scalar, by putting a `CometBatchSplitExec` under it. It covers
projection, filter, hash aggregate, sort with a limit (TopK), and the hash and
range shuffle writers. Other native operators evaluate expressions over a batch
and expand a scalar the same way, but they get no split, so a large scalar can
still fail with `byte array offset overflow`.
These were raised in the review of #7316 and are not yet covered:
- **Join filter.** The residual join condition is built as a DataFusion
`JoinFilter` in `native/core/src/execution/planner/join.rs`, and nothing bounds
it. It is evaluated over the matched index pairs, so the bound is the batch
size, not the input width. A residual that references both sides stays a join
condition, for example:
```sql
SELECT count(*) FROM a JOIN b
ON a.k = b.k AND length(concat(repeat('x', 262144), a.s || b.s)) > 0
```
The same query in a plain DataFusion session (default 8192-row batch, a
hash join with one matching key) fails with `Execution error: byte array offset
overflow`. It has not yet been run end to end through Comet.
- **`ExpandExec`.** `native/operators/src/expand.rs` calls
`into_array(batch.num_rows())` on each projection expression, so a large scalar
in a grouping-sets projection is expanded to the full batch length. Not yet
reproduced.
- **Window partition and order expressions.** These are evaluated over the
batch the window receives, which can be a whole partition. Not yet reproduced.
- **Sort without a limit.** #7316 puts the split under every sort, but the
external sorter concatenates its in-memory input into larger runs before
evaluating the keys, so the split under it does not bound what the key
expressions see. Not yet reproduced.
### Steps to reproduce
See the queries above. For comparison, the cases #7316 fixed are in
`spark/src/test/resources/sql-tests/operators/large_scalar_string.sql`.
### Expected behavior
Each query returns the same result as Spark, as the operators covered by
#7316 now do.
### Additional context
Follow-up to #7247 and #7316. Part of #7267.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]