andygrove opened a new issue, #7247:
URL: https://github.com/apache/datafusion-comet/issues/7247
### Describe the bug
When a large string scalar, from a scalar subquery or a literal, is expanded
to the batch length, the result can exceed Arrow's 32-bit offsets. With a 256
KiB string and the default 8,192-row batch, the expanded column needs 2 GiB of
value bytes. Native execution then panics with `offset overflow`, or fails with
`byte array offset overflow`. Spark evaluates the same queries row by row and
succeeds.
### Steps to reproduce
```scala
spark.conf.set("spark.sql.adaptive.enabled", "false")
sql("SELECT repeat('x', 262144) AS
s").repartition(1).write.parquet("/tmp/wide")
sql("SELECT CAST(id % 2 AS INT) AS k FROM range(8192)")
.repartition(1).write.parquet("/tmp/outer_ids")
Seq("wide", "outer_ids").foreach(t =>
spark.read.parquet(s"/tmp/$t").createOrReplaceTempView(t))
```
| Query | Spark | Comet |
| --- | --- | --- |
| `SELECT k, count((SELECT max(s) FROM wide)) FROM outer_ids GROUP BY k` |
`(0, 4096), (1, 4096)` | `native panic: offset overflow` |
| `SELECT count(*) FROM (SELECT k FROM outer_ids DISTRIBUTE BY k, (SELECT
max(s) FROM wide))` | `8192` | `native panic: offset overflow` |
| `SELECT sum(length(c)) FROM (SELECT concat((SELECT max(s) FROM wide),
CAST(k AS STRING)) AS c FROM outer_ids)` | `2147491840` | `byte array offset
overflow` |
| `SELECT sum(length(c)) FROM (SELECT concat(repeat('x', 262144), CAST(k AS
STRING)) AS c FROM outer_ids)` | `2147491840` | `byte array offset overflow` |
The last query has no subquery. The literal is enough. All four plans run
natively, through `CometHashAggregate`, `CometExchange` and `CometProject`.
Reproduced on `main` at c4a0b1f2bc with the default Spark 4.1 profile.
### Expected behavior
The same results as Spark.
### Additional context
`count` of a scalar doesn't need the expanded column, and a hash partitioner
could hash a scalar once. For kernels whose output really exceeds 2 GiB, the
batch would have to be split, use 64-bit offsets, or fall back. #4718 and #7192
are other offset-size failures. #5889 adds bounded evaluation for merged struct
subqueries only, so these plain-subquery and literal shapes still fail on that
branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
