andygrove opened a new issue, #7246:
URL: https://github.com/apache/datafusion-comet/issues/7246
### Describe the bug
With ANSI mode on, several native expressions evaluate an operand on rows
where Spark never evaluates it. A query that succeeds in Spark then fails in
Comet with `CAST_INVALID_INPUT` or `ARITHMETIC_OVERFLOW`.
Spark evaluates these operands per row, and only when it needs them. It
evaluates the right side of `OR` and `AND` only when the left side doesn't
decide the result, and later `IN` candidates only until one matches. The right
operand of `+` is evaluated only when the left is not NULL, and the numerator
of `DIV` only when the divisor is not NULL. With Spark's default generated
code, later `concat` arguments are evaluated only until one is NULL. Later sort
keys of a top-K are evaluated only when the earlier keys tie. Comet evaluates
all of them over the whole batch.
This is a focused follow-up under #6006. #7147 covers the related case of
rows skipped by a `LIMIT`.
### Steps to reproduce
```scala
spark.conf.set("spark.sql.ansi.enabled", "true")
spark.conf.set("spark.sql.adaptive.enabled", "false")
Seq((1, "bad"), (2, "2")).toDF("id",
"txt").repartition(1).write.parquet("/tmp/items")
Seq[(Int, Option[Int], String)]((1, None, "bad"), (2, None, "2"))
.toDF("id", "n", "txt").repartition(1).write.parquet("/tmp/items2")
Seq((1, Int.MaxValue), (2, 5), (3, Int.MaxValue), (4, 7), (5, 9))
.toDF("a", "b").repartition(1).write.parquet("/tmp/topk")
Seq("items", "items2", "topk").foreach(t =>
spark.read.parquet(s"/tmp/$t").createOrReplaceTempView(t))
```
Each table is a single Parquet file, so its rows arrive in one batch.
| Query | Spark | Comet |
| --- | --- | --- |
| `SELECT id, id = 1 OR CAST(txt AS INT) = 2 FROM items` | `(1, true), (2,
true)` | `CAST_INVALID_INPUT` |
| `SELECT id, id <> 1 AND CAST(txt AS INT) = 2 FROM items` | `(1, false),
(2, true)` | `CAST_INVALID_INPUT` |
| `SELECT id, id IN (1, CAST(txt AS INT)) FROM items` | `(1, true), (2,
true)` | `CAST_INVALID_INPUT` |
| `SELECT id, n + CAST(txt AS INT) FROM items2` | `(1, NULL), (2, NULL)` |
`CAST_INVALID_INPUT` |
| `SELECT id, CAST(txt AS INT) DIV n FROM items2` | `(1, NULL), (2, NULL)` |
`CAST_INVALID_INPUT` |
| `SELECT id, concat(IF(id = 1, CAST(NULL AS STRING), CAST(id AS STRING)),
CAST(CAST(txt AS INT) AS STRING)) FROM items` | `(1, NULL), (2, '22')` |
`CAST_INVALID_INPUT` |
| `SELECT a, b FROM topk ORDER BY a, b + 1 LIMIT 2` | `(1, 2147483647), (2,
5)` | `ARITHMETIC_OVERFLOW` |
The plans are fully native (`CometProject` or `CometTakeOrderedAndProject`
over `CometNativeScan`). The same failures happen when a literal is replaced by
a plain scalar subquery, for example `id = (SELECT max(k) FROM nums) OR
CAST(txt AS INT) = 2`.
Reproduced on `main` at c4a0b1f2bc with the default Spark 4.1 profile.
### Expected behavior
The same rows as Spark, or a fallback to Spark for these shapes.
### Additional context
#5889 makes these shapes fall back when the subquery is a merged struct
subquery, which protects only that form. A fix here would cover columns,
literals and plain subqueries too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
