viirya opened a new issue, #6762:
URL: https://github.com/apache/datafusion-comet/issues/6762
### Describe the bug
`IN` returns wrong results, without any error, when a candidate is an `IF`,
`CASE WHEN` or `nullif` that reads a column.
```sql
SELECT id, id IN (IF(id = 1, NULL, id)), id IN (nullif(id, 1), 5L) FROM
range(0, 3)
```
Spark returns `[0,true,true]`, `[1,null,null]`, `[2,true,true]`. Comet
returns NULL in both `IN` columns for every row. `id IN (CASE WHEN id = 1 THEN
NULL ELSE id END)` is wrong the same way. So are nested operands:
`named_struct('a', id) IN (named_struct('a', IF(id = 1, NULL, id)))` returns
false where Spark returns true.
DataFusion's `InListExpr::try_new` (`try_evaluate_constant_list`) and
Comet's `spark_in_list` both decide that the candidate list is constant by
evaluating each candidate on an empty `RecordBatch` and checking that it
returns a `ColumnarValue::Scalar`. In that case they build the list once and
use it for every row.
`CaseWhenExpr`, which `IF` and `nullif` also use, returns
`ColumnarValue::Scalar(NULL)` for a 0-row batch: no row chooses a branch, so
the eager path from #6350 ends with no branches and returns a scalar NULL.
DataFusion's lazy `CaseExpr` does the same when there is no ELSE. So the
column-dependent candidate is taken as the constant NULL. `CreateNamedStruct`
returns a scalar when all its children do, so the nested case goes wrong the
same way.
### Steps to reproduce
Run the query above with Comet and these configs, so that the projection
runs natively over `range`:
```
spark.comet.exec.range.enabled=true
spark.comet.sparkToColumnar.enabled=true
spark.comet.sparkToColumnar.supportedOperatorList=Range
```
Seen on Spark 4.1.
### Expected behavior
The same results as Spark. A CASE or IF that reads a column should return an
empty array for an empty batch, not a scalar, so that `IN` does not treat it as
a constant.
### Additional context
_No response_
### Willingness to contribute
I can contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
