viirya opened a new pull request, #6763: URL: https://github.com/apache/datafusion-comet/pull/6763
## Which issue does this PR close? Closes #6762. ## Rationale for this change `IN` builds its candidate list once when every candidate is a constant. Both DataFusion's `InListExpr` and Comet's `spark_in_list` decide that by evaluating each candidate on an empty batch and checking for a scalar. `CaseWhenExpr`, which `IF` and `nullif` also use, returned a scalar NULL for an empty batch, because no row chose a branch. So a CASE or IF that reads a column was taken as the constant NULL, and for example `id IN (IF(id = 1, NULL, id))` returned NULL for every row. ## What changes are included in this PR? - `CaseWhenExpr::evaluate` returns an empty array of the result type for an empty batch. The check runs before both the eager evaluation from #6350 and the lazy one, because DataFusion's `CaseExpr` also returns a scalar NULL there when there is no ELSE. As a result, a CASE made only of literals is no longer treated as a constant by IN. That only affects performance, and Spark folds such a CASE before Comet sees it. - `spark_in_list` no longer treats a candidate that reads a column as a constant, whatever it returns for the empty batch. I also looked for other Comet expressions that return a scalar for an empty batch while reading a column, and found none. The other `PhysicalExpr` implementations and scalar functions return a scalar only when all their inputs are scalars, or when a NULL scalar argument makes every row NULL. ## How are these changes tested? - Rust unit tests: CASE and IF return an empty array for an empty batch on both the eager and the lazy path; DataFusion's `in_list` over an IF or CASE candidate gives Spark's results; and `spark_in_list` keeps a column-reading candidate dynamic, both for a CASE and for an expression that returns a scalar for the empty batch. They fail without the fix. - `expressions/conditional/in_case_when_candidate.sql` runs the queries from the issue with native `range`: IF, `nullif`, CASE with and without ELSE, `NOT IN`, and struct and array operands, including a double leaf so that `spark_in_list` is used. It fails without the fix. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
