hanke580 opened a new issue, #25369:
URL: https://github.com/apache/datafusion/issues/25369
### Describe the bug
An aggregate over an expression that appears twice fails to plan. Common
subexpression
elimination extracts the shared expression as `__common_expr_1`, and its
nullability is
computed one way for the logical plan and another for the physical plan; the
aggregate is
the operator that checks the two agree, so the query never runs:
```
Internal error: Physical input schema should be the same as the one
converted from logical
input schema. Differences:
- field nullability at index 0 [__common_expr_1]: (physical) true vs
(logical) false.
This issue was likely caused by a bug in DataFusion's code. Please help us
to resolve this
by filing a bug report in our issue tracker:
https://github.com/apache/datafusion/issues
```
Reproduces on **datafusion-cli 55.1.0** (latest crates.io release) and on the
**datafusion 54.0.0** Python wheel.
### To Reproduce
Three rows are enough, and no session settings are changed:
```sql
CREATE TABLE t(a INT, b INT) AS VALUES (1,1),(2,2),(3,3);
SELECT sum(CASE WHEN (CASE WHEN (a > 1 AND b < 3) THEN a ELSE 0 END)
BETWEEN 1 AND 2 THEN 1 ELSE 0 END)
FROM t;
```
`datafusion-cli -f repro.sql` raises the Internal error above.
Three ingredients are each necessary -- removing any one makes the query
plan and run
normally. Measured on 55.1.0:
| query | result |
|---|---|
| aggregate + two columns + duplicated expression (above) | **Internal
error** |
| `count()` instead of `sum()` | **Internal error** |
| `BETWEEN` written out as `>= 1 AND <= 2` | **Internal error** |
| the same expression with no aggregate | planned and ran |
| one column used twice (`a > 1 AND a < 3`) instead of two | planned and ran
|
| a single comparison, so nothing is duplicated | planned and ran |
| one condition in the inner `CASE` | planned and ran |
So it needs (1) an **aggregate** over the expression, (2) an inner `CASE`
condition
referencing **two distinct columns** -- the same column twice is fine, which
points at
nullability inference rather than at `CASE` itself -- and (3) the expression
appearing
**twice**, which is what invites CSE to extract it. `BETWEEN` duplicates it
implicitly;
writing the two comparisons out by hand fails the same way.
### Expected behavior
The query plans and returns a result, as the same expression does without
the aggregate.
For the three-row table above the answer is 1.
### Additional context
- Reproduced on **datafusion-cli 55.1.0** (built from crates.io with `cargo
install
datafusion-cli --version 55.1.0 --locked`) and on the **datafusion
54.0.0** Python wheel
(the newest on PyPI). All seven rows of the table above behave identically
on both.
- It fires regardless of batch size, target partition count, and optimizer
settings.
- I searched the tracker for the error text and for `__common_expr` together
with
nullability and did not find this reported. The nearest items look like
different
triggers: #19049 is the same message but arises from
`optimize_projections` over a
UNION (there is no union here); #15190 (CSE extracting window functions)
and #14540
(push-down vs CSE) are CSE defects but not nullability mismatches under an
aggregate;
#17801 (a TPC-DS nullability mismatch) is closed.
- Environment: Ubuntu 22.04.2 LTS, x86_64, kernel 5.15.0-187-generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
