lonless9 opened a new issue, #24372:
URL: https://github.com/apache/datafusion/issues/24372
### Describe the bug
Physical lambda expressions can fail during execution after an optimizer
rewrite moves the expression across a schema boundary.
This affects at least two paths:
1. collapsing consecutive `ProjectionExec`s;
2. pushing a join-filter expression below a `NestedLoopJoinExec`.
In both cases, the query plans successfully, but the physical
`LambdaVariable` retains an index and `FieldRef` that no longer match the
`RecordBatch` against which it is evaluated.
### To Reproduce
#### Reproducer 1: projection-chain collapse
```sql
SET datafusion.sql_parser.dialect = spark;
SELECT array_transform(arr, x -> x)
FROM (
SELECT arr
FROM (VALUES ([1, 2], 7)) AS t(arr, padding)
) AS q;
```
Actual result:
```text
Error: Execution error: Field of physical LambdaVariable with index 0
doesn't match batch field during evaluation Field { "x": nullable Int64 } !=
Field { "column2": Int64 }
```
The optimized physical plan has collapsed the intermediate projection:
```text
ProjectionExec
array_transform(column1, (x) -> x@1)
DataSourceExec
```
#### Reproducer 2: NestedLoopJoin filter pushdown
```sql
SET datafusion.sql_parser.dialect = spark;
SELECT l.id, r.k
FROM (
VALUES
(1, [1], 10, 20),
(2, arrow_cast([NULL], 'List(Int64)'), 30, 40)
) AS l(id, arr, padding_1, padding_2)
JOIN (VALUES (0), (1), (2)) AS r(k)
ON (cardinality(array_filter(l.arr, x -> x IS NOT NULL)) > 0)
= (l.id > r.k)
ORDER BY l.id, r.k;
```
Actual result:
```text
Error: Execution error: Field of physical LambdaVariable with index 0
doesn't match batch field during evaluation Field { "x": nullable Int64 } !=
Field { "column4": Int64 }
```
`EXPLAIN` confirms that this goes through `NestedLoopJoinExec` and
join-filter projection pushdown:
```text
NestedLoopJoinExec
ProjectionExec
arr: column2
id: column1
join_proj_push_down_1:
cardinality(
array_filter(column2, (x) -> x@3 IS NOT NULL)
) > 0
```
### Expected behavior
The first query should return:
```text
[1, 2]
```
The second query should return:
```text
1, 0
2, 2
```
### Additional context
Related **https://github.com/lakehq/sail/pull/2413**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
