adriangb opened a new issue, #25305:
URL: https://github.com/apache/datafusion/issues/25305
### Describe the bug
`named_struct(...)` and `struct(...)` can never return a NULL row:
`invoke_with_args` builds the output `StructArray` with no null buffer.
However, both functions report a **nullable** return field. Since
`ExprSchemable::nullable` for a scalar function reads that field, the
simplifier rule that rewrites `a IS NOT NULL` to `true` when `a` is not
nullable never fires for these constructors.
The user-visible effect is that a guard such as `WHERE s IS NOT NULL` on a
struct built by a view keeps the entire struct expression alive. Projection
pushdown can then no longer prune the scan down to the fields that are actually
read, even though the existing `get_field(named_struct(...), 'f')`
simplification (#22239) would otherwise make the struct disappear entirely.
### To Reproduce
Run with `datafusion-cli -f repro.sql` (DataFusion 55.1.0, `main` at the
time of writing):
```sql
SET datafusion.explain.format = 'indent';
CREATE TABLE t (a INT, b INT, c INT) AS VALUES (1, 2, 3), (NULL, 5, 6);
CREATE VIEW v AS SELECT named_struct('a', a, 'b', b, 'c', c) AS s FROM t;
-- prunes to projection=[b]
EXPLAIN SELECT s['b'] FROM v;
-- same query with a redundant guard: the whole named_struct is kept
EXPLAIN SELECT s['b'] FROM v WHERE s IS NOT NULL;
EXPLAIN SELECT CASE WHEN named_struct('a', a) IS NOT NULL THEN b END FROM t;
```
### Current behavior
The unguarded query prunes correctly:
```
logical_plan
Projection: __datafusion_extracted_1 AS v.s[b]
SubqueryAlias: v
Projection: t.b AS __datafusion_extracted_1
TableScan: t projection=[b]
```
Adding `WHERE s IS NOT NULL`, which cannot remove any row, defeats the
pruning and reads all three columns:
```
logical_plan
Projection: __datafusion_extracted_1 AS v.s[b]
SubqueryAlias: v
Projection: __datafusion_extracted_1
Filter: named_struct(Utf8("a"), t.a, Utf8("b"), t.b, Utf8("c"), t.c)
IS NOT NULL
Projection: t.b AS __datafusion_extracted_1, t.a, t.b, t.c
TableScan: t projection=[a, b, c]
```
The `CASE WHEN` form keeps the constructor and the extra column too:
```
logical_plan
Projection: CASE WHEN named_struct(Utf8("a"), t.a) IS NOT NULL THEN t.b END
TableScan: t projection=[a, b]
```
### Expected behavior
`IS NOT NULL` on a struct constructor should fold to `true` (and `IS NULL`
to `false`), because the constructor never produces a NULL row. The guard
should then disappear and the scan should read only `b` in all three plans:
```
logical_plan
Projection: __datafusion_extracted_1 AS v.s[b]
SubqueryAlias: v
Projection: t.b AS __datafusion_extracted_1
TableScan: t projection=[b]
```
```
logical_plan
Projection: t.b AS CASE WHEN named_struct(Utf8("a"),t.a) IS NOT NULL THEN
t.b END
TableScan: t projection=[b]
```
### Cause
`return_field_from_args` in `datafusion/functions/src/core/named_struct.rs`
marks the returned `Field` nullable, and `struct.rs` relies on the default
`return_field_from_args`, which does the same. Both should report a
non-nullable field to match what `invoke_with_args` actually builds.
### Additional context
DataFusion 55.1.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
