adriangb opened a new pull request, #25306:
URL: https://github.com/apache/datafusion/pull/25306
## Which issue does this PR close?
- Closes #25305.
## Rationale for this change
`named_struct(...)` and `struct(...)` never produce a NULL row:
`invoke_with_args` builds the output `StructArray` with no null buffer. Both
functions nevertheless reported a nullable return field, and
`ExprSchemable::nullable` for a scalar function reads that field. As a result
the simplifier rule that rewrites `a IS NOT NULL` to `true` for a non-nullable
`a` never fired on a struct constructor.
The user-visible cost is lost pruning. A guard such as `WHERE s IS NOT NULL`
on a struct built by a view cannot remove any row, but it keeps the whole
struct expression alive, so projection pushdown can no longer prune the scan to
the fields that are read:
```sql
SET datafusion.explain.format = 'indent';
CREATE TABLE t (a INT, b INT, c INT) AS VALUES (1, 2, 3), (NULL, 5, 6);
CREATE VIEW v AS SELECT named_struct('a', a, 'b', b, 'c', c) AS s FROM t;
EXPLAIN SELECT s['b'] FROM v WHERE s IS NOT NULL;
```
Before:
```
logical_plan
Projection: __datafusion_extracted_1 AS v.s[b]
SubqueryAlias: v
Projection: __datafusion_extracted_1
Filter: named_struct(Utf8("a"), t.a, Utf8("b"), t.b, Utf8("c"), t.c)
IS NOT NULL
Projection: t.b AS __datafusion_extracted_1, t.a, t.b, t.c
TableScan: t projection=[a, b, c]
```
After:
```
logical_plan
Projection: __datafusion_extracted_1 AS v.s[b]
SubqueryAlias: v
Projection: t.b AS __datafusion_extracted_1
TableScan: t projection=[b]
```
This matches the plan the same query already gets without the guard, and
lets the existing `get_field(named_struct(...), 'f')` simplification (#22239)
remove the constructor.
## What changes are included in this PR?
- `datafusion/functions/src/core/named_struct.rs`: `return_field_from_args`
now reports a non-nullable field instead of a nullable one.
- `datafusion/functions/src/core/struct.rs`: adds a `return_field_from_args`
override that reports a non-nullable field. The default implementation it
previously used made the field nullable.
Both are a one-line nullability correction; the inner struct fields stay
nullable, since an individual member value can still be NULL.
## What is the testing strategy for this PR?
- New `sqllogictest` cases at the end of
`datafusion/sqllogictest/test_files/struct.slt` covering the folded scalar
results (`named_struct(...) IS NOT NULL`, `struct(...) IS NULL`), the `EXPLAIN`
for `WHERE s IS NOT NULL` over a view showing `TableScan: t projection=[b]`,
the values that query returns, and the `CASE WHEN named_struct(...) IS NOT
NULL` fold.
- Existing `datafusion-functions` unit tests: `cargo test -p
datafusion-functions --lib`, 356 passed.
- The full `sqllogictest` suite: all 518 files pass, with no other
expectation changes needed.
## Are there any user-facing changes?
Yes, but no API change.
- The schema nullability of the output of `named_struct()` and `struct()`
changes from nullable to non-nullable. This now matches the array those
functions actually build.
- `IS NULL` and `IS NOT NULL` applied directly to one of these constructors
constant-fold to `false` and `true`. Query results are unchanged; plans get
smaller and can prune more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
