amitvijapur opened a new pull request, #25633: URL: https://github.com/apache/datafusion/pull/25633
## Which issue does this PR close? - Closes #21795. ## Rationale for this change `WHERE struct_col IS NOT NULL` cannot be pushed into the Parquet scan today: `PushdownChecker` rejects any filter that references a whole struct column, so the check stays in a `FilterExec` above the scan and every leaf of the struct is decoded just to read its null bitmap. #20822 and #20854 added pushdown for `get_field` on struct fields; the null check on the struct itself was left out. ## What changes are included in this PR? The checker now recognises `IS NULL` / `IS NOT NULL` whose argument is a `Column` of struct type and records it as a `StructFieldAccess` to the struct's first non-struct descendant, so the existing `get_field` read plan projects that one leaf. Parquet encodes nullability at every nesting level in the definition levels, which each leaf carries, so the decoder rebuilds the struct's null bitmap from that single leaf. Other whole-struct references (for example `s = s`) are still rejected. One consequence to be aware of: once the predicate reaches `DataSourceExec`, a row group pruning predicate `s_null_count != row_count` is built for it. That would be wrong if a struct's null count were taken from a leaf's statistics, because a non-null struct with a null first field counts toward the leaf. It is not: struct statistics are reported as unknown, and the slt below pins the case where a whole row group has non-null structs with a null first leaf. ## Are these changes tested? Unit tests cover both checks being pushable, a whole-struct reference (`struct_col = struct_col`) still being rejected, the projection mask selecting exactly one of a struct's three leaves, and `first_leaf_path` for nested, list-child, empty-first-child and non-struct inputs. `parquet_filter_pushdown.slt` gains the `EXPLAIN`, both null checks, a null check combined with a field predicate, and the two-row-group pruning case. `cargo test -p datafusion-datasource-parquet` and the full sqllogictest suite pass. ## Are there any user-facing changes? No API change. A null check on a struct column is now evaluated during Parquet decoding and reads one leaf of the struct instead of all of them. Written with AI assistance (Claude Code); I verified the single-leaf null bitmap premise, the pruning-predicate behaviour and the test results myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
