adriangb commented on issue #4859:
URL:
https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4916909423
I've spent a bit exploring this issue with Claude and these are the
conclusions we came to. I reviewed them and they make sense to me, I'm not sure
what the path forward is though. Keep in mind I am not familiar with Spark or
Comet.
This largely confirms @mixermt's analysis and adds some detail on *why* the
gap exists and why the obvious workarounds don't apply.
### Why nested pruning works in plain DataFusion but not here
DataFusion *can* prune nested leaves — `SELECT s.foo FROM t` in
datafusion-cli only reads the `s.foo` leaf. But that works because the pruning
is carried **in the expression**: the projection is `get_field(Column("s"),
"foo")`, and `build_projection_read_plan` in
`datafusion-datasource-parquet/src/row_filter.rs` translates literal
`get_field` chains into a `ProjectionMask::leaves`.
Spark communicates nested pruning completely differently. The
`SchemaPruning` optimizer rule *consumes* the `GetStructField` expressions and
bakes the result into the scan node as a narrower `requiredSchema` (a
`Column("events")` whose *type* is a narrower `array<struct<...>>` than the
file's); the remaining downstream field accesses are rebased against the pruned
scan output. So at the scan boundary the pruning exists only as a **type**, not
as expressions — there is nothing for Comet to translate into `get_field`.
### What DataFusion does with a pruned schema today (DF 54, still true on
current main)
When the logical file schema says `events: array<struct<3 fields>>` but the
physical file has `array<struct<18 fields>>`:
1. The `DefaultPhysicalExprAdapter` rewrites the projection against the
physical file schema, turning the type mismatch into a whole-column
`CastExpr(Column("events"), narrow_type)`
(`datafusion-physical-expr-adapter/src/schema_rewriter.rs`).
2. `build_projection_read_plan` derives the projection mask from the
rewritten expressions. Its only sources of leaf-level masks are literal
`get_field` chains (and struct-field filter accesses). A `CastExpr(Column)`
falls through to the generic path, which records the root column and expands it
to **every physical leaf** under `events`.
3. The full column chunks are fetched and decoded; the cast drops the
unwanted subfields in memory. This is exactly why @mixermt measured identical
`bytes_scanned` with the pruned schema — the output is fixed, the I/O is not.
The parquet reader itself is not the limitation: parquet-rs
`ProjectionMask::leaves` handles arbitrary nesting, and DataFusion already uses
it for pushed-down struct filters. The gap is purely that nothing translates a
*schema-level* diff (logical file schema narrower than physical) into a leaf
mask.
### Why Comet can't just re-synthesize `get_field` expressions
Even setting aside the `array<struct>` problem, translating the pruned
schema back into DataFusion expressions is lossy:
- **Placement**: the leaf-mask derivation only looks at expressions inside
the `DataSourceExec`'s own projection. Comet builds the scan with an index
projection and keeps its projection logic in a separate `ProjectionExec` using
its own ordinal-based `GetStructField` expression, which DataFusion's
`PushdownChecker` can't recognize (it downcasts specifically to
`GetFieldFunc`). Comet would have to push DF-native `get_field` exprs into the
scan via `FileSource::try_pushdown_projection`.
- **Ordinal vs. name resolution**: Spark resolves struct fields by ordinal
and Parquet structs may contain duplicate or case-colliding field names;
`get_field` is name-based. (This appears to be why Comet's `GetStructField` is
ordinal-based in the first place.)
- **Struct-level nullability**: the scan must still emit the struct column,
so the expression encoding is `named_struct('foo', get_field(s, 'foo'))` — but
if `s` is NULL that yields a non-null `{foo: NULL}` instead of NULL, silently
changing `s IS NULL` semantics. A clipped-schema read preserves struct nullness
via the surviving leaves' definition levels.
- **It stops at structs**: there is no expression form at all for pruning
subfields of `array<struct>` elements or map values — which is the shape in
this report. (Element access like `events[1].foo` is orthogonal: pruning
`list<struct<foo>>` means "only `foo`, for every element"; Parquet shreds
nested data into one column chunk per leaf, so leaf selection is the only
prunable axis at the schema level.)
### What seems needed upstream in DataFusion
Schema-driven nested pruning in the parquet opener: when a required root
column's logical file-schema type is a structural subset of its physical type,
compute the leaf mask by walking the two type trees and matching fields by name
— the equivalent of Spark's `ParquetReadSupport.clipParquetSchema` — instead of
expanding the root to all leaves. After clipping, the decoder emits the narrow
type directly and the adapter-inserted cast degrades to a cheap adjustment,
while still handling schema evolution (missing subfield → null fill,
reordering, promotion). Edge cases to spec: map key/value handling, case
sensitivity, and field-id-based matching for Iceberg (possibly making the
clipping policy pluggable, like the expr adapter factory already is).
This can be reproduced and tested in pure DataFusion with no Comet involved:
write a parquet file with a wide `struct` (or `array<struct>`) column, `CREATE
EXTERNAL TABLE` over it declaring a *narrower* nested type, select that column,
and observe `bytes_scanned` — the full column chunk is read despite the narrow
table schema. That isolates the schema-diff path from the `get_field` path that
already works.
An interim alternative would be an API for `ParquetSource` callers to supply
the leaf-level projection directly (per-file `ProjectionMask` hook or an
authoritative "read schema"), which Comet could drive itself — but the
diff-based approach would benefit every embedder that hands DataFusion a
pre-pruned schema (delta-rs, Iceberg integrations, etc.).
Happy to help move the DataFusion side forward if there's agreement on the
direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
