mbutrovich commented on issue #4859: URL: https://github.com/apache/datafusion-comet/issues/4859#issuecomment-4917100051
Thanks @adriangb, this matches what I found independently, and the detail on why re-synthesizing `get_field` is a dead end (ordinal vs name, struct nullability, no expression form for `array<struct>`/map) lines up with what I saw. I have a working prototype of the clip you describe. It lives in `build_projection_read_plan`: when a projected column's requested type is a nested type narrower than the file's, it walks the two type trees, matches fields by name, and emits a `ProjectionMask::leaves` plus the pruned schema, instead of expanding the root to all leaves. Validated end to end in Comet (nested reader and fuzz suites pass, and `bytes_scanned` for a single nested leaf drops to a fraction of the full column). One design fork worth settling before a DataFusion PR. I key the clip off the projection expression's output type (`expr.data_type()`), which is the whole-column cast the adapter already inserts, so it needs no opener signature change. Your framing keys it off the logical-vs-physical file schema diff directly, which is more general (works even without an adapter cast) but needs the logical file schema threaded into the mask derivation. Same algorithm either way. I lean toward whichever you think reads best upstream. Agree the spec needs map key/value, case sensitivity, and field-id matching for Iceberg, and a pluggable clipping policy (mirroring the expr-adapter factory) is the right shape. Your pure-DataFusion repro (`CREATE EXTERNAL TABLE` with a narrower nested type, watch `bytes_scanned`) is exactly the integration test the PR should carry. I can share the prototype branch and repro as a starting point. Happy to collaborate on the DataFusion side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
