mbutrovich opened a new pull request, #4866: URL: https://github.com/apache/datafusion-comet/pull/4866
## Which issue does this PR close? Relates to #4859. (Draft: it validates the fix end to end but should not merge until the DataFusion change is released. See below.) ## Rationale for this change Comet's native Parquet scan read entire nested columns even when Spark's plan showed a pruned nested `ReadSchema`, so a query needing one leaf of a wide `array<struct<...>>` fetched and decoded every leaf (#4859 measured ~1.35 TB vs plain Spark's ~30.9 GB for the same files and rows). The root cause is in DataFusion, not Comet. Spark's `SchemaPruning` bakes nested pruning into the scan's `requiredSchema` as a narrower type, and Comet already forwards that pruned schema. DataFusion's projection-mask derivation only pruned nested leaves for `get_field` expression chains and expanded any other projection (including the whole-column cast the schema adapter inserts for a narrowed nested type) to every leaf of the top-level column. So the pruning was correct in the output but not in the I/O. The fix is upstream in apache/datafusion#23391, which derives a leaf-level `ProjectionMask` when a projected column's requested type is a nested subtree of the file's type. ## What changes are included in this PR? No Comet production code changes. Comet already forwards the pruned schema, so the fix is entirely on the DataFusion side. This draft points the DataFusion dependencies at a fork branch that is DataFusion 54.0.0 plus the fix (via `[patch.crates-io]` in `native/Cargo.toml`, with the regenerated `native/Cargo.lock`), so CI can exercise the change end to end. That patch and lock are temporary and will be replaced by a normal DataFusion version bump once the fix lands in a release, at which point this closes #4859. Not for merge as-is. It also adds regression tests to `CometNativeReaderSuite`. ## How are these changes tested? New tests in `CometNativeReaderSuite` write an `array<struct<...>>` (and a deeper `array<struct<struct<...>>>`) where a high-entropy leaf dominates the column size, then assert `CometNativeScanExec`'s `bytes_scanned` when projecting only the small leaf is a fraction of projecting the large leaf too. Before the fix these were equal (whole column read). Locally, `CometFuzzTestSuite` and `CometNativeReaderSuite` pass against the fork branch, and this PR runs full CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
