Yuchuan Huang created SPARK-60768:
-------------------------------------
Summary: Skip nested schema pruning in DSv2 column pruning when no
nested field is requested
Key: SPARK-60768
URL: https://issues.apache.org/jira/browse/SPARK-60768
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yuchuan Huang
The v1 SchemaPruning rule only prunes the nested schema when at least one
requested root field is a nested field access, i.e.
requestedRootFields.exists(!_.derivedFromAtt). Otherwise, it skips the pruning.
PushDownUtils.pruneColumns for DSv2 always calls SchemaPruning.pruneSchema
whenever any root field is requested, even when all requested root fields are
top-level attributes. In that case the nested pruning is a no-op that only adds
the cost of merging and sorting the requested fields against the relation
schema. This ticket adds the same check to DSv2 for parity with v1.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
