maropu commented on a change in pull request #28560:
URL: https://github.com/apache/spark/pull/28560#discussion_r428662926
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##########
@@ -48,7 +53,10 @@ object NestedColumnAliasing {
case Project(projectList, child) =>
Project(
getNewProjectList(projectList, nestedFieldToAlias),
- replaceChildrenWithAliases(child, attrToAliases))
+ replaceChildrenWithAliases(child, nestedFieldToAlias, attrToAliases))
+
+ case other =>
Review comment:
This case only matches `Aggregate` and `Expand` now?
##########
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
##########
@@ -341,6 +341,78 @@ class NestedColumnAliasingSuite extends SchemaPruningTest {
.analyze
comparePlans(optimized, expected)
}
+
+ test("Nested field pruning for Aggregate") {
+ val query1 =
contact.groupBy($"id")(first($"name.first").as("first")).analyze
Review comment:
Could you add tests for the cases,
`contact.(limit/repartition/sample).groupBy()...`? IIUC this PR could support
them, too?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##########
@@ -35,6 +35,11 @@ object NestedColumnAliasing {
case Project(projectList, child)
if SQLConf.get.nestedSchemaPruningEnabled &&
canProjectPushThrough(child) =>
getAliasSubMap(projectList)
+
+ case plan if SQLConf.get.nestedSchemaPruningEnabled && canPruneOn(plan) =>
+ val exprsToPrune = plan.expressions
Review comment:
nit: the consistent var name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
