[GitHub] [spark] viirya commented on a change in pull request #24599: [SPARK-27701][SQL] Extend NestedColumnAliasing to more nested field cases
viirya commented on a change in pull request #24599: [SPARK-27701][SQL] Extend
NestedColumnAliasing to more nested field cases
URL: https://github.com/apache/spark/pull/24599#discussion_r291924205
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -86,32 +86,39 @@ object NestedColumnAliasing {
}
/**
- * Return root references that are individually accessed as a whole, and
`GetStructField`s.
+ * Return root references that are individually accessed as a whole, and
`GetStructField`s
+ * or `GetArrayStructField`s which on top of other `ExtractValue`s or
special expressions.
+ * Check `SelectedField` to see which expressions should be listed here.
*/
- private def collectRootReferenceAndGetStructField(e: Expression):
Seq[Expression] = e match {
-case _: AttributeReference | _: GetStructField => Seq(e)
-case es if es.children.nonEmpty =>
es.children.flatMap(collectRootReferenceAndGetStructField)
+ private def collectRootReferenceAndExtractValue(e: Expression):
Seq[Expression] = e match {
+case _: AttributeReference |
+ GetStructField(_: ExtractValue | _: AttributeReference, _, _) |
+ GetArrayStructFields(_: MapValues |
+ _: MapKeys |
+ _: ExtractValue |
+ _: AttributeReference, _, _, _, _) => Seq(e)
Review comment:
Ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24599: [SPARK-27701][SQL] Extend NestedColumnAliasing to more nested field cases
viirya commented on a change in pull request #24599: [SPARK-27701][SQL] Extend
NestedColumnAliasing to more nested field cases
URL: https://github.com/apache/spark/pull/24599#discussion_r291924153
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
##
@@ -31,7 +31,7 @@ import org.apache.spark.sql.types._
object NestedColumnAliasing {
def unapply(plan: LogicalPlan)
-: Option[(Map[GetStructField, Alias], Map[ExprId, Seq[Alias]])] = plan
match {
+: Option[(Map[ExtractValue, Alias], Map[ExprId, Seq[Alias]])] = plan match
{
case Project(projectList, child)
if SQLConf.get.nestedSchemaPruningEnabled &&
canProjectPushThrough(child) =>
Review comment:
`NestedColumnAliasingSuite` enables
`spark.sql.optimizer.nestedSchemaPruning.enabled`. Is this you asked?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24599: [SPARK-27701][SQL] Extend NestedColumnAliasing to more nested field cases
viirya commented on a change in pull request #24599: [SPARK-27701][SQL] Extend
NestedColumnAliasing to more nested field cases
URL: https://github.com/apache/spark/pull/24599#discussion_r286286753
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
##
@@ -25,7 +25,7 @@ import org.apache.spark.sql.catalyst.dsl.plans._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.RuleExecutor
-import org.apache.spark.sql.types.{StringType, StructType}
+import org.apache.spark.sql.types.{StringType, StructField, StructType}
class NestedColumnAliasingSuite extends SchemaPruningTest {
Review comment:
There still are many usage of `GetStructField` in this test suite. Maybe
make a minor PR to rewrite them.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
