Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20911#discussion_r177642168
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
---
@@ -327,18 +314,69 @@ class ComplexTypesSuite extends PlanTest with
ExpressionEvalHelper {
// but it cannot override a potential match with ('id + 2L),
// which is exactly what [[Coalesce]] would do in this case.
(Literal.TrueLiteral, 'id))) as "a")
- .analyze
- comparePlans(Optimizer execute rel, expected)
+ checkRule(rel, expected)
+ }
+
+ test("SPARK-23500: Simplify array ops that are not at the top node") {
+ val query = LocalRelation('id.long)
+ .select(
+ CreateArray(Seq(
+ CreateNamedStruct(Seq(
+ "att1", 'id,
+ "att2", 'id * 'id)),
+ CreateNamedStruct(Seq(
+ "att1", 'id + 1,
+ "att2", ('id + 1) * ('id + 1))
+ ))
+ ) as "arr")
+ .select(
+ GetStructField(GetArrayItem('arr, 1), 0, None) as "a1",
+ GetArrayItem(
+ GetArrayStructFields('arr,
+ StructField("att1", LongType, nullable = false),
+ ordinal = 0,
+ numFields = 1,
+ containsNull = false),
+ ordinal = 1) as "a2")
+ .where('id > 0L)
--- End diff --
shall we use sort? We are trying to test when the complex expressions are
not in the root node. It's hard to say if we push filter through project or not.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
