Github user heary-cao commented on a diff in the pull request:
https://github.com/apache/spark/pull/20541#discussion_r167113352
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -400,13 +400,24 @@ object PushProjectionThroughUnion extends
Rule[LogicalPlan] with PredicateHelper
// Push down deterministic projection through UNION ALL
case p @ Project(projectList, Union(children)) =>
assert(children.nonEmpty)
- if (projectList.forall(_.deterministic)) {
- val newFirstChild = Project(projectList, children.head)
+ val (deterministicList, nonDeterministic) =
projectList.partition(_.deterministic)
+
+ if (deterministicList.nonEmpty) {
+ val newFirstChild = Project(deterministicList, children.head)
val newOtherChildren = children.tail.map { child =>
val rewrites = buildRewrites(children.head, child)
- Project(projectList.map(pushToRight(_, rewrites)), child)
+ Project(deterministicList.map(pushToRight(_, rewrites)), child)
--- End diff --
if we push a + 1 to union, just a + 1.
seems this lack of test cases. let's add this test cases. thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
