Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20699#discussion_r171697012
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -613,6 +626,13 @@ object CollapseRepartition extends Rule[LogicalPlan] {
// we can remove the child.
case r @ RepartitionByExpression(_, child: RepartitionOperation, _) =>
r.copy(child = child.child)
+
+ // Case 3: When the children of the join are Repartition or
RepartitionByExpression
+ case j @ Join(left, right, _, _) if canRemoveRepartition(left)
--- End diff --
This could cause the behavior change. To properly eliminate the useless
Repartition, we need to do it in the planner .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
