tanelk commented on pull request #30093: URL: https://github.com/apache/spark/pull/30093#issuecomment-713059186
Shouldn't it be like this? ```diff diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala index c37dac90c5..79610b4ab2 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala @@ -1058,15 +1058,9 @@ object EliminateSorts extends Rule[LogicalPlan] { case s @ Sort(orders, _, child) if orders.isEmpty || orders.exists(_.child.foldable) => val newOrders = orders.filterNot(_.child.foldable) if (newOrders.isEmpty) child else s.copy(order = newOrders) - case s @ Sort(orders, global, child) - if SortOrder.orderingSatisfies(child.outputOrdering, orders) => - (global, child) match { - case (false, _) => child - case (true, r: Range) => r - case (true, s @ Sort(_, true, _)) => s - case (true, _) => s.copy(child = recursiveRemoveSort(child)) - } - case s @ Sort(_, _, child) => s.copy(child = recursiveRemoveSort(child)) + case Sort(orders, false, child) if SortOrder.orderingSatisfies(child.outputOrdering, orders) => + child + case s @ Sort(_, true, child) => s.copy(child = recursiveRemoveSort(child)) case j @ Join(originLeft, originRight, _, cond, _) if cond.forall(_.deterministic) => j.copy(left = recursiveRemoveSort(originLeft), right = recursiveRemoveSort(originRight)) case g @ Aggregate(_, aggs, originChild) if isOrderIrrelevantAggs(aggs) => ``` Two simpler rules: * Remove local sort if child satisfies the required ordering * Keep global sorts, but remove all sorts from child This fails two tests: * Range test, but it is not correct because of what @viirya said * In the "remove two consecutive global sorts with same ordering" this keeps the parent sort, but in the test it keeps the child sort. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org