cloud-fan commented on issue #27096: [SPARK-28148][SQL] Repartition after join is not optimized away URL: https://github.com/apache/spark/pull/27096#issuecomment-572348283 an issue is that at logical phase we don't know the physical partitioning/sorting info (e.g. SMJ), so we can't optimize the user-specified sort/repartition well at logical phase. I see 2 options here: 1. run an extra physical rule after `EnsureRequirements`, to remove useless shuffles 2. do as the TODO says, create a new physical node for `RepartitionByExpression` then we can go through the main logic in the `ensureDistributionAndOrdering` method, instead of a small hack in the `apply` method.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
