[GitHub] [spark] maryannxue commented on pull request #30829: [SPARK-33832][SQL] Add an option in AQE to mitigate skew even if it c…

2021-02-05 Thread GitBox
maryannxue commented on pull request #30829: URL: https://github.com/apache/spark/pull/30829#issuecomment-77371 @ekoifman This check is to guard against re-planning that turns an already shuffle-materialized SMJ to BHJ while causing extra shuffles downstream. One way around this

[GitHub] [spark] maryannxue commented on pull request #30829: [SPARK-33832][SQL] Add an option in AQE to mitigate skew even if it c…

2021-02-04 Thread GitBox
maryannxue commented on pull request #30829: URL: https://github.com/apache/spark/pull/30829#issuecomment-77371 @ekoifman This check is to guard against re-planning that turns an already shuffle-materialized SMJ to BHJ while causing extra shuffles downstream. One way around this