chenliang created SPARK-26543: --------------------------------- Summary: Support the coordinator to demerminte post-shuffle partitions more reasonably Key: SPARK-26543 URL: https://issues.apache.org/jira/browse/SPARK-26543 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.3.2, 2.3.1, 2.3.0, 2.2.2, 2.2.1, 2.2.0 Reporter: chenliang Fix For: 2.3.0
For SparkSQL ,when we open AE by 'set spark.sql.adapative.enable=true',the ExchangeCoordinator will introduced to determine the number of post-shuffle partitions. But in some certain conditions,the coordinator performed not very well, there are always some tasks retained and they worked with Shuffle Read Size / Records 0.0B/0 ,We could increase the spark.sql.adaptive.shuffle.targetPostShuffleInputSize to solve this,but this action is unreasonable as targetPostShuffleInputSize Should not be set too large. We can filter the useless partition(0B) with ExchangeCoorditinator automatically -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org