wuyi created SPARK-36530: ---------------------------- Summary: Avoid finalizing when there's no push at all in a shuffle Key: SPARK-36530 URL: https://issues.apache.org/jira/browse/SPARK-36530 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 3.2.0, 3.3.0 Reporter: wuyi
When all partition data of a map output is bigger thanĀ spark.shuffle.push.maxBlockSizeToPush, there will be no push. When all map outputs don't have push, the shuffle doesn't have push. In that case, we don't need to launch the finalizing request. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org