This is an automated email from the ASF dual-hosted git repository. jshao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
commit 6bdf49e1a68131545a8385123da558be287a196f Author: xunxunmimi5577 <52647492+xunxunmimi5...@users.noreply.github.com> AuthorDate: Fri Jun 24 02:12:40 2022 +0800 [Improvement] Check ADAPTIVE_EXECUTION_ENABLED in RssShuffleManager (#197) ### What changes were proposed in this pull request? 1. Add checking of spark.sql.adaptive.enabled=false in RssShuffleManager's constructor for spark2. 2. Add a description of this parameter in the Deploy Spark Client section of the readme. ### Why are the changes needed? When use firestorm+spark2+spark.sql.adaptive.enabled=true,the result is wrong,but we didn't get any hints. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test --- README.md | 1 + .../src/main/java/org/apache/spark/shuffle/RssShuffleManager.java | 3 +++ 2 files changed, 4 insertions(+) diff --git a/README.md b/README.md index 50903ce..0fb65e5 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,7 @@ rss-xxx.tgz will be generated for deployment ``` spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager spark.rss.coordinator.quorum <coordinatorIp1>:19999,<coordinatorIp2>:19999 + # Note: For Spark2, spark.sql.adaptive.enabled should be false because Spark2 doesn't support AQE. ``` ### Support Spark dynamic allocation diff --git a/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java b/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java index 8a2c385..28f1a8d 100644 --- a/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java +++ b/client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java @@ -136,6 +136,9 @@ public class RssShuffleManager implements ShuffleManager { }; public RssShuffleManager(SparkConf sparkConf, boolean isDriver) { + if (sparkConf.getBoolean("spark.sql.adaptive.enabled", false)) { + throw new IllegalArgumentException("Spark2 doesn't support AQE, spark.sql.adaptive.enabled should be false."); + } this.sparkConf = sparkConf; // set & check replica config