cloud-fan commented on a change in pull request #33172:
URL: https://github.com/apache/spark/pull/33172#discussion_r662468622
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -525,10 +525,35 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ private val MIN_PARTITION_SIZE_KEY =
"spark.sql.adaptive.coalescePartitions.minPartitionSize"
+
+ val COALESCE_PARTITIONS_PARALLELISM_FIRST =
+ buildConf("spark.sql.adaptive.coalescePartitions.parallelismFirst")
+ .doc("When true, Spark ignores the target size specified by " +
+ s"'${ADVISORY_PARTITION_SIZE_IN_BYTES.key}' (default 64MB) when
coalescing contiguous " +
+ "shuffle partitions, and only respect the minimum partition size
specified by " +
+ s"'$MIN_PARTITION_SIZE_KEY' (default 1MB), to maximize the
parallelism. " +
+ "This is to avoid performance regression when enabling adaptive query
execution. " +
+ "It's recommended to set this config to false and respect the target
size specified by " +
+ s"'${ADVISORY_PARTITION_SIZE_IN_BYTES.key}'.")
Review comment:
@viirya is it clear to you now?
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -525,10 +525,35 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ private val MIN_PARTITION_SIZE_KEY =
"spark.sql.adaptive.coalescePartitions.minPartitionSize"
+
+ val COALESCE_PARTITIONS_PARALLELISM_FIRST =
+ buildConf("spark.sql.adaptive.coalescePartitions.parallelismFirst")
+ .doc("When true, Spark ignores the target size specified by " +
+ s"'${ADVISORY_PARTITION_SIZE_IN_BYTES.key}' (default 64MB) when
coalescing contiguous " +
+ "shuffle partitions, and only respect the minimum partition size
specified by " +
+ s"'$MIN_PARTITION_SIZE_KEY' (default 1MB), to maximize the
parallelism. " +
+ "This is to avoid performance regression when enabling adaptive query
execution. " +
+ "It's recommended to set this config to false and respect the target
size specified by " +
+ s"'${ADVISORY_PARTITION_SIZE_IN_BYTES.key}'.")
Review comment:
@viirya is it clearer to you now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
