[GitHub] [spark] gatorsmile commented on a change in pull request #27793: [SPARK-31037][SQL] refine AQE config names
gatorsmile commented on a change in pull request #27793: [SPARK-31037][SQL]
refine AQE config names
URL: https://github.com/apache/spark/pull/27793#discussion_r388135978
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -388,99 +397,99 @@ object SQLConf {
.internal()
.doc("Adaptive query execution is skipped when the query does not have
exchanges or " +
"sub-queries. By setting this config to true (together with " +
- s"'${ADAPTIVE_EXECUTION_ENABLED.key}' enabled), Spark will force apply
adaptive query " +
+ s"'${ADAPTIVE_EXECUTION_ENABLED.key}' set to true), Spark will force
apply adaptive query " +
"execution for all supported queries.")
.version("3.0.0")
.booleanConf
.createWithDefault(false)
- val REDUCE_POST_SHUFFLE_PARTITIONS_ENABLED =
-buildConf("spark.sql.adaptive.shuffle.reducePostShufflePartitions")
- .doc(s"When true and '${ADAPTIVE_EXECUTION_ENABLED.key}' is enabled,
this enables reducing " +
-"the number of post-shuffle partitions based on map output
statistics.")
+ val ADVISORY_PARTITION_SIZE_IN_BYTES =
+buildConf("spark.sql.adaptive.advisoryPartitionSizeInBytes")
+ .doc("The advisory size in bytes of the shuffle partition during
adaptive optimization. " +
Review comment:
> The advisory size in bytes of the shuffle partition during adaptive
optimization (when '${ADAPTIVE_EXECUTION_ENABLED.key}' is true).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27793: [SPARK-31037][SQL] refine AQE config names
gatorsmile commented on a change in pull request #27793: [SPARK-31037][SQL]
refine AQE config names
URL: https://github.com/apache/spark/pull/27793#discussion_r388134549
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -378,8 +378,17 @@ object SQLConf {
.checkValue(_ > 0, "The value of spark.sql.shuffle.partitions must be
positive")
.createWithDefault(200)
+ val SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE =
+buildConf("spark.sql.adaptive.shuffle.targetPostShuffleInputSize")
+ .internal()
+ .doc("(Deprecated since Spark 3.0)")
Review comment:
Also tell users what is the new conf that replaces it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27793: [SPARK-31037][SQL] refine AQE config names
gatorsmile commented on a change in pull request #27793: [SPARK-31037][SQL]
refine AQE config names
URL: https://github.com/apache/spark/pull/27793#discussion_r388133710
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala
##
@@ -67,8 +67,8 @@ case class OptimizeSkewedJoin(conf: SQLConf) extends
Rule[SparkPlan] {
* SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE.
Review comment:
?change it to ADVISORY_PARTITION_SIZE_IN_BYTES?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
