beliefer commented on a change in pull request #28096: [SPARK-31295][DOC][FOLLOWUP] Supplement version for configuration appear in doc URL: https://github.com/apache/spark/pull/28096#discussion_r402073545
########## File path: docs/sql-performance-tuning.md ########## @@ -230,26 +241,29 @@ AQE converts sort-merge join to broadcast hash join when the runtime statistics ### Optimizing Skew Join Data skew can severely downgrade the performance of join queries. This feature dynamically handles skew in sort-merge join by splitting (and replicating if needed) skewed tasks into roughly evenly sized tasks. It takes effect when both `spark.sql.adaptive.enabled` and `spark.sql.adaptive.skewJoin.enabled` configurations are enabled. <table class="table"> - <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> + <tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr> <tr> <td><code>spark.sql.adaptive.skewJoin.enabled</code></td> <td>true</td> <td> When true and <code>spark.sql.adaptive.enabled</code> is true, Spark dynamically handles skew in sort-merge join by splitting (and replicating if needed) skewed partitions. </td> + <td>3.0.0</td> </tr> <tr> <td><code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code></td> <td>10</td> <td> A partition is considered as skewed if its size is larger than this factor multiplying the median partition size and also larger than <code>spark.sql.adaptive.skewedPartitionThresholdInBytes</code>. </td> + <td>3.0.0</td> </tr> <tr> <td><code>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</code></td> <td>256MB</td> <td> A partition is considered as skewed if its size in bytes is larger than this threshold and also larger than <code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code> multiplying the median partition size. Ideally this config should be set larger than <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code>. </td> + <td>3.0.0</td> Review comment: SPARK-31201, commit ID: 8d0800a0803d3c47938bddefa15328d654739bc5#diff-9a6b543db706f1a90f790783d6930a13 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org