This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push: new 842fbed [SPARK-31442][SQL] Print shuffle id at coalesce partitions target size 842fbed is described below commit 842fbed1d43f2ce5224d1367a7891f636243382e Author: ulysses <youxi...@weidian.com> AuthorDate: Sat Apr 18 09:27:44 2020 -0700 [SPARK-31442][SQL] Print shuffle id at coalesce partitions target size ### What changes were proposed in this pull request? Minor change. Print shuffle id. ### Why are the changes needed? Make log more clear. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Not need. Closes #28211 from ulysses-you/print-shuffle-id. Authored-by: ulysses <youxi...@weidian.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> (cherry picked from commit 6c2bf8248a439951fe5157b34ad269cb31df0e16) Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- .../apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala index 208cc05..e10ed4f 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala @@ -68,7 +68,9 @@ object ShufflePartitionsUtil extends Logging { math.ceil(totalPostShuffleInputSize / minNumPartitions.toDouble).toLong, 16) val targetSize = math.min(maxTargetSize, advisoryTargetSize) - logInfo(s"advisory target size: $advisoryTargetSize, actual target size $targetSize.") + val shuffleIds = mapOutputStatistics.map(_.shuffleId).mkString(", ") + logInfo(s"For shuffle($shuffleIds), advisory target size: $advisoryTargetSize, " + + s"actual target size $targetSize.") // Make sure these shuffles have the same number of partitions. val distinctNumShufflePartitions = --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org