[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356918#comment-17356918 ]
Apache Spark commented on SPARK-35639: -------------------------------------- User 'ekoifman' has created a pull request for this issue: https://github.com/apache/spark/pull/32776 > Add metrics about coalesced partitions to CustomShuffleReader in AQE > -------------------------------------------------------------------- > > Key: SPARK-35639 > URL: https://issues.apache.org/jira/browse/SPARK-35639 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Eugene Koifman > Priority: Major > > {{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number > of skewed partition splits". > It would be useful to also report "number of partitions to coalesce" and > "number of coalesced partitions" and include this in string rendering of the > SparkPlan node so that it looks like this > {code:java} > (12) CustomShuffleReader > Input [2]: [a#23, b#24] > Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org