Eugene Koifman created SPARK-35639: -------------------------------------- Summary: Add metrics about coalesced partitions to CustomShuffleReader in AQE Key: SPARK-35639 URL: https://issues.apache.org/jira/browse/SPARK-35639 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.0.0 Reporter: Eugene Koifman
{{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number of skewed partition splits". It would be useful to also report "number of partitions to coalesce" and "number of coalesced partitions" and include this in string rendering of the SparkPlan node so that it looks like this {code:java} (12) CustomShuffleReader Input [2]: [a#23, b#24] Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4 {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org