[
https://issues.apache.org/jira/browse/SPARK-55052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-55052:
-----------------------------------
Labels: pull-request-available (was: )
> Add AQEShuffleRead properties to Physical Plan Tree
> ---------------------------------------------------
>
> Key: SPARK-55052
> URL: https://issues.apache.org/jira/browse/SPARK-55052
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Eren Avsarogullari
> Priority: Major
> Labels: pull-request-available
>
> AQEShuffleRead can have *local / coalesced / skewed* / *coalesced and skewed*
> properties when reading shuffle files. When Physical Plan Tree is complex, it
> is hard to track this info by correlating with AQEShuffleRead details such as
> which AQEShuffleRead has local read or skewed partition info etc. For
> example, following skew SMJ case, this helps to understand which SMJ leg has
> AQEShuffleRead with skew. This addition aims to help this kind of use-cases
> at physical plan tree level. Plan Tree details section per AQEShuffleRead
> node also shows these properties but when query plan tree is too complex
> (e.g: composed by 1000+ physical nodes), it is hard to correlate this
> information with AQEShuffleRead details.
> *Current Physical Plan Tree:*
> {code:java}
> == Physical Plan ==
> AdaptiveSparkPlan (24)
> +- == Final Plan ==
> ResultQueryStage (17), Statistics(sizeInBytes=8.0 EiB)
> +- * Project (16)
> +- * SortMergeJoin(skew=true) Inner (15)
> :- * Sort (7)
> : +- AQEShuffleRead (6)
> : +- ShuffleQueryStage (5), Statistics(sizeInBytes=15.6 KiB,
> rowCount=1.00E+3)
> : +- Exchange (4)
> : +- * Project (3)
> : +- * Filter (2)
> : +- * Range (1)
> +- * Sort (14)
> +- AQEShuffleRead (13)
> +- ShuffleQueryStage (12), Statistics(sizeInBytes=3.1 KiB,
> rowCount=200)
> +- Exchange (11)
> +- * Project (10)
> +- * Filter (9)
> +- * Range (8){code}
> *New Physical Plan Tree:*
> {code:java}
> == Physical Plan ==
> AdaptiveSparkPlan (24)
> +- == Final Plan ==
> ResultQueryStage (17), Statistics(sizeInBytes=8.0 EiB)
> +- * Project (16)
> +- * SortMergeJoin(skew=true) Inner (15)
> :- * Sort (7)
> : +- AQEShuffleRead (6), coalesced
> : +- ShuffleQueryStage (5), Statistics(sizeInBytes=15.6 KiB,
> rowCount=1.00E+3)
> : +- Exchange (4)
> : +- * Project (3)
> : +- * Filter (2)
> : +- * Range (1)
> +- * Sort (14)
> +- AQEShuffleRead (13), coalesced and skewed
> +- ShuffleQueryStage (12), Statistics(sizeInBytes=3.1 KiB,
> rowCount=200)
> +- Exchange (11)
> +- * Project (10)
> +- * Filter (9)
> +- * Range (8){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
