XiDuo You created SPARK-35923:
---------------------------------
Summary: Coalesce empty partition with mixed
CoalescedPartitionSpec and PartialReducerPartitionSpec
Key: SPARK-35923
URL: https://issues.apache.org/jira/browse/SPARK-35923
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0
Reporter: XiDuo You
Since [SPARK-35447](https://issues.apache.org/jira/browse/SPARK-35447), we
apply `OptimizeSkewedJoin` before `CoalesceShufflePartitions`. However, There
are something different with the order of these two rules.
Let's say if we have a skewed partitions: [0, 128MB, 0, 128MB, 0]:
# coalesce partitions first then optimize skewed partitions:
[64MB, 64MB, 64MB, 64MB]
# optimize skew partition first then coalesce partitions:
[0, 64MB, 64MB, 0, 64MB, 64MB, 0]
So we can do coalesce in ShufflePartitionsUtil.coalescePartitionsWithSkew with
mixed CoalescedPartitionSpec and PartialReducerPartitionSpec if
CoalescedPartitionSpec is empty.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
