XiDuo You created SPARK-35725:
---------------------------------
Summary: Support repartition expand partitions
Key: SPARK-35725
URL: https://issues.apache.org/jira/browse/SPARK-35725
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0
Reporter: XiDuo You
Currently, we don't support expand partition dynamically in AQE which is not
friendly for some data skew job.
Let's say if we have a simple query:
{code:java}
SELECT * FROM table DISTRIBUTE BY col
{code}
The column of `col` is skewed, then some shuffle partitions would handle too
much data than others.
If we haven't inroduced extra shuffle, we can optimize this case by expanding
partitions in AQE.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
