Chao Sun created SPARK-42038: -------------------------------- Summary: SPJ: Support partially clustered distribution Key: SPARK-42038 URL: https://issues.apache.org/jira/browse/SPARK-42038 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.3.1 Reporter: Chao Sun
Currently the storage-partitioned join requires both sides to be fully clustered on the partition values, that is, all input partitions reported by a V2 data source shall be grouped by partition values before the join happens. This could lead to data skew issues if a particular partition value is associated with a large amount of rows. To combat this, we can introduce the idea of partially clustered distribution, which means that only one side of the join is required to be fully clustered, while the other side is not. This allows Spark to increase the parallelism of the join and avoid the data skewness. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org