cloud-fan commented on a change in pull request #33541:
URL: https://github.com/apache/spark/pull/33541#discussion_r678751988
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala
##########
@@ -69,6 +70,21 @@ case class AQEShuffleReadExec private(
case _ =>
throw new IllegalStateException("operating on canonicalization plan")
}
+ } else if (isCoalescedRead) {
+ // For coalesced shuffle read, the data distribution is not changed,
only the number of
+ // partitions is changed.
+ child.outputPartitioning match {
+ case h: HashPartitioning =>
+ CurrentOrigin.withOrigin(h.origin)(h.copy(numPartitions =
partitionSpecs.length))
+ case r: RangePartitioning =>
+ CurrentOrigin.withOrigin(r.origin)(r.copy(numPartitions =
partitionSpecs.length))
+ // This can only happen for `REBALANCE_PARTITIONS_BY_NONE`, which uses
+ // `RoundRobinPartitioning` but we don't need to retain the number of
partitions.
+ case r: RoundRobinPartitioning =>
+ r.copy(numPartitions = partitionSpecs.length)
+ case other => throw new IllegalStateException(
Review comment:
shuffle can only be added by `df.repartition` or `EnsureRequirements`,
all the possible partitioning has been listed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
