cloud-fan commented on a change in pull request #34634:
URL: https://github.com/apache/spark/pull/34634#discussion_r755239272
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala
##########
@@ -316,21 +316,25 @@ object ShufflePartitionsUtil extends Logging {
* start of a partition.
*/
// Visible for testing
- private[sql] def splitSizeListByTargetSize(sizes: Seq[Long], targetSize:
Long): Array[Int] = {
+ private[sql] def splitSizeListByTargetSize(
+ sizes: Seq[Long],
+ targetSize: Long,
+ minPartitionSize: Long): Array[Int] = {
val partitionStartIndices = ArrayBuffer[Int]()
partitionStartIndices += 0
var i = 0
var currentPartitionSize = 0L
var lastPartitionSize = -1L
+ val minSize =
+ Math.min(targetSize, Math.max(minPartitionSize, targetSize *
SMALL_PARTITION_FACTOR))
def tryMergePartitions() = {
// When we are going to start a new partition, it's possible that the
current partition or
// the previous partition is very small and it's better to merge the
current partition into
// the previous partition.
val shouldMergePartitions = lastPartitionSize > -1 &&
((currentPartitionSize + lastPartitionSize) < targetSize *
MERGED_PARTITION_FACTOR ||
Review comment:
I see a conflict here:
1. we want to make each partition smaller than `targetSize *
MERGED_PARTITION_FACTOR`
2. we want to make each partition larger than `targetSize *
SMALL_PARTITION_FACTOR`
Seems like we prioritize 2 over 1, but in some cases
`SMALL_PARTITION_FACTOR` needs to be smaller.
How about we make `SMALL_PARTITION_FACTOR` a parameter of the function? Then
we can pass different values from skew join rule and rebalance rule. e.g. The
value can be `0.1` in the rebalance rule, or even making it configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
