CoalesceShufflePartitions will merge small partitions into bigger
ones. Say, if you set maxShuffledHashJoinLocalMapThreshold to 32MB,
but the advisoryPartitionSize is 64MB, then the final each reducer
partition size will be close to 64MB. It breaks the
maxShuffledHashJoinLocalMapThreshold. So we should make sure the size
of maxShuffledHashJoinLocalMapThreshold is not smaller than
advisoryPartitionSize.
盛森林 <434780...@qq.com> 于2023年8月16日周三 12:12写道：
>
> dear community,
> I want to set maxShuffledHashJoinLocalMapThreshold to enable convert
> smj to hash join,but I found maxShuffledHashJoinLocalMapThreshold must be
> large than advisoryPartitionSize. I want to know what happend if
> maxShuffledHashJoinLocalMapThreshold < advisoryPartitionSize
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
