CoalesceShufflePartitions will merge small partitions into bigger ones. Say, if you set maxShuffledHashJoinLocalMapThreshold to 32MB, but the advisoryPartitionSize is 64MB, then the final each reducer partition size will be close to 64MB. It breaks the maxShuffledHashJoinLocalMapThreshold. So we should make sure the size of maxShuffledHashJoinLocalMapThreshold is not smaller than advisoryPartitionSize.
盛森林 <434780...@qq.com> 于2023年8月16日周三 12:12写道： > > dear community, > I want to set maxShuffledHashJoinLocalMapThreshold to enable convert > smj to hash join,but I found maxShuffledHashJoinLocalMapThreshold must be > large than advisoryPartitionSize. I want to know what happend if > maxShuffledHashJoinLocalMapThreshold < advisoryPartitionSize --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
