Yuming Wang created SPARK-60382:
-----------------------------------
Summary: CoalesceBucketsInJoin uses a broadcast join's build-side
bucket count
Key: SPARK-60382
URL: https://issues.apache.org/jira/browse/SPARK-60382
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yuming Wang
When spark.sql.bucketing.coalesceBucketsInJoin.enabled is true, a sort-merge
join whose child is a BuildLeft broadcast hash join, broadcast nested-loop
join, or broadcast range join can coalesce the streamed bucketed scan using the
build side's bucket count.
ExtractJoinWithBuckets.getBucketSpec uses collectFirst, which walks the left
child first. For BuildLeft the left child is the broadcast build side, so a
bucketed scan there is reported as the join input's bucket count.
updateNumCoalescedBucketsInScan then stamps every bucketed scan in that
subtree, including the streamed scan whose partitioning the parent join
actually uses.
Bucket files are grouped by the original bucket id modulo the coalesced count,
while the scan advertises HashPartitioning of that count. If the streamed
table's bucket count is not divisible by the build side's count, hash(key) %
streamedBuckets and hash(key) % coalescedBuckets land in different partitions.
The other side of the sort-merge join is a real table of the coalesced bucket
count, EnsureRequirements inserts no shuffle, and an inner join drops matching
rows. An outer join null-pads them.
The count that reaches the parent join is the streamed side's. The lookup
should follow the same streamed-side walk as hasScanOperation (Filter, Project,
then the streamed child of a broadcast hash, nested-loop, or range join), and
only that scan should be rewritten.
The config defaults to false, so this does not affect plans unless bucket
coalescing is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
