Peter Toth created SPARK-61534:
----------------------------------
Summary: Estimate the size of an existence join from its left side
Key: SPARK-61534
URL: https://issues.apache.org/jira/browse/SPARK-61534
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
The size estimate of a logical join of type {{ExistenceJoin}} is the product of
the sizes of its two sides. {{SizeInBytesOnlyStatsPlanVisitor.visitJoin}} takes
the left side's size only for {{LeftSemi}} and {{LeftAnti}}. Every other type
outside the inner and outer joins falls back to {{default}}, which multiplies
the sizes of the children. CBO's {{JoinEstimation}} does not support
{{ExistenceJoin}} either, so it falls back to the same product.
An existence join outputs each row of its left side once, with one boolean
column added. So its size is at most a little more than the left side's.
{{RewritePredicateSubquery}} plans an existence join for an {{EXISTS}} or
{{IN}} subquery under an {{OR}}, as in TPC-DS q10, q35 and q45. The joins above
it see that side as far bigger than it is, and can miss a broadcast.
Taking the left side's size, as for {{LeftSemi}} and {{LeftAnti}}, would fix
it. It can change the plans of those TPC-DS queries.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]