[
https://issues.apache.org/jira/browse/SPARK-61534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-61534:
----------------------------------
Parent: SPARK-58540
Issue Type: Sub-task (was: Improvement)
> Estimate the size of an existence join from its left side
> ---------------------------------------------------------
>
> Key: SPARK-61534
> URL: https://issues.apache.org/jira/browse/SPARK-61534
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Peter Toth
> Assignee: Peter Toth
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> The size estimate of a logical join of type {{ExistenceJoin}} is the product
> of the sizes of its two sides. {{SizeInBytesOnlyStatsPlanVisitor.visitJoin}}
> takes the left side's size only for {{LeftSemi}} and {{LeftAnti}}. Every
> other type outside the inner and outer joins falls back to {{default}}, which
> multiplies the sizes of the children. CBO's {{JoinEstimation}} does not
> support {{ExistenceJoin}} either, so it falls back to the same product.
> An existence join outputs each row of its left side once, with one boolean
> column added. So its size is at most a little more than the left side's.
> {{RewritePredicateSubquery}} plans an existence join for an {{EXISTS}} or
> {{IN}} subquery under an {{OR}}, as in TPC-DS q10, q35 and q45. The joins
> above it see that side as far bigger than it is, and can miss a broadcast.
> Taking the left side's size, as for {{LeftSemi}} and {{LeftAnti}}, would fix
> it. It can change the plans of those TPC-DS queries.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]