Github user nsyca commented on a diff in the pull request:
https://github.com/apache/spark/pull/17491#discussion_r109181709
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
---
@@ -498,3 +498,31 @@ object RewriteCorrelatedScalarSubquery extends
Rule[LogicalPlan] {
}
}
}
+
+/**
+ * This rule rewrites a EXISTS predicate sub-queries into an Aggregate
with count.
+ * So it doesn't be converted to a JOIN later.
+ */
+object RewriteEmptyExists extends Rule[LogicalPlan] with PredicateHelper {
+ private def containsAgg(plan: LogicalPlan): Boolean = {
+ plan.collect {
+ case a: Aggregate => a
+ }.nonEmpty
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case Filter(condition, child) =>
+ val (withSubquery, withoutSubquery) =
+
splitConjunctivePredicates(condition).partition(SubqueryExpression.hasInOrExistsSubquery)
+ val newWithSubquery = withSubquery.map(_.transform {
+ case e @ Exists(sub, conditions, exprId) if conditions.isEmpty &&
!containsAgg(sub) =>
--- End diff --
@viirya what we should do is to mark the outer join as an "early out" in
the run-time join processing. An aggregation is not cheap as it needs to read
the entire table to give the (first) answer. An "early out" logic is just like
what we currently have in the LeftSemi join where only the first match of the
join value is returned (and the subsequent matches are discarded). A LeftSemi
join is a special case of an "early out" inner join where the columns of the
right table are not permitted in the output of the join.
IMO, a step forward is to implement the "early out" mechanism in all the
run-time join operators, nested-loop, sorted-merge, and hash; and inner, left
outer, right outer. Then LeftSemi and LeftAnti will just be special cases of
one of those operators.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
