GitHub user hvanhovell opened a pull request:
https://github.com/apache/spark/pull/16026
[SPARK-18597][SQL] Do push-down predicates to right side for a LEFT ANTI
join
## What changes were proposed in this pull request?
We currently push down join predicates for a Left Anti join to both sides
of the join if we can. This is similar to Inner, Left Semi and Existence
(specialized left semi) join. The problem is that this changes the semantics of
join; a left anti join filters out rows that matches the join condition.
This PR fixes this by only allowing the push down to the left hand side of
the join. This is similar to the behavior of left outer join.
## How was this patch tested?
Added tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite
file for left anti joins with a regression test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hvanhovell/spark SPARK-18597
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16026.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16026
----
commit 7696cf2cf67ed8ea541c0b8a90901f15ae393cbe
Author: Herman van Hovell <hvanhov...@databricks.com>
Date: 2016-11-20T21:42:35Z
Fix 2.10 compilation
commit d51b63583670d274acf0fda2042915a29b174d2f
Author: Herman van Hovell <hvanhov...@databricks.com>
Date: 2016-11-21T10:11:56Z
Merge remote-tracking branch 'apache-github/master'
commit 7ccb680ebe8a2939e0b81c31be6703c3927dc7bb
Author: Herman van Hovell <hvanhov...@databricks.com>
Date: 2016-11-25T14:35:09Z
Merge remote-tracking branch 'apache-github/master'
commit e386fed62ad992eedc06e63aaeaf3524fff19b54
Author: Herman van Hovell <hvanhov...@databricks.com>
Date: 2016-11-27T20:48:40Z
Merge remote-tracking branch 'apache-github/master'
commit 4d6b18361828e56fbf1e18c1b694eb318b0851f2
Author: Herman van Hovell <hvanhov...@databricks.com>
Date: 2016-11-28T00:18:16Z
Do not change left anti join semantics by pushing down to the right side of
the join.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
