GitHub user hvanhovell opened a pull request: https://github.com/apache/spark/pull/16026
[SPARK-18597][SQL] Do push-down predicates to right side for a LEFT ANTI join ## What changes were proposed in this pull request? We currently push down join predicates for a Left Anti join to both sides of the join if we can. This is similar to Inner, Left Semi and Existence (specialized left semi) join. The problem is that this changes the semantics of join; a left anti join filters out rows that matches the join condition. This PR fixes this by only allowing the push down to the left hand side of the join. This is similar to the behavior of left outer join. ## How was this patch tested? Added tests to `FilterPushdownSuite.scala` and created a SQLQueryTestSuite file for left anti joins with a regression test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hvanhovell/spark SPARK-18597 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16026.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16026 ---- commit 7696cf2cf67ed8ea541c0b8a90901f15ae393cbe Author: Herman van Hovell <hvanhov...@databricks.com> Date: 2016-11-20T21:42:35Z Fix 2.10 compilation commit d51b63583670d274acf0fda2042915a29b174d2f Author: Herman van Hovell <hvanhov...@databricks.com> Date: 2016-11-21T10:11:56Z Merge remote-tracking branch 'apache-github/master' commit 7ccb680ebe8a2939e0b81c31be6703c3927dc7bb Author: Herman van Hovell <hvanhov...@databricks.com> Date: 2016-11-25T14:35:09Z Merge remote-tracking branch 'apache-github/master' commit e386fed62ad992eedc06e63aaeaf3524fff19b54 Author: Herman van Hovell <hvanhov...@databricks.com> Date: 2016-11-27T20:48:40Z Merge remote-tracking branch 'apache-github/master' commit 4d6b18361828e56fbf1e18c1b694eb318b0851f2 Author: Herman van Hovell <hvanhov...@databricks.com> Date: 2016-11-28T00:18:16Z Do not change left anti join semantics by pushing down to the right side of the join. ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org