GitHub user nsyca opened a pull request: https://github.com/apache/spark/pull/15763
[SPARK-17348][SQL] Incorrect results from subquery transformation ## What changes were proposed in this pull request? Return an Analysis exception when there is a correlated non-equality predicate in a subquery and the correlated column from the outer reference is not from the immediate parent operator of the subquery. This PR prevents incorrect results from subquery transformation in such case. Test cases, both positive and negative tests, are added. ## How was this patch tested? sql/test, catalyst/test, hive/test, and scenarios that will produce incorrect results without this PR and product correct results when subquery transformation does happen. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nsyca/spark spark-17348 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15763.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15763 ---- commit b98865127a39bde885f9b1680cfe608629d59d51 Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-07-29T21:43:56Z [SPARK-16804][SQL] Correlated subqueries containing LIMIT return incorrect results ## What changes were proposed in this pull request? This patch fixes the incorrect results in the rule ResolveSubquery in Catalyst's Analysis phase. ## How was this patch tested? ./dev/run-tests a new unit test on the problematic pattern. commit 069ed8f8e5f14dca7a15701945d42fc27fe82f3c Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-07-29T21:50:02Z [SPARK-16804][SQL] Correlated subqueries containing LIMIT return incorrect results ## What changes were proposed in this pull request? This patch fixes the incorrect results in the rule ResolveSubquery in Catalyst's Analysis phase. ## How was this patch tested? ./dev/run-tests a new unit test on the problematic pattern. commit edca333c081e6d4e53a91b496fba4a3ef4ee89ac Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-07-30T00:28:15Z New positive test cases commit 64184fdb77c1a305bb2932e82582da28bb4c0e53 Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-08-01T13:20:09Z Fix unit test case failure commit 29f82b05c9e40e7934397257c674b260a8e8a996 Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-08-05T17:42:01Z blocking TABLESAMPLE commit ac43ab47907a1ccd6d22f920415fbb4de93d4720 Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-08-05T21:10:19Z Fixing code styling commit 631d396031e8bf627eb1f4872a4d3a17c144536c Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-08-07T18:39:44Z Correcting Scala test style commit 7eb9b2dbba3633a1958e38e0019e3ce816300514 Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-08-08T02:31:09Z One (last) attempt to correct the Scala style tests commit 1387cf51541408ac20048064fa5e559836af932c Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-08-12T20:11:50Z Merge remote-tracking branch 'upstream/master' commit 6d9bade4df8954987078c479274d90a7612cc772 Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-11-04T03:51:54Z Merge remote-tracking branch 'upstream/master' commit baf0e6084a838ce2d72eeeac9d7618ae4536ffb6 Author: Nattavut Sutyanyong <nsy....@gmail.com> Date: 2016-11-04T03:54:24Z First version of code+test cases ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org