GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/9659
[SPARK-11676][SQL] Parquet filter tests all pass if filters are not really pushed down Currently Parquet predicate tests all pass even if filters are not pushed down or this is disabled. In this PR, For checking evaluating filters, Simply it makes the expression from `expression.Filter` and then try to create filters just like Spark does. For checking the results, this manually accesses to the child rdd (of `expression.Filter`) and produces the results which should be filtered properly, and then compares it to expected values. Now, if filters are not pushed down or this is disabled, this throws exceptions. cc @liancheng You can merge this pull request into a Git repository by running: $ git pull https://github.com/HyukjinKwon/spark SPARK-11676 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9659.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9659 ---- commit 1c6a5480df53d16fe38d14146deedaf035a6024c Author: hyukjinkwon <gurwls...@gmail.com> Date: 2015-11-12T09:31:41Z [SPARK-11676][SQL] Parquet filter tests all pass if filters are not really pushed down commit 6b722ef4ad3a29a1bd0cde0961f6bc12638e2f31 Author: hyukjinkwon <gurwls...@gmail.com> Date: 2015-11-12T09:37:27Z [SPARK-11676][SQL] Update indentation and comments commit d007c3f996f1fd4e093a590741b92a4bc4072d41 Author: hyukjinkwon <gurwls...@gmail.com> Date: 2015-11-12T09:39:05Z [SPARK-11676][SQL] Remove unused imports commit 8cc842fa3f97c39e61fa072a916f7056a85be1b7 Author: hyukjinkwon <gurwls...@gmail.com> Date: 2015-11-12T09:49:11Z [SPARK-111676][SQL] Set true to pushing down filters ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org