GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/22687
[SPARK-25702][SQL] Push down filters with `Not` operator in Parquet
## What changes were proposed in this pull request?
Currently, in ParquetFilters, predicates inside `Not` operator are
considered as unable to perform partial push down.
However, the following cases is still possible for push down:
1. `Not(Or(left, right))` can be conversed as `And(Not(left), Not(right))`
2. `Not(Not(pred))` can be conversed as `pred`
Both cases should be quite trivial, since the `Not` operator should be
pushed down by optimization rule `BooleanSimplification` already.
But I think it should be good to handle such cases in Parquet data source
module as well.
## How was this patch tested?
New unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark parquetNotFilters
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22687.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22687
----
commit 0f43db656c3567ab7f8b711c8f0b27b16caa4bf7
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-10-10T09:40:45Z
push down more parquet filters
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
