GitHub user jliwork opened a pull request:
https://github.com/apache/spark/pull/19776
[SPARK-22548][SQL] Incorrect nested AND expression pushed down to JDBâ¦
â¦C data source
## What changes were proposed in this pull request?
Letâs say I have a nested AND expression shown below and p2 can not be
pushed down,
(p1 AND p2) OR p3
In current Spark code, during data source filter translation, (p1 AND p2)
is returned as p1 only and p2 is simply lost. This issue occurs with JDBC data
source and is similar to SPARK-12218 for Parquet. When we have AND nested below
another expression, we should either push both legs or nothing. Note that 1)
the current Spark code will always split conjunctive predicate before it
determines if a predicate can be pushed down or not; 2) the same translation
method is also called by Data Source V2.
## How was this patch tested?
Added new unit test cases to JDBCSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jliwork/spark spark-22548
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19776.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19776
----
commit 58de88c21210d469b8ef14b1f23764c31ca5651e
Author: Jia Li <ji...@us.ibm.com>
Date: 2017-11-18T01:15:01Z
[SPARK-22548][SQL] Incorrect nested AND expression pushed down to JDBC data
source
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
