[GitHub] spark pull request #21875: [SPARK-24288][SQL] Enable preventing predicate pu...

2018-07-25 Thread TomaszGaweda 
Github user TomaszGaweda commented on a diff in the pull request:

https://github.com/apache/spark/pull/21875#discussion_r205224686
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
 ---
@@ -183,6 +183,9 @@ class JDBCOptions(
 }
   // An option to execute custom SQL before fetching data from the remote 
DB
   val sessionInitStatement = parameters.get(JDBC_SESSION_INIT_STATEMENT)
+
+  // An option to allow/disallow pushing down predicate into JDBC data 
source
+  val pushDownPredicate = parameters.getOrElse(JDBC_PUSHDOWN_PREDICATE, 
"true").toBoolean
--- End diff --

Super Nit: Shouldn't it be in plural, pushDownPredicates? There may be many 
predicates


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21875: [SPARK-24288][SQL] Enable preventing predicate pu...

2018-07-25 Thread TomaszGaweda 
Github user TomaszGaweda commented on a diff in the pull request:

https://github.com/apache/spark/pull/21875#discussion_r205227335
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
 ---
@@ -172,7 +172,11 @@ private[sql] case class JDBCRelation(
 
   // Check if JDBCRDD.compileFilter can accept input filters
   override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
-filters.filter(JDBCRDD.compileFilter(_, 
JdbcDialects.get(jdbcOptions.url)).isEmpty)
+if (jdbcOptions.pushDownPredicate) {
--- End diff --

Are you sure, that this is the only place? JDBCRDD.scanTable defines 
filters as all filters that may be pushed down. Probably we should use `filters 
-- unhandledFilters` in JdbcRelation.buildScan


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21875: [SPARK-24288][SQL] Enable preventing predicate pu...

2018-07-25 Thread maryannxue 
GitHub user maryannxue opened a pull request:

https://github.com/apache/spark/pull/21875

[SPARK-24288][SQL] Enable preventing predicate pushdown

## What changes were proposed in this pull request?

Add a JDBC Option "pushDownPredicate" (default `true`) to allow/disallow 
predicate push-down in JDBC data source.

## How was this patch tested?

Add a test in `JDBCSuite`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/maryannxue/spark spark-24288

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21875.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21875


commit a83b64b53aa043d4b1cc9b5572c3676ec168027c
Author: maryannxue 
Date:   2018-07-25T18:36:53Z

[SPARK-24288][SQL] Enable preventing predicate pushdown




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org