[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181786940 **[Test build #50968 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50968/consoleFull)** for PR 10427 at commit [`7038bc0`](https://github.com/apache/spark/commit/7038bc05b9887eac7d5d61eac179124e6349abc2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181786010 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50966/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181786009 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181817385 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50968/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181817384 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181817281 **[Test build #50968 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50968/consoleFull)** for PR 10427 at commit [`7038bc0`](https://github.com/apache/spark/commit/7038bc05b9887eac7d5d61eac179124e6349abc2). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181904428 **[Test build #50976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50976/consoleFull)** for PR 10427 at commit [`7a7b9fa`](https://github.com/apache/spark/commit/7a7b9fa352bc8e28491d3ba22d2a68b943575044). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181937545 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181937547 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50976/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181937219 **[Test build #50976 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50976/consoleFull)** for PR 10427 at commit [`7a7b9fa`](https://github.com/apache/spark/commit/7a7b9fa352bc8e28491d3ba22d2a68b943575044). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-182160420 @yhuai okay, ready to commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-182162950 LGTM. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10427 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r52120299
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,11 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ // Check if JDBCRDD.compileFilter can accept input filters
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
+filters.filter(JDBCRDD.compileFilter(_).isEmpty)
--- End diff --
Added tests though, ISTM that `JDBCRDD.compileFilter` does not return
`None` for given `Filter` because the function can compile all the `Filter`
implemented in `sql.sources.filters`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181050301 **[Test build #50904 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50904/consoleFull)** for PR 10427 at commit [`1ebacc2`](https://github.com/apache/spark/commit/1ebacc28b3d10d574315bd37f4c4a16837fa2936). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181070297 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50904/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181070296 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181070255 **[Test build #50904 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50904/consoleFull)** for PR 10427 at commit [`1ebacc2`](https://github.com/apache/spark/commit/1ebacc28b3d10d574315bd37f4c4a16837fa2936). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-181151008 Thank you for the update! Overall looks good. There are two lines of unnecessary changes. Let's remove them and we can merge it once it passes jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r52128728
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -210,6 +218,24 @@ class JDBCSuite extends SparkFunSuite
val df2 = sql("SELECT * FROM foobar WHERE NOT (THEID != 2) OR NOT
(NAME != 'mary')")
assert(df1.collect.toSet === Set(Row("mary", 2)))
assert(df2.collect.toSet === Set(Row("mary", 2)))
+
+def checkNotPushdown(df: DataFrame): DataFrame = {
+ val parentPlan = df.queryExecution.executedPlan
+ // Check if SparkPlan Filter is not removed in a physical plan
because JDBCRDD
+ // cannot compile given predicates.
+
assert(parentPlan.isInstanceOf[org.apache.spark.sql.execution.Filter])
+ df
+}
+sql("SELECT * FROM foobar WHERE THEID < 1").explain(true)
+sql("SELECT * FROM foobar WHERE (THEID + 1) < 2").explain(true)
+assert(checkNotPushdown(sql("SELECT * FROM foobar WHERE (THEID + 1) <
2")).collect().size == 0)
+assert(checkNotPushdown(sql("SELECT * FROM foobar WHERE (THEID + 2) !=
4")).collect().size == 2)
+ }
+
+ test("SELECT COUNT(1) WHERE (predicates)") {
+// Check if an answer is correct when filters are pushed down into
JDBC data sources
+// and some columns are pruned in DataSourceStrategy.
--- End diff --
Can you also add comment to explain the reason of adding this test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r52128717
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -210,6 +218,24 @@ class JDBCSuite extends SparkFunSuite
val df2 = sql("SELECT * FROM foobar WHERE NOT (THEID != 2) OR NOT
(NAME != 'mary')")
assert(df1.collect.toSet === Set(Row("mary", 2)))
assert(df2.collect.toSet === Set(Row("mary", 2)))
+
+def checkNotPushdown(df: DataFrame): DataFrame = {
+ val parentPlan = df.queryExecution.executedPlan
+ // Check if SparkPlan Filter is not removed in a physical plan
because JDBCRDD
+ // cannot compile given predicates.
+
assert(parentPlan.isInstanceOf[org.apache.spark.sql.execution.Filter])
+ df
+}
+sql("SELECT * FROM foobar WHERE THEID < 1").explain(true)
+sql("SELECT * FROM foobar WHERE (THEID + 1) < 2").explain(true)
--- End diff --
Remove these two lines?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r52110498
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,11 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ // Check if JDBCRDD.compileFilter can accept input filters
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
+filters.filter(JDBCRDD.compileFilter(_).isEmpty)
--- End diff --
Thank you for the update!
When we were using ` == null` (I guess all filters were marked as
unhandled, right?), all tests still passed. So, I am wondering if existing
tests are sufficient?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-180388822 **[Test build #50818 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50818/consoleFull)** for PR 10427 at commit [`b4dc961`](https://github.com/apache/spark/commit/b4dc961dc2ac2437ca080e011fbb6cf8a5688538). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-180389108 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50818/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-180389106 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r51994731
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,11 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ // Check if JDBCRDD.compileFilter can accept input filters
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
+filters.filter(JDBCRDD.compileFilter(_) == null)
--- End diff --
Seems you want to use `isEmpty`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-180359628 **[Test build #50818 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50818/consoleFull)** for PR 10427 at commit [`b4dc961`](https://github.com/apache/spark/commit/b4dc961dc2ac2437ca080e011fbb6cf8a5688538). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-180354486 @yhuai fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-180237461 @yhuai ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-178372154 @yhuai ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-175997712 @yhuai ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-173552368 @yhuai ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-171631018 **[Test build #49397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49397/consoleFull)** for PR 10427 at commit [`22e20ea`](https://github.com/apache/spark/commit/22e20eaea920f7c68f904c1e5d5d7bc39113fdf9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-171651804 **[Test build #49397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49397/consoleFull)** for PR 10427 at commit [`22e20ea`](https://github.com/apache/spark/commit/22e20eaea920f7c68f904c1e5d5d7bc39113fdf9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-171652041 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-171652043 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49397/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-171544801 @yhuai ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168651319 @yuhai Oh... my bad :(( Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168618638 @yuhai Could you review it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user yuhai commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168620757 Hi Takeshi, You should cc wrong address. :) @yhuai Thanks, Hai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168881340 **[Test build #48705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48705/consoleFull)** for PR 10427 at commit [`9dcb64e`](https://github.com/apache/spark/commit/9dcb64e88cb40e3e76afcf7efb5d3c1fd466b1d0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168865004 **[Test build #48705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48705/consoleFull)** for PR 10427 at commit [`9dcb64e`](https://github.com/apache/spark/commit/9dcb64e88cb40e3e76afcf7efb5d3c1fd466b1d0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168881503 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48705/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168881502 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168714566 **[Test build #48666 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48666/consoleFull)** for PR 10427 at commit [`b0d53d1`](https://github.com/apache/spark/commit/b0d53d1921733f020f0270abce6edb642523b778). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168715759 **[Test build #48666 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48666/consoleFull)** for PR 10427 at commit [`b0d53d1`](https://github.com/apache/spark/commit/b0d53d1921733f020f0270abce6edb642523b778). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168715769 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48666/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-168715766 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167071998 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167072000 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48293/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167071782 **[Test build #48293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48293/consoleFull)** for PR 10427 at commit [`3fb9199`](https://github.com/apache/spark/commit/3fb919981bd9b1af3b9baebf23ab2aa6eef126a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/10427#issuecomment-167017518
Let me leave a comment. I tested some cases with this PR and looks
generally working fine. But I would like to mention one thing that I am pretty
sure you guys already know though.
Since now `Filter` is removed, for operation such as `count()`which does
not require any columns, it works different internally. It looks okay with JDBC
but it does not with others such as Parquet.
For example, in case of Parquet, I implemented this locally and tested and
found it calculating a wrong results. That produced the plans below:
```
== Physical Plan ==
TungstenAggregate(key=[],
functions=[(count(1),mode=Final,isDistinct=false)], output=[count#10L])
+- TungstenExchange SinglePartition, None
+- TungstenAggregate(key=[],
functions=[(count(1),mode=Partial,isDistinct=false)], output=[count#14L])
+- Project
+- Scan ParquetRelation[] InputPaths:
file:/private/var/folders/9j/gf_c342d7d150mwrxvkqnc18gn/T/spark-bc91cad7-b7c6-4ef0-985d-abaad921572d/part=1,
PushedFilters: [EqualTo(a,2)]
```
Here, I could find any `requiredColumns` not given and producing wrong
results by Parquet.
I tried with the original code and it showed the plan below assigning
`requiredColumns` properly.
```
== Physical Plan ==
TungstenAggregate(key=[],
functions=[(count(1),mode=Final,isDistinct=false)], output=[count#10L])
+- TungstenExchange SinglePartition, None
+- TungstenAggregate(key=[],
functions=[(count(1),mode=Partial,isDistinct=false)], output=[count#14L])
+- Project
+- Filter (a#8 = 2)
+- Scan ParquetRelation[a#8] InputPaths:
file:/private/var/folders/9j/gf_c342d7d150mwrxvkqnc18gn/T/spark-ef271ec6-95e1-43ae-9b3e-1d4dae6f69c3/part=1,
PushedFilters: [EqualTo(a,2)]
```
The reason seems that it adds all the columns in projects and filters to
`requiredColumns` but it could not produce any columns as it ends up with no
columns from `Project` without any columns and without `Filter`.
For this `JDBCRelation`, I tested and saw the `count()` working fine as it
gives `1` when `requiredColumns` is empty, producing the plan below.
```
== Physical Plan ==
TungstenAggregate(key=[],
functions=[(count(1),mode=Final,isDistinct=false)], output=[count#38L])
+- TungstenExchange SinglePartition, None
+- TungstenAggregate(key=[],
functions=[(count(1),mode=Partial,isDistinct=false)], output=[count#42L])
+- Project
+- Scan
JDBCRelation(jdbc:h2:mem:testdb0,TEST.PEOPLE,[Lorg.apache.spark.Partition;@3e906375,{user=testUser,
password=testPass, url=jdbc:h2:mem:testdb0, dbtable=TEST.PEOPLE})[]
PushedFilters: [EqualTo(NAME,fred)]
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48391917
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,11 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ // Check if JDBCRDD.compileFilter can accept input filters
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
+filters.filterNot(JDBCRDD.compileFilter(_) != null)
--- End diff --
Okay and I'll fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167026635 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167033198 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167035311 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48280/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167039229 @HyukjinKwon ISTM the root problem is that catalyst cannot fill required columns on-the-fly for data sources. `unhandleFilters` is little related to the problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167026636 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48276/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167035310 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167039831 If this patch merged, the comment below is not suitable for `JDBCRelation`. https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala#L281 Is it okay to fix it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167025712 **[Test build #48277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48277/consoleFull)** for PR 10427 at commit [`56d14cf`](https://github.com/apache/spark/commit/56d14cff6e7d1d26e8c238726e5568c28818365f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167037407 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167037408 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48277/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167037362 **[Test build #48277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48277/consoleFull)** for PR 10427 at commit [`56d14cf`](https://github.com/apache/spark/commit/56d14cff6e7d1d26e8c238726e5568c28818365f). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class JavaWordBlacklist `\n * `class JavaDroppedWordsCounter `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167040023 And you are right I think the comments I said is not related with this PR. Let's wait for their comments! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167039644 @maropu I believe it is a Parquet stuff. AFAIK, the columns in filters should be set to `requestedSchema` for Parquet. But [this](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L342-L348) exlcudes the columns for pushed filters. Actually, I am now talking with @yhuai to check if it is really an issue. For me I think we can simply include the columns for pushed filters as well just for safety. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167042101 Adding a special handing for Parquet is better? I think that required columns for push-down filters should be passed to each data sources, and this is a more natural way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167044200 @maropu I agree that can be another way! but I just think an interface should be inclusive not exclusive. Handing in `ParquetRelation` might mean other datasources might have to deal with similar problems. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167051327 **[Test build #48290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48290/consoleFull)** for PR 10427 at commit [`40d52c9`](https://github.com/apache/spark/commit/40d52c9caa43aad088fbbc79d6b523c1663895c2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167063911 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48290/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167063910 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167046672 **[Test build #48287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48287/consoleFull)** for PR 10427 at commit [`8d5710a`](https://github.com/apache/spark/commit/8d5710aaab8d1b0d1c595ffefd683355f8c8464e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167060472 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167060473 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48287/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167060381 **[Test build #48287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48287/consoleFull)** for PR 10427 at commit [`8d5710a`](https://github.com/apache/spark/commit/8d5710aaab8d1b0d1c595ffefd683355f8c8464e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167045790 @HyukjinKwon Anyway, I added a test by reflecting this discussion. thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167060213 **[Test build #48293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48293/consoleFull)** for PR 10427 at commit [`3fb9199`](https://github.com/apache/spark/commit/3fb919981bd9b1af3b9baebf23ab2aa6eef126a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-167063848 **[Test build #48290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48290/consoleFull)** for PR 10427 at commit [`40d52c9`](https://github.com/apache/spark/commit/40d52c9caa43aad088fbbc79d6b523c1663895c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48261569
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,21 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ /**
+ * Check if [[JDBCRDD.compileFilter]] can accept input filters.
+ */
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
+filters.filterNot(canCompileFilter)
+ }
+
+ private def canCompileFilter(filter: Filter): Boolean = filter match {
+case EqualTo(_, _) | Not(EqualTo(_, _)) => true
+case LessThan(_, _) | LessThanOrEqual(_, _) => true
+case GreaterThan(_, _) | GreaterThanOrEqual(_, _) => true
+case IsNull(_) | IsNotNull(_) => true
+case _ => false
+ }
+
--- End diff --
Oh, smart idea. Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-166645536 **[Test build #48201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48201/consoleFull)** for PR 10427 at commit [`c362c91`](https://github.com/apache/spark/commit/c362c91231f95b37fb8cfb389f95a675603b30ac). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48313071
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -180,14 +181,23 @@ class JDBCSuite extends SparkFunSuite
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+def checkPlan(df: DataFrame): DataFrame = {
+ val parentPlan = df.queryExecution.executedPlan
+ // Check if SparkPlan Filter is removed in a physical plan and
+ // the plan only has PhysicalRDD to scan JDBCRelation.
--- End diff --
I see. Thanks! That was my bad.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48242420
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -176,14 +178,23 @@ class JDBCSuite extends SparkFunSuite with
BeforeAndAfter with SharedSQLContext
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+val doFilterPushdown = (df: DataFrame) => {
+ val schema = df.schema
+ val parentPlan = df.queryExecution.executedPlan
+ assert(parentPlan.isInstanceOf[PhysicalRDD])
--- End diff --
Added the comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48242446
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -176,14 +178,23 @@ class JDBCSuite extends SparkFunSuite with
BeforeAndAfter with SharedSQLContext
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+val doFilterPushdown = (df: DataFrame) => {
+ val schema = df.schema
+ val parentPlan = df.queryExecution.executedPlan
+ assert(parentPlan.isInstanceOf[PhysicalRDD])
+
assert(parentPlan.asInstanceOf[PhysicalRDD].nodeName.contains("JDBCRelation"))
+ val rdd = parentPlan.execute().map(row =>
Row.fromSeq(row.toSeq(schema)))
--- End diff --
Oh, u're right. Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-166588013 @liancheng would you tell me what you think on [this](https://github.com/apache/spark/pull/10427#discussion_r48231701)? I made some commits locally and want to be sure and make them structurally consistent with this PR becore creating new PRs for them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48231102
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,19 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ // Check if JDBCRDD#compileFilter can accept input filters
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
+filters.filterNot(canCompileFilter)
+ }
+
+ private def canCompileFilter(filter: Filter): Boolean = filter match {
+case EqualTo(_, _) | Not(EqualTo(_, _)) => true
+case LessThan(_, _) | LessThanOrEqual(_, _) => true
+case GreaterThan(_, _) | GreaterThanOrEqual(_, _) => true
+case IsNull(_) | IsNotNull(_) => true
+case _ => false
--- End diff --
Can this push down `And`?
As far as I know `JDBCRelation` pushes down `And` filter as well when `Or`
is not given.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-166568408 cc @liancheng @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/10427#discussion_r48243182 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -90,6 +90,21 @@ private[sql] case class JDBCRelation( override val schema: StructType = JDBCRDD.resolveTable(url, table, properties) + /** + * Check if [[JDBCRDD.compileFilter]] can accept input filters. --- End diff -- Hm.. for this one i'm not too sure if the function is also wrapped with [[ ]]. I will look through Scala guidelines. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-166570264 cc @liancheng @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/10427#discussion_r48237984 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -90,6 +90,19 @@ private[sql] case class JDBCRelation( override val schema: StructType = JDBCRDD.resolveTable(url, table, properties) + // Check if JDBCRDD#compileFilter can accept input filters --- End diff -- This is a little nit I was pointed out before :). We might have to write a command with [[ ]] for Scala class and with \` \` or with nothing for functions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/10427#discussion_r48242522 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -90,6 +90,19 @@ private[sql] case class JDBCRelation( override val schema: StructType = JDBCRDD.resolveTable(url, table, properties) + // Check if JDBCRDD#compileFilter can accept input filters --- End diff -- thanks and fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48242480
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -176,14 +178,23 @@ class JDBCSuite extends SparkFunSuite with
BeforeAndAfter with SharedSQLContext
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+val doFilterPushdown = (df: DataFrame) => {
--- End diff --
No reason, so fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48242805
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -180,14 +181,23 @@ class JDBCSuite extends SparkFunSuite
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+def checkPlan(df: DataFrame): DataFrame = {
+ val parentPlan = df.queryExecution.executedPlan
+ // Check if SparkPlan Filter is removed in a physical plan and
+ // the plan only has PhysicalRDD to scan JDBCRelation.
--- End diff --
A nit. Please wrap with [[]] for Scala classes (`SparkPlan`, `Filter`and
`JDBCRelation`) in comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-166581827 **[Test build #48188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48188/consoleFull)** for PR 10427 at commit [`a910a7e`](https://github.com/apache/spark/commit/a910a7ee57d020849040dc1f35f99f153f9d2a35). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-166587134 @liancheng Would you tell me what you think on [this](https://github.com/apache/spark/pull/10427#discussion_r48231701)? I made some commits locally but want to be sure and make them structurally consistent with this PR before creating new PR for them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48236730
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -176,14 +178,23 @@ class JDBCSuite extends SparkFunSuite with
BeforeAndAfter with SharedSQLContext
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+val doFilterPushdown = (df: DataFrame) => {
--- End diff --
I usually see `def` for nested functions in other test codes in Spark
instead of using `val`. For me using `val` might make a little bit hard to
read. Is there a special reason to use this maybe?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10427#issuecomment-166579538 **[Test build #48187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48187/consoleFull)** for PR 10427 at commit [`9ead74a`](https://github.com/apache/spark/commit/9ead74a291ecc4b4fc6d5800f6d438e04a402b8c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48243987
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,19 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ // Check if JDBCRDD#compileFilter can accept input filters
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
--- End diff --
Hm.. Let's wait for the reviews from the guys I cc :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48229951
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -176,14 +178,23 @@ class JDBCSuite extends SparkFunSuite with
BeforeAndAfter with SharedSQLContext
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+val doFilterPushdown = (df: DataFrame) => {
+ val schema = df.schema
+ val parentPlan = df.queryExecution.executedPlan
+ assert(parentPlan.isInstanceOf[PhysicalRDD])
--- End diff --
Should we maybe add some comments or messages to know what this `assert`
mean?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48230062
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -176,14 +178,23 @@ class JDBCSuite extends SparkFunSuite with
BeforeAndAfter with SharedSQLContext
}
test("SELECT * WHERE (simple predicates)") {
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID <
1")).collect().size == 0)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID !=
2")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE THEID =
1")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME =
'fred'")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME >
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM foobar WHERE NAME !=
'fred'")).collect().size == 2)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS
NULL")).collect().size == 1)
-assert(stripSparkFilter(sql("SELECT * FROM nulltypes WHERE A IS NOT
NULL")).collect().size == 0)
+val doFilterPushdown = (df: DataFrame) => {
+ val schema = df.schema
+ val parentPlan = df.queryExecution.executedPlan
+ assert(parentPlan.isInstanceOf[PhysicalRDD])
+
assert(parentPlan.asInstanceOf[PhysicalRDD].nodeName.contains("JDBCRelation"))
+ val rdd = parentPlan.execute().map(row =>
Row.fromSeq(row.toSeq(schema)))
--- End diff --
And.. I think we might not have to create a new `DataFrame` here since
`df.queryExecution.executedPlan` is `PhysicalRDD` and it means the `df` itself
is not wrapped with `Filter`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12476][SQL] Implement JdbcRelation#unha...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/10427#discussion_r48231701
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -90,6 +90,19 @@ private[sql] case class JDBCRelation(
override val schema: StructType = JDBCRDD.resolveTable(url, table,
properties)
+ // Check if JDBCRDD#compileFilter can accept input filters
+ override def unhandledFilters(filters: Array[Filter]): Array[Filter] = {
--- End diff --
This is my personal thought and I do not think we should fix but how about
making the structure like `SimpleFilteredScan` in `FilteredScanSuite`? I was
working on [those issues](https://issues.apache.org/jira/browse/SPARK-12354)
and it would be great if we have the same structure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
