Github user maryannxue commented on the issue:
https://github.com/apache/spark/pull/21360
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
@gatorsmile That makes sense :) Simple predicates can be placed in dbtable
option. Current approach is still more powerful, but if you think that the risk
is too big, we can switch to reader's
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21360
sorry, just updated the comment. We can add a JDBC connector option. Users
can have two JDBC sources for a single JDBC table. One is with predicate
pushdown; another without predicate pushdown.
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
@gatorsmile This will reduce usability a lot. With current approach you can
push down filters that may speed up reading. Global option will affect every
other Dataset. To be honest new jdbc
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21360
@TomaszGaweda @maryannxue Let us reduce the complexity and introduce a new
JDBC option for controlling the predicate pushdown.
```Scala
val JDBC_FILTER_PUSHDOWN_ENABLED =
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
Hi @maryannxue, can you please rebase this PR? Then maybe review will be
possible by others. Would be great to include this in Spark 2.4 :) Thanks :)
---
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
I've tesed it with my application that had problem with predicate pushdowns
to database. Looks good, performance is degradated a bit, but it was previously
ran on Spark 2.3, not 2.4. However,
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
Hi @maryannxue, thanks for the PR! Could you please rebase it?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90799/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90799 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90799/testReport)**
for PR 21360 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90802/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90802 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90802/testReport)**
for PR 21360 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3342/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90802 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90802/testReport)**
for PR 21360 at commit
Github user maryannxue commented on the issue:
https://github.com/apache/spark/pull/21360
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3340/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90799 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90799/testReport)**
for PR 21360 at commit
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
@viirya I've written it in the ticket. In my case, pushing down ORs with
non-equality predicates caused DB2 to slow down; workaround was to cache data
before filtering, it was approx. 10 times
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21360
Can you tell more about the usage about `withOptimizerBarrier`? I'm curious
how it can be used in production.
---
-
To
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90779/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90779 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90779/testReport)**
for PR 21360 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3325/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90779 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90779/testReport)**
for PR 21360 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90778/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90778 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90778/testReport)**
for PR 21360 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3324/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21360
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21360
**[Test build #90778 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90778/testReport)**
for PR 21360 at commit
35 matches
Mail list logo
