Github user TomaszGaweda closed the pull request at:
https://github.com/apache/spark/pull/22249
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/22399
I like this idea :) +1 as it's only refactor, without logic change
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22399#discussion_r216824283
--- Diff:
common/kvstore/src/test/java/org/apache/spark/util/kvstore/DBIteratorSuite.java
---
@@ -383,7 +383,7 @@ public void
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213232338
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213213173
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213126158
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213112726
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22249#discussion_r213110408
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2459,6 +2459,26 @@ object functions {
StringTrimLeft(e.expr
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/22249
@gatorsmile @cloud-fan @HyukjinKwon @mgaido91 Could you please review this
PR and start tests?
---
-
To unsubscribe, e
GitHub user TomaszGaweda opened a pull request:
https://github.com/apache/spark/pull/22249
[SPARK-16281][SQL][FOLLOW-UP] Add parse_url to functions.scala
## What changes were proposed in this pull request?
Add `parse_url` function to `functions.scala`. This will allow users
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/22063#discussion_r209713521
--- Diff: project/MimaExcludes.scala ---
@@ -36,6 +36,11 @@ object MimaExcludes {
// Exclude rules for 2.4.x
lazy val v24excludes
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21979
@vanzin @squito, it's probably deleted by mistake in this commit:
https://github.com/devaraj-kavali/spark/commit/3cb82047f2f51af553df09b9323796af507d36f8
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/21875#discussion_r205269695
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -172,7 +172,11 @@ private[sql] case class
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/21875#discussion_r205269684
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -172,7 +172,11 @@ private[sql] case class
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/21875#discussion_r205268370
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -183,6 +183,9 @@ class JDBCOptions
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/21875#discussion_r205267937
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -172,7 +172,11 @@ private[sql] case class
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/21858#discussion_r205243526
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -1150,16 +1150,48 @@ object functions {
/**
* A column
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/21875#discussion_r205224686
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -183,6 +183,9 @@ class JDBCOptions
Github user TomaszGaweda commented on a diff in the pull request:
https://github.com/apache/spark/pull/21875#discussion_r205227335
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
---
@@ -172,7 +172,11 @@ private[sql] case class
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21875
Thanks! :) LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
@gatorsmile That makes sense :) Simple predicates can be placed in dbtable
option. Current approach is still more powerful, but if you think that the risk
is too big, we can switch to reader's
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
@gatorsmile This will reduce usability a lot. With current approach you can
push down filters that may speed up reading. Global option will affect every
other Dataset. To be honest new jdbc
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
Hi @maryannxue, can you please rebase this PR? Then maybe review will be
possible by others. Would be great to include this in Spark 2.4 :) Thanks
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21594
IMHO it is good, but may confuse users. Could you please add some JavaDocs
to explain the difference?
---
-
To unsubscribe
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
I've tesed it with my application that had problem with predicate pushdowns
to database. Looks good, performance is degradated a bit, but it was previously
ran on Spark 2.3, not 2.4. However
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21500
@HeartSaVioR IMHO we should consider new state provider such as RocksDB,
like Flink and Databricks Delta did. It is not a direct fix, but will improve
latency and memory consumption, maybe
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
Hi @maryannxue, thanks for the PR! Could you please rebase it?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21444
Please close this PR
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21432
Probably similar change should be also in KeyValueGroupedDataset. It also
uses logicalPlan without AnalysisBarrie
Github user TomaszGaweda commented on the issue:
https://github.com/apache/spark/pull/21360
@viirya I've written it in the ticket. In my case, pushing down ORs with
non-equality predicates caused DB2 to slow down; workaround was to cache data
before filtering, it was approx. 10 times
30 matches
Mail list logo
