Github user danielvdende closed the pull request at:
https://github.com/apache/spark/pull/18304
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/18304
Sounds good to me :+1: @HyukjinKwon
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Just made the changes you mentioned @gatorsmile :-)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r203955129
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -102,7 +102,12 @@ object JdbcUtils extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@dongjoon-hyun @gatorsmile Any update, guys?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@gatorsmile @dongjoon-hyun Guys, any update?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Gentle ping again @gatorsmile @dongjoon-hyun :-)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Gentle ping @gatorsmile :-)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@dongjoon-hyun I saw that Spark 2.3 was released a few days ago, congrats
on the release! :-) Is there anything stopping us from merging this PR into
master now
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@dongjoon-hyun Made the changes you pointed out, thanks! ð
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r169583476
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/TeradataDialect.scala ---
@@ -31,4 +31,19 @@ private case object TeradataDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r169025602
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala ---
@@ -85,15 +85,24 @@ private object PostgresDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168943159
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala ---
@@ -85,15 +85,24 @@ private object PostgresDialect extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Thanks again @dongjoon-hyun, much appreciated ð. I've made changes
accordingly, and added comments in this PR where appropriate
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168929874
--- Diff: docs/sql-programming-guide.md ---
@@ -1372,6 +1372,13 @@ the following case-insensitive options:
This is a JDBC writer related
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168929798
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala ---
@@ -94,5 +94,21 @@ private case object OracleDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168929792
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala ---
@@ -42,4 +42,17 @@ private object MsSqlServerDialect extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Thanks for your review @dongjoon-hyun ð I've corrected all the
indentation problems (didn't show up locally when running scalastyle checks for
some reason). Added comments where necessary
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168922010
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -860,14 +860,41 @@ class JDBCSuite extends SparkFunSuite
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168922006
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -860,14 +860,41 @@ class JDBCSuite extends SparkFunSuite
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168921851
--- Diff: docs/sql-programming-guide.md ---
@@ -1372,6 +1372,13 @@ the following case-insensitive options:
This is a JDBC writer related
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168921833
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -102,7 +102,12 @@ object JdbcUtils extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r168921808
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala
---
@@ -119,6 +119,8 @@ class JDBCOptions
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Thanks guys! @gatorsmile @dongjoon-hyun
Happy to help out expanding Spark SQL JDBC where necessary to match and
improve on sqoop
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Hi guys, @Fokko @gatorsmile, completely agree with what @Fokko mentioned,
our main reason for wanting to get away from Sqoop is also for stability
reasons and to get rid of MapReduce
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Hmm, not it fails the OrcQuerySuite. This PR doesn't touch any of the Orc
implementation in Spark. Could this be a flaky test @gatorsmile
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
Tests are failing on a spark streaming test. I think it's probably because
of the age of this PR, will rebase to get the changes into the PR that were
merged into master since I opened the PR
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@dongjoon-hyun @gatorsmile sorry to keep asking, but could you let me know
when we can get this merged
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@Stephan202 thanks for pointing out those docs issues, just pushed the
changes :-).
@gatorsmile @dongjoon-hyun would you have a chance to take a look at this
again
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@dongjoon-hyun @gatorsmile any further update?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@dongjoon-hyun @gatorsmile any further thoughts on this?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
I've made some changes to the code; the hardcoded `false` value for
`cascade` in `JdbcDialects` is now replaced by a default value of
`isCascadingTruncateTable`. This is also the case for each
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r159371171
--- Diff: docs/sql-programming-guide.md ---
@@ -1339,6 +1339,13 @@ the following case-insensitive options:
This is a JDBC writer related
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r159370871
--- Diff: docs/sql-programming-guide.md ---
@@ -1339,6 +1339,13 @@ the following case-insensitive options:
This is a JDBC writer related
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r159183234
--- Diff: docs/sql-programming-guide.md ---
@@ -1339,6 +1339,13 @@ the following case-insensitive options:
This is a JDBC writer related
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@gatorsmile would be great to hear why you doubt the value of the feature
:). I know that for us it would be extremely valuable (at the moment we have to
do an extra step in our data pipeline
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@gatorsmile could you explain why you have doubts about the feature? Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r158557222
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -120,11 +121,12 @@ abstract class JdbcDialect extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
I think the default for all dialects should be false, regardless of whether
the cascade feature is even supported. And for those for which it is supported,
it should default to false
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/20057#discussion_r158555400
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -120,11 +121,12 @@ abstract class JdbcDialect extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/20057
@dongjoon-hyun this is the functionality we discussed in PR for
SPARK-22729, would be great to hear your opinion
GitHub user danielvdende opened a pull request:
https://github.com/apache/spark/pull/20057
[SPARK-22880][SQL] Add cascadeTruncate option to JDBC datasource
This commit adds the `cascadeTruncate` option to the JDBC datasource
API, for databases that support this functionality
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
@srowen Sorry, should have been clearer, I meant the triggering of the
tests (I see they're running now, thanks
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
Forgive my impatience, but is it supposed to take this long? @dongjoon-hyun
@gatorsmile
---
-
To unsubscribe, e-mail
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
@dongjoon-hyun @gatorsmile As @gatorsmile pointed out, the
`isCascadingTruncateTable` is a method in the public API, so we can't just drop
it. I've made changes again, now the truncate query
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155916550
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -147,14 +157,6 @@ abstract class JdbcDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155916248
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -147,14 +157,6 @@ abstract class JdbcDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155622687
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/DB2Dialect.scala ---
@@ -48,5 +48,7 @@ private object DB2Dialect extends JdbcDialect
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
@dongjoon-hyun ok made the changes, also replaced the test that was in
place for `isCascadingTruncateTable` with one for the `getTruncateQuery`
method. Right now, I've left the method
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
@dongjoon-hyun I was considering this. Wanted to hear your opinion on it
first ;-). I'll make those changes
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155515480
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala ---
@@ -85,6 +85,10 @@ private object PostgresDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155515466
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -116,6 +116,17 @@ abstract class JdbcDialect extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
@dongjoon-hyun I've addressed the issues you mentioned :)
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155477421
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala ---
@@ -85,6 +85,10 @@ private object PostgresDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155477578
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -116,6 +116,17 @@ abstract class JdbcDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155477436
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -116,6 +116,17 @@ abstract class JdbcDialect extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
@dongjoon-hyun I made the changes as suggested by @bolkedebruin (so
modifying the Postgres dialect to use `TRUNCATE ONLY`). Let me know what you
think :) (and if I missed anything
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
Sure, I'll make the changes :). I'll use this PR, seems to make sense, as
it would fix truncate for postgres
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155335791
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala ---
@@ -100,5 +100,5 @@ private object PostgresDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155331702
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala ---
@@ -100,5 +100,5 @@ private object PostgresDialect extends
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/19911#discussion_r155330583
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala ---
@@ -100,5 +100,5 @@ private object PostgresDialect extends
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/19911
@srowen yep, I can add the details from the JIRA to the PR if you like
(just to make it easier to read this PR in future if necessary
GitHub user danielvdende opened a pull request:
https://github.com/apache/spark/pull/19911
[SPARK-21098] Correct cascade default for postgres
The PostgresDialect indicates that cascade is true by default for Postgres.
This is not the case. This patch fixes this.
## What
Github user danielvdende commented on the issue:
https://github.com/apache/spark/pull/18304
@HyukjinKwon any further update on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122194350
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122189644
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122144727
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val
Github user danielvdende commented on a diff in the pull request:
https://github.com/apache/spark/pull/18304#discussion_r122135698
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -90,6 +90,7 @@ class CSVOptions(
val
GitHub user danielvdende opened a pull request:
https://github.com/apache/spark/pull/18304
[SPARK-21098] Add lineseparator parameter to csv options
This commit adds the lineSeparator option to the spark support for
reading/writing csv files.
## What changes were
70 matches
Mail list logo
