[GitHub] spark issue #19238: [SPARK-22016][SQL] Add HiveDialect for JDBC connection t...
Github user danielfx90 commented on the issue: https://github.com/apache/spark/pull/19238 Seems logical. Then, unless someone disagrees, feel free to close this PR and we will create a new spark package with this feature in a new repository. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19238: [SPARK-22016][SQL] Add HiveDialect for JDBC conne...
Github user danielfx90 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19238#discussion_r139441849
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -1103,6 +1103,17 @@ class JDBCSuite extends SparkFunSuite
""".stripMargin)
val df3 = sql("SELECT * FROM test_sessionInitStatement")
- assert(df3.collect() === Array(Row(21519, 1234)))
-}
+ assert(df3.collect() === Array(Row(21519, 1234))
+)
--- End diff --
@dongjoon-hyun You are right! I misread the parenthesis. I think now is
correct. Thank you for the observation :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19238: [SPARK-22016][SQL] Add HiveDialect for JDBC conne...
Github user danielfx90 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19238#discussion_r139202470
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -1103,6 +1103,17 @@ class JDBCSuite extends SparkFunSuite
""".stripMargin)
val df3 = sql("SELECT * FROM test_sessionInitStatement")
- assert(df3.collect() === Array(Row(21519, 1234)))
-}
+ assert(df3.collect() === Array(Row(21519, 1234))
+)
--- End diff --
@dongjoon-hyun done! Thank you!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19238: [SPARK-22016][SQL] Add HiveDialect for JDBC conne...
Github user danielfx90 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19238#discussion_r139201902
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -1103,6 +1103,17 @@ class JDBCSuite extends SparkFunSuite
""".stripMargin)
val df3 = sql("SELECT * FROM test_sessionInitStatement")
- assert(df3.collect() === Array(Row(21519, 1234)))
-}
+ assert(df3.collect() === Array(Row(21519, 1234))
+)
--- End diff --
It must have changed when formatting the code using the IDE. Scalastyle
checks passed though, but let me rollback that anyway.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19238: [SPARK-22016][SQL] Add HiveDialect for JDBC connection t...
Github user danielfx90 commented on the issue: https://github.com/apache/spark/pull/19238 @gatorsmile if Hive lies on the same infrastructure as the application, then the metastore should definitely solve the issue, but a connection over JDBC is needed when data comes from an external source which **only** exposes such a connection through its Hive server. We encountered this and ended up adding the HiveDialect to solve it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19238: [SPARK-22016][SQL] Add HiveDialect for JDBC conne...
GitHub user danielfx90 opened a pull request: https://github.com/apache/spark/pull/19238 [SPARK-22016][SQL] Add HiveDialect for JDBC connection to Hive ## What changes were proposed in this pull request? Added a HiveDialect for JDBC connection to Hive. It overrides two methods: * canHandle * quoteIdentifier ## How was this patch tested? It passes the added tests and it was used with a real Hive instance with real data. You can merge this pull request into a Git repository by running: $ git pull https://github.com/GranData/spark hive-dialect Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19238.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19238 commit 3f486bee879c6711851b310e37839491a3b93270 Author: Daniel Fernandez Date: 2017-09-14T21:11:40Z HiveDialect implementation done commit c0d262455633b7d302124e963b05efb7afda2dcb Author: Daniel Fernandez Date: 2017-09-14T21:12:01Z HiveDialect registration added commit f704950eedcd7662db4020eae70aa5351bb3e611 Author: Daniel Fernandez Date: 2017-09-14T21:12:18Z Tests for the HiveDialect added --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
