[GitHub] spark pull request #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/21404#discussion_r215232389
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
---
@@ -99,6 +99,7 @@ private[hive] object IsolatedClientLoader extends Logging
{
case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
case "2.2" | "2.2.0" => hive.v2_2
case "2.3" | "2.3.0" | "2.3.1" | "2.3.2" | "2.3.3" => hive.v2_3
+case "3.0" | "3.0.0" => hive.v3_0
--- End diff --
@dongjoon-hyun Please update sql-programming-guide.md:
https://github.com/apache/spark/blob/05974f9431e9718a5f331a9892b7d81aca8387a6/docs/sql-programming-guide.md#L1217
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21404#discussion_r190101719
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -1148,3 +1149,128 @@ private[client] class Shim_v2_1 extends Shim_v2_0 {
private[client] class Shim_v2_2 extends Shim_v2_1
private[client] class Shim_v2_3 extends Shim_v2_1
+
+private[client] class Shim_v3_0 extends Shim_v2_3 {
+ // Spark supports only non-ACID operations
+ protected lazy val isAcidIUDoperation = JBoolean.FALSE
+
+ // Writer ID can be 0 for non-ACID operations
+ protected lazy val writeIdInLoadTableOrPartition: JLong = 0L
+
+ // Statement ID
+ protected lazy val stmtIdInLoadTableOrPartition: JInteger = 0
+
+ protected lazy val listBucketingLevel: JInteger = 0
+
+ private lazy val clazzLoadFileType = getClass.getClassLoader.loadClass(
+"org.apache.hadoop.hive.ql.plan.LoadTableDesc$LoadFileType")
+
+ private lazy val loadPartitionMethod =
--- End diff --
Thank you, @HyukjinKwon .
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21404#discussion_r190097176
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -1148,3 +1149,128 @@ private[client] class Shim_v2_1 extends Shim_v2_0 {
private[client] class Shim_v2_2 extends Shim_v2_1
private[client] class Shim_v2_3 extends Shim_v2_1
+
+private[client] class Shim_v3_0 extends Shim_v2_3 {
+ // Spark supports only non-ACID operations
+ protected lazy val isAcidIUDoperation = JBoolean.FALSE
+
+ // Writer ID can be 0 for non-ACID operations
+ protected lazy val writeIdInLoadTableOrPartition: JLong = 0L
+
+ // Statement ID
+ protected lazy val stmtIdInLoadTableOrPartition: JInteger = 0
+
+ protected lazy val listBucketingLevel: JInteger = 0
+
+ private lazy val clazzLoadFileType = getClass.getClassLoader.loadClass(
+"org.apache.hadoop.hive.ql.plan.LoadTableDesc$LoadFileType")
+
+ private lazy val loadPartitionMethod =
--- End diff --
BTW, I tracked and checked all the signature changed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/21404#discussion_r190088876
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala ---
@@ -79,7 +79,12 @@ package object client {
exclusions = Seq("org.apache.curator:*",
"org.pentaho:pentaho-aggdesigner-algorithm"))
-val allSupportedHiveVersions = Set(v12, v13, v14, v1_0, v1_1, v1_2,
v2_0, v2_1, v2_2, v2_3)
+case object v3_0 extends HiveVersion("3.0.0",
+ exclusions = Seq("org.apache.curator:*",
+"org.apache.hadoop:hadoop-aws",
--- End diff --
Thanks. I'll remove this in this PR.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21404#discussion_r190086082
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala ---
@@ -79,7 +79,12 @@ package object client {
exclusions = Seq("org.apache.curator:*",
"org.pentaho:pentaho-aggdesigner-algorithm"))
-val allSupportedHiveVersions = Set(v12, v13, v14, v1_0, v1_1, v1_2,
v2_0, v2_1, v2_2, v2_3)
+case object v3_0 extends HiveVersion("3.0.0",
+ exclusions = Seq("org.apache.curator:*",
+"org.apache.hadoop:hadoop-aws",
--- End diff --
What happened if we do not have this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21404: [SPARK-24360][SQL] Support Hive 3.0 metastore
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/21404 [SPARK-24360][SQL] Support Hive 3.0 metastore ## What changes were proposed in this pull request? Hive 3.0 is released. This PR aims to support Hive 3.0 metastore. ## How was this patch tested? Pass the Jenkins with the updated test cases including 3.0. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dongjoon-hyun/spark SPARK-24360 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21404.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21404 commit 1523b6592725c3ca465ec0b6b63167cb4e6e3aec Author: Dongjoon Hyun Date: 2018-05-22T17:16:57Z [SPARK-24360][SQL] Support Hive 3.0 metastore --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
