[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6627 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111863871 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32382879
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala ---
@@ -19,15 +19,27 @@ package org.apache.spark.sql.hive
/** Support for interacting with different versions of the
HiveMetastoreClient */
package object client {
- private[client] abstract class HiveVersion(val fullVersion: String, val
hasBuiltinsJar: Boolean)
+ private[client] abstract class HiveVersion(
+ val fullVersion: String,
+ val extraDeps: Seq[String] = Nil,
+ val exclusions: Seq[String] = Nil)
// scalastyle:off
private[client] object hive {
-case object v10 extends HiveVersion("0.10.0", true)
-case object v11 extends HiveVersion("0.11.0", false)
-case object v12 extends HiveVersion("0.12.0", false)
-case object v13 extends HiveVersion("0.13.1", false)
+case object v12 extends HiveVersion("0.12.0")
+case object v13 extends HiveVersion("0.13.1")
+
+// Hive 0.14 depends on calcite 0.9.2-incubating-SNAPSHOT which does
not exist in
+// maven central anymore, so override those with a version that exists.
+//
+// org.pentaho:pentaho-aggdesigner-algorithm is also nowhere to be
found, so exclude
+// it explicitly. If it's needed by the metastore client, users will
have to dig it
+// out of somewhere and use configuration to point Spark at the
correct jars.
--- End diff --
The JIRA I was thinking of would ask them to make sure that future versions
of Hive can be used as a library simply by depending on them in maven. Seems
like they should be able to ensure future releases don't depend on SNAPSHOT
dependencies and whatnot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111336878 [Test build #34735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34735/console) for PR 6627 at commit [`3fa4270`](https://github.com/apache/spark/commit/3fa42702a1060fd618baefc39344e32757a71d89). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111336882 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111314541 [Test build #34735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34735/consoleFull) for PR 6627 at commit [`3fa4270`](https://github.com/apache/spark/commit/3fa42702a1060fd618baefc39344e32757a71d89). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111314439 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111314425 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32282547
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala ---
@@ -19,15 +19,27 @@ package org.apache.spark.sql.hive
/** Support for interacting with different versions of the
HiveMetastoreClient */
package object client {
- private[client] abstract class HiveVersion(val fullVersion: String, val
hasBuiltinsJar: Boolean)
+ private[client] abstract class HiveVersion(
+ val fullVersion: String,
+ val extraDeps: Seq[String] = Nil,
+ val exclusions: Seq[String] = Nil)
// scalastyle:off
private[client] object hive {
-case object v10 extends HiveVersion("0.10.0", true)
-case object v11 extends HiveVersion("0.11.0", false)
-case object v12 extends HiveVersion("0.12.0", false)
-case object v13 extends HiveVersion("0.13.1", false)
+case object v12 extends HiveVersion("0.12.0")
+case object v13 extends HiveVersion("0.13.1")
+
+// Hive 0.14 depends on calcite 0.9.2-incubating-SNAPSHOT which does
not exist in
+// maven central anymore, so override those with a version that exists.
+//
+// org.pentaho:pentaho-aggdesigner-algorithm is also nowhere to be
found, so exclude
+// it explicitly. If it's needed by the metastore client, users will
have to dig it
+// out of somewhere and use configuration to point Spark at the
correct jars.
--- End diff --
> maybe we should open a JIRA for Hive though
Not sure how much they'd be able to fix; the dependencies that have
disappeared are not Hive artifacts, and those Hive versions have already been
published.
(Or maybe those have never been on maven central and you're supposed to use
a 3rd-party repo to get them?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32282458
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.client
+
+import java.lang.{Boolean => JBoolean, Integer => JInteger}
+import java.lang.reflect.{Method, Modifier}
+import java.net.URI
+import java.util.{ArrayList => JArrayList, List => JList, Map => JMap, Set
=> JSet}
+
+import scala.collection.JavaConversions._
+
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.ql.Driver
+import org.apache.hadoop.hive.ql.metadata.{Hive, Partition, Table}
+import org.apache.hadoop.hive.ql.processors.{CommandProcessor,
CommandProcessorFactory}
+import org.apache.hadoop.hive.ql.session.SessionState
+
+/**
+ * A shim that defines the interface between ClientWrapper and the
underlying Hive library used to
+ * talk to the metastore. Each Hive version has its own implementation of
this class, defining
+ * version-specific version of needed functions.
+ *
+ * The guideline for writing shims is:
+ * - always extend from the previous version unless really not possible
+ * - initialize methods in lazy vals, both for quicker access for multiple
invocations, and to
+ * avoid runtime errors due to the above guideline.
+ */
+private[client] sealed abstract class Shim {
+
+ def setCurrentSessionState(state: SessionState): Unit
+
+ /**
+ * This shim is necessary because the return type is different on
different versions of Hive.
+ * All parameters are the same, though.
+ */
+ def getDataLocation(table: Table): Option[String]
+
+ def setDataLocation(table: Table, loc: String): Unit
+
+ def getAllPartitions(hive: Hive, table: Table): Seq[Partition]
+
+ def getCommandProcessor(token: String, conf: HiveConf): CommandProcessor
+
+ def getDriverResults(driver: Driver): Seq[String]
+
+ def loadPartition(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ holdDDLTime: Boolean,
+ inheritTableSpecs: Boolean,
+ isSkewedStoreAsSubdir: Boolean): Unit
+
+ def loadTable(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ replace: Boolean,
+ holdDDLTime: Boolean): Unit
+
+ def loadDynamicPartitions(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ numDP: Int,
+ holdDDLTime: Boolean,
+ listBucketingEnabled: Boolean): Unit
+
+ protected def findStaticMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+val method = findMethod(klass, name, args: _*)
+require(Modifier.isStatic(method.getModifiers()),
+ s"Method $name of class $klass is not static.")
+method
+ }
+
+ protected def findMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+klass.getMethod(name, args: _*)
+ }
+
+}
+
+private[client] class Shim_v0_12 extends Shim {
+
+ private lazy val startMethod = findStaticMethod(classOf[SessionState],
"start",
--- End diff --
Yes, because otherwise you can't have the shim for the next version extend
this one. I mention this in the scaladoc for the `Shim` class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32282420
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.client
+
+import java.lang.{Boolean => JBoolean, Integer => JInteger}
+import java.lang.reflect.{Method, Modifier}
+import java.net.URI
+import java.util.{ArrayList => JArrayList, List => JList, Map => JMap, Set
=> JSet}
+
+import scala.collection.JavaConversions._
+
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.ql.Driver
+import org.apache.hadoop.hive.ql.metadata.{Hive, Partition, Table}
+import org.apache.hadoop.hive.ql.processors.{CommandProcessor,
CommandProcessorFactory}
+import org.apache.hadoop.hive.ql.session.SessionState
+
+/**
+ * A shim that defines the interface between ClientWrapper and the
underlying Hive library used to
+ * talk to the metastore. Each Hive version has its own implementation of
this class, defining
+ * version-specific version of needed functions.
+ *
+ * The guideline for writing shims is:
+ * - always extend from the previous version unless really not possible
+ * - initialize methods in lazy vals, both for quicker access for multiple
invocations, and to
+ * avoid runtime errors due to the above guideline.
+ */
+private[client] sealed abstract class Shim {
+
+ def setCurrentSessionState(state: SessionState): Unit
+
+ /**
+ * This shim is necessary because the return type is different on
different versions of Hive.
+ * All parameters are the same, though.
+ */
+ def getDataLocation(table: Table): Option[String]
+
+ def setDataLocation(table: Table, loc: String): Unit
+
+ def getAllPartitions(hive: Hive, table: Table): Seq[Partition]
+
+ def getCommandProcessor(token: String, conf: HiveConf): CommandProcessor
+
+ def getDriverResults(driver: Driver): Seq[String]
+
+ def loadPartition(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ holdDDLTime: Boolean,
+ inheritTableSpecs: Boolean,
+ isSkewedStoreAsSubdir: Boolean): Unit
+
+ def loadTable(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ replace: Boolean,
+ holdDDLTime: Boolean): Unit
+
+ def loadDynamicPartitions(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ numDP: Int,
+ holdDDLTime: Boolean,
+ listBucketingEnabled: Boolean): Unit
+
+ protected def findStaticMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+val method = findMethod(klass, name, args: _*)
+require(Modifier.isStatic(method.getModifiers()),
+ s"Method $name of class $klass is not static.")
+method
+ }
+
+ protected def findMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+klass.getMethod(name, args: _*)
+ }
+
+}
+
+private[client] class Shim_v0_12 extends Shim {
+
+ private lazy val startMethod = findStaticMethod(classOf[SessionState],
"start",
+classOf[SessionState])
+ private lazy val getDataLocationMethod = findMethod(classOf[Table],
"getDataLocation")
+ private lazy val setDataLocationMethod = findMethod(classOf[Table],
"setDataLocation",
+classOf[URI])
+ private lazy val getAllPartitionsMethod = findMethod(classOf[Hive],
"getAllPartitionsForPruner",
+classOf[Table])
+ private lazy val getCommandProcessorMethod =
findStaticMethod(classOf[CommandProcessorFactory],
+"get", classOf[String], classOf[HiveConf])
+ private lazy val getDr
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111303675 I'm okay with this change. A few minor comments otherwise LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-111303709 Thanks for doing the upgrade BTW! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32278001
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.client
+
+import java.lang.{Boolean => JBoolean, Integer => JInteger}
+import java.lang.reflect.{Method, Modifier}
+import java.net.URI
+import java.util.{ArrayList => JArrayList, List => JList, Map => JMap, Set
=> JSet}
+
+import scala.collection.JavaConversions._
+
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.ql.Driver
+import org.apache.hadoop.hive.ql.metadata.{Hive, Partition, Table}
+import org.apache.hadoop.hive.ql.processors.{CommandProcessor,
CommandProcessorFactory}
+import org.apache.hadoop.hive.ql.session.SessionState
+
+/**
+ * A shim that defines the interface between ClientWrapper and the
underlying Hive library used to
+ * talk to the metastore. Each Hive version has its own implementation of
this class, defining
+ * version-specific version of needed functions.
+ *
+ * The guideline for writing shims is:
+ * - always extend from the previous version unless really not possible
+ * - initialize methods in lazy vals, both for quicker access for multiple
invocations, and to
+ * avoid runtime errors due to the above guideline.
+ */
+private[client] sealed abstract class Shim {
+
+ def setCurrentSessionState(state: SessionState): Unit
+
+ /**
+ * This shim is necessary because the return type is different on
different versions of Hive.
+ * All parameters are the same, though.
+ */
+ def getDataLocation(table: Table): Option[String]
+
+ def setDataLocation(table: Table, loc: String): Unit
+
+ def getAllPartitions(hive: Hive, table: Table): Seq[Partition]
+
+ def getCommandProcessor(token: String, conf: HiveConf): CommandProcessor
+
+ def getDriverResults(driver: Driver): Seq[String]
+
+ def loadPartition(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ holdDDLTime: Boolean,
+ inheritTableSpecs: Boolean,
+ isSkewedStoreAsSubdir: Boolean): Unit
+
+ def loadTable(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ replace: Boolean,
+ holdDDLTime: Boolean): Unit
+
+ def loadDynamicPartitions(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ numDP: Int,
+ holdDDLTime: Boolean,
+ listBucketingEnabled: Boolean): Unit
+
+ protected def findStaticMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+val method = findMethod(klass, name, args: _*)
+require(Modifier.isStatic(method.getModifiers()),
+ s"Method $name of class $klass is not static.")
+method
+ }
+
+ protected def findMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+klass.getMethod(name, args: _*)
+ }
+
+}
+
+private[client] class Shim_v0_12 extends Shim {
+
+ private lazy val startMethod = findStaticMethod(classOf[SessionState],
"start",
--- End diff --
Minor, but do they actually have to be `lazy`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32277894
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala ---
@@ -19,15 +19,27 @@ package org.apache.spark.sql.hive
/** Support for interacting with different versions of the
HiveMetastoreClient */
package object client {
- private[client] abstract class HiveVersion(val fullVersion: String, val
hasBuiltinsJar: Boolean)
+ private[client] abstract class HiveVersion(
+ val fullVersion: String,
+ val extraDeps: Seq[String] = Nil,
+ val exclusions: Seq[String] = Nil)
// scalastyle:off
private[client] object hive {
-case object v10 extends HiveVersion("0.10.0", true)
-case object v11 extends HiveVersion("0.11.0", false)
-case object v12 extends HiveVersion("0.12.0", false)
-case object v13 extends HiveVersion("0.13.1", false)
+case object v12 extends HiveVersion("0.12.0")
+case object v13 extends HiveVersion("0.13.1")
+
+// Hive 0.14 depends on calcite 0.9.2-incubating-SNAPSHOT which does
not exist in
+// maven central anymore, so override those with a version that exists.
+//
+// org.pentaho:pentaho-aggdesigner-algorithm is also nowhere to be
found, so exclude
+// it explicitly. If it's needed by the metastore client, users will
have to dig it
+// out of somewhere and use configuration to point Spark at the
correct jars.
--- End diff --
By default Spark SQL will use the hive jars that are bundled with Spark,
and for read deployments I expect most users will do as @vanzin suggests and
set `spark.sql.hive.metastore.jars`. This is really just for testing all the
versions easily. If someone wants to come up with a way to easily package all
the various hive versions and find a way to get them on to jenkins / other
places people want to run the tests we could use that instead.
@vanzin maybe we should open a JIRA for Hive though, as it would be good if
we can depend on them directly eventually and the probably don't realize that
is not possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32277322
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.client
+
+import java.lang.{Boolean => JBoolean, Integer => JInteger}
+import java.lang.reflect.{Method, Modifier}
+import java.net.URI
+import java.util.{ArrayList => JArrayList, List => JList, Map => JMap, Set
=> JSet}
+
+import scala.collection.JavaConversions._
+
+import org.apache.hadoop.fs.Path
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.ql.Driver
+import org.apache.hadoop.hive.ql.metadata.{Hive, Partition, Table}
+import org.apache.hadoop.hive.ql.processors.{CommandProcessor,
CommandProcessorFactory}
+import org.apache.hadoop.hive.ql.session.SessionState
+
+/**
+ * A shim that defines the interface between ClientWrapper and the
underlying Hive library used to
+ * talk to the metastore. Each Hive version has its own implementation of
this class, defining
+ * version-specific version of needed functions.
+ *
+ * The guideline for writing shims is:
+ * - always extend from the previous version unless really not possible
+ * - initialize methods in lazy vals, both for quicker access for multiple
invocations, and to
+ * avoid runtime errors due to the above guideline.
+ */
+private[client] sealed abstract class Shim {
+
+ def setCurrentSessionState(state: SessionState): Unit
+
+ /**
+ * This shim is necessary because the return type is different on
different versions of Hive.
+ * All parameters are the same, though.
+ */
+ def getDataLocation(table: Table): Option[String]
+
+ def setDataLocation(table: Table, loc: String): Unit
+
+ def getAllPartitions(hive: Hive, table: Table): Seq[Partition]
+
+ def getCommandProcessor(token: String, conf: HiveConf): CommandProcessor
+
+ def getDriverResults(driver: Driver): Seq[String]
+
+ def loadPartition(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ holdDDLTime: Boolean,
+ inheritTableSpecs: Boolean,
+ isSkewedStoreAsSubdir: Boolean): Unit
+
+ def loadTable(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ replace: Boolean,
+ holdDDLTime: Boolean): Unit
+
+ def loadDynamicPartitions(
+ hive: Hive,
+ loadPath: Path,
+ tableName: String,
+ partSpec: JMap[String, String],
+ replace: Boolean,
+ numDP: Int,
+ holdDDLTime: Boolean,
+ listBucketingEnabled: Boolean): Unit
+
+ protected def findStaticMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+val method = findMethod(klass, name, args: _*)
+require(Modifier.isStatic(method.getModifiers()),
+ s"Method $name of class $klass is not static.")
+method
+ }
+
+ protected def findMethod(klass: Class[_], name: String, args:
Class[_]*): Method = {
+klass.getMethod(name, args: _*)
+ }
+
+}
+
+private[client] class Shim_v0_12 extends Shim {
+
+ private lazy val startMethod = findStaticMethod(classOf[SessionState],
"start",
+classOf[SessionState])
+ private lazy val getDataLocationMethod = findMethod(classOf[Table],
"getDataLocation")
+ private lazy val setDataLocationMethod = findMethod(classOf[Table],
"setDataLocation",
+classOf[URI])
+ private lazy val getAllPartitionsMethod = findMethod(classOf[Hive],
"getAllPartitionsForPruner",
+classOf[Table])
+ private lazy val getCommandProcessorMethod =
findStaticMethod(classOf[CommandProcessorFactory],
+"get", classOf[String], classOf[HiveConf])
+ private lazy val get
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32068495
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala ---
@@ -19,15 +19,27 @@ package org.apache.spark.sql.hive
/** Support for interacting with different versions of the
HiveMetastoreClient */
package object client {
- private[client] abstract class HiveVersion(val fullVersion: String, val
hasBuiltinsJar: Boolean)
+ private[client] abstract class HiveVersion(
+ val fullVersion: String,
+ val extraDeps: Seq[String] = Nil,
+ val exclusions: Seq[String] = Nil)
// scalastyle:off
private[client] object hive {
-case object v10 extends HiveVersion("0.10.0", true)
-case object v11 extends HiveVersion("0.11.0", false)
-case object v12 extends HiveVersion("0.12.0", false)
-case object v13 extends HiveVersion("0.13.1", false)
+case object v12 extends HiveVersion("0.12.0")
+case object v13 extends HiveVersion("0.13.1")
+
+// Hive 0.14 depends on calcite 0.9.2-incubating-SNAPSHOT which does
not exist in
+// maven central anymore, so override those with a version that exists.
+//
+// org.pentaho:pentaho-aggdesigner-algorithm is also nowhere to be
found, so exclude
+// it explicitly. If it's needed by the metastore client, users will
have to dig it
+// out of somewhere and use configuration to point Spark at the
correct jars.
--- End diff --
I'm sure there's gonna be some documentation needed; relying on Ivy to
download these things is kind of a bad thing to start with (and I'd argue that
it should be disabled by default).
In my view, the normal process would be for users to set the location of
Hive jars in their configuration (along with the version to use) instead of
relying on this. I think I saw an option for that but don't remember off the
top of my head.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110514359 code changes lgtm. I'm a little concerned about the dependency issues, but I'm not sure what to do about it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32068173
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala ---
@@ -19,15 +19,27 @@ package org.apache.spark.sql.hive
/** Support for interacting with different versions of the
HiveMetastoreClient */
package object client {
- private[client] abstract class HiveVersion(val fullVersion: String, val
hasBuiltinsJar: Boolean)
+ private[client] abstract class HiveVersion(
+ val fullVersion: String,
+ val extraDeps: Seq[String] = Nil,
+ val exclusions: Seq[String] = Nil)
// scalastyle:off
private[client] object hive {
-case object v10 extends HiveVersion("0.10.0", true)
-case object v11 extends HiveVersion("0.11.0", false)
-case object v12 extends HiveVersion("0.12.0", false)
-case object v13 extends HiveVersion("0.13.1", false)
+case object v12 extends HiveVersion("0.12.0")
+case object v13 extends HiveVersion("0.13.1")
+
+// Hive 0.14 depends on calcite 0.9.2-incubating-SNAPSHOT which does
not exist in
+// maven central anymore, so override those with a version that exists.
+//
+// org.pentaho:pentaho-aggdesigner-algorithm is also nowhere to be
found, so exclude
+// it explicitly. If it's needed by the metastore client, users will
have to dig it
+// out of somewhere and use configuration to point Spark at the
correct jars.
--- End diff --
this comment is kinda scary ... do we need to document this somewhere?
though I'm not sure what the appropriate place would be
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32067835
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -27,7 +27,7 @@ import scala.language.reflectiveCalls
import org.apache.hadoop.fs.Path
import org.apache.hadoop.hive.metastore.api.Database
import org.apache.hadoop.hive.conf.HiveConf
-import org.apache.hadoop.hive.metastore.TableType
+import org.apache.hadoop.hive.metastore.{TableType => HTableType}
--- End diff --
It gets rid of a scalac warning (since `TableType` is also defined in the
Spark side).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6627#discussion_r32067595
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -27,7 +27,7 @@ import scala.language.reflectiveCalls
import org.apache.hadoop.fs.Path
import org.apache.hadoop.hive.metastore.api.Database
import org.apache.hadoop.hive.conf.HiveConf
-import org.apache.hadoop.hive.metastore.TableType
+import org.apache.hadoop.hive.metastore.{TableType => HTableType}
--- End diff --
this change isn't required, is it? just curious, I don't particularly care
one way or the other
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110152327 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110152313 [Test build #34462 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34462/console) for PR 6627 at commit [`4b8a3d4`](https://github.com/apache/spark/commit/4b8a3d428d246f81de91da048a0dc24c64734619). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110115733 [Test build #34462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34462/consoleFull) for PR 6627 at commit [`4b8a3d4`](https://github.com/apache/spark/commit/4b8a3d428d246f81de91da048a0dc24c64734619). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110115296 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110115280 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110112767 [Test build #34453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34453/console) for PR 6627 at commit [`be3d0cc`](https://github.com/apache/spark/commit/be3d0cc1c2182828368e5b8b9c5edd7bf4873595). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110112799 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110090341 [Test build #34453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34453/consoleFull) for PR 6627 at commit [`be3d0cc`](https://github.com/apache/spark/commit/be3d0cc1c2182828368e5b8b9c5edd7bf4873595). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110090151 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110090120 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-110072897 I believe the API in 1.0/1.1/1.2 is very similar, if not identical, to 0.14. But this is a problem that we'll have to live with unless Spark is willing to reimplement some internal Hive APIs. While some of the metastore APIs used exist in public/stable form, at least a few (loadTable/loadPartitions) do not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109951833 A high level question: how about the hive 1.0/1.1/1.2? It might be hard to support so many versions if there's no compatibility between them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109473502 **[Test build #34307 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34307/console)** for PR 6627 at commit [`ca3fb1e`](https://github.com/apache/spark/commit/ca3fb1e4702aebb89b184038c865349d8e842d59) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109473533 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109424128 [Test build #34307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34307/consoleFull) for PR 6627 at commit [`ca3fb1e`](https://github.com/apache/spark/commit/ca3fb1e4702aebb89b184038c865349d8e842d59). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109423537 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109423610 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109415345 [Test build #34290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34290/console) for PR 6627 at commit [`73bd161`](https://github.com/apache/spark/commit/73bd16199d500d7cf9d8777bb32b9bd416e48ef3). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109415356 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109368504 [Test build #34290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34290/consoleFull) for PR 6627 at commit [`73bd161`](https://github.com/apache/spark/commit/73bd16199d500d7cf9d8777bb32b9bd416e48ef3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109367996 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109367976 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109365049 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109365029 [Test build #34287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34287/console) for PR 6627 at commit [`d2ddf01`](https://github.com/apache/spark/commit/d2ddf01e825ff20264b70f5d7cbfe132d282b34b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109360735 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109360810 [Test build #34287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34287/consoleFull) for PR 6627 at commit [`d2ddf01`](https://github.com/apache/spark/commit/d2ddf01e825ff20264b70f5d7cbfe132d282b34b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109360751 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109190802 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109190793 [Test build #34252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34252/consoleFull) for PR 6627 at commit [`2c3c02e`](https://github.com/apache/spark/commit/2c3c02e3e24462a4b0febb47d497422ed9a760c6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109166348 [Test build #34252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34252/consoleFull) for PR 6627 at commit [`2c3c02e`](https://github.com/apache/spark/commit/2c3c02e3e24462a4b0febb47d497422ed9a760c6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109165979 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109166043 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109090773 org.pentaho#pentaho-aggdesigner-algorithm is still missing after my last attempt... not sure what's the best way to proceed now (maven central doesn't have that artifact at all, not even a different version). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109073035 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109073029 [Test build #34215 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34215/consoleFull) for PR 6627 at commit [`0a03470`](https://github.com/apache/spark/commit/0a0347033e2a5edd5dba86326cf5d1787c1649a5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109060685 [Test build #34215 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34215/consoleFull) for PR 6627 at commit [`0a03470`](https://github.com/apache/spark/commit/0a0347033e2a5edd5dba86326cf5d1787c1649a5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109060400 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109060448 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109058150 (Another minor update: the missing dependencies are all transitive dependencies of calcite. Not sure if it would be safe to upgrade the calcite version to a newer one. The same dependencies exist in Hive 1.0.0.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109049828 Or I could remove the tests for 0.14 (and you'd have to trust me that they pass). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109046340 The tests are not finding some dependencies that don't seem to exist in the main maven repo anymore but are available in other mirrors. Any suggestions for what to do here? I can add another repo, or try to override the dependencies (although I'm not sure that will work since at least one has changed groupId). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-109039290 I will let Michael comment on this. Explicitly not @ him so he won't see this during vacation; otherwise his heart might bleed... :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108993676 [Test build #34195 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34195/consoleFull) for PR 6627 at commit [`13b2dfa`](https://github.com/apache/spark/commit/13b2dfa2956f9eb96fdb40aaa4e11c854cbc825d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108993694 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108973729 [Test build #34195 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34195/consoleFull) for PR 6627 at commit [`13b2dfa`](https://github.com/apache/spark/commit/13b2dfa2956f9eb96fdb40aaa4e11c854cbc825d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108972366 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108972303 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108669462 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108669450 [Test build #34136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34136/consoleFull) for PR 6627 at commit [`6439d88`](https://github.com/apache/spark/commit/6439d88920ff25a88268ea43ba8c84e9822d46a4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108658550 [Test build #34133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34133/consoleFull) for PR 6627 at commit [`69b017b`](https://github.com/apache/spark/commit/69b017b4e40112bec4fc3b8753bcdc1c97c1c63d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108658558 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108653446 [Test build #34136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34136/consoleFull) for PR 6627 at commit [`6439d88`](https://github.com/apache/spark/commit/6439d88920ff25a88268ea43ba8c84e9822d46a4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108652819 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108652780 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108648178 [Test build #34133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34133/consoleFull) for PR 6627 at commit [`69b017b`](https://github.com/apache/spark/commit/69b017b4e40112bec4fc3b8753bcdc1c97c1c63d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108647752 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108647722 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108645240 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108645238 [Test build #34132 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34132/consoleFull) for PR 6627 at commit [`a21cad8`](https://github.com/apache/spark/commit/a21cad8ede9d37cb93d765efc2c8c02c9fa1e3e8). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108644333 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6627#issuecomment-108644297 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8065] [hive] Add support for Hive 0.14 ...
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/6627 [SPARK-8065] [hive] Add support for Hive 0.14 metastores This change has two parts. The first one gets rid of "ReflectionMagic". That worked well for the differences between 0.12 and 0.13, but breaks in 0.14, since some of the APIs that need to be used have primitive types. I could not figure out a way to make that class work with primitive types. So instead I wrote some shims (I can already hear the collective sigh) that find the appropriate methods via reflection. This should be faster since the method instances are cached, and the code is not much uglier than before, with the advantage that all the ugliness is local to one file (instead of multiple switch statements on the version being used scattered in ClientWrapper). The second part is simple: add code to handle Hive 0.14. A few new methods had to be added to the new shims. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vanzin/spark SPARK-8065 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6627.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6627 commit ae98c876da5679347a6878f92d48ea2298107eca Author: Marcelo Vanzin Date: 2015-06-03T18:06:02Z PART I: Get rid of reflection magic. The reflection magic stuff in sql/hive does not work with primitive Java types. Lots of APIs added in Hive 0.14 use primitive types in their signature, so there's no way to make them work with that code. Instead, have explicit shim-like classes that use reflection to call into actual Hive code. Each version has its own version-specific implementations of each needed method. commit a21cad8ede9d37cb93d765efc2c8c02c9fa1e3e8 Author: Marcelo Vanzin Date: 2015-06-03T23:10:37Z Part II: Add shims / version for Hive 0.14. Some dependency mess needed to be sorted out; in the process I removed the unused constants for v10 and v11 (which, if desired, can be restored later when someone decides to write shims for them). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
