[jira] [Updated] (SPARK-14171) UDAF aggregates argument object inspector not parsed correctly

2016-05-11 Thread Jianfeng Hu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfeng Hu updated SPARK-14171:

Priority: Critical  (was: Major)

> UDAF aggregates argument object inspector not parsed correctly
> --
>
> Key: SPARK-14171
> URL: https://issues.apache.org/jira/browse/SPARK-14171
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.6.1
>Reporter: Jianfeng Hu
>Priority: Critical
>
> For example, when using percentile_approx and count distinct together, it 
> raises an error complaining the argument is not constant. We have a test case 
> to reproduce. Could you help look into a fix of this? This was working in 
> previous version (Spark 1.4 + Hive 0.13). Thanks!
> {code}--- 
> a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
> +++ 
> b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
> @@ -148,6 +148,9 @@ class HiveUDFSuite extends QueryTest with 
> TestHiveSingleton with SQLTestUtils {
>  checkAnswer(sql("SELECT percentile_approx(100.0, array(0.9, 0.9)) FROM 
> src LIMIT 1"),
>sql("SELECT array(100, 100) FROM src LIMIT 1").collect().toSeq)
> +
> +checkAnswer(sql("SELECT percentile_approx(key, 0.9), count(distinct 
> key) FROM src LIMIT 1"),
> +  sql("SELECT max(key), 1 FROM src LIMIT 1").collect().toSeq)
> }
>test("UDFIntegerToString") {
> {code}
> When running the test suite, we can see this error:
> {code}
> - Generic UDAF aggregates *** FAILED ***
>   org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, 
> tree: 
> hiveudaffunction(HiveFunctionWrapper(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox,org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox@6e1dc6a7),key#51176,0.9,false,0,0)
>   at 
> org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)
>   at org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:357)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode.withNewChildren(TreeNode.scala:238)
>   at 
> org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter.org$apache$spark$sql$catalyst$analysis$DistinctAggregationRewriter$$patchAggregateFunctionChildren$1(DistinctAggregationRewriter.scala:148)
>   at 
> org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:192)
>   at 
> org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:190)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   ...
>   Cause: java.lang.reflect.InvocationTargetException:
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:368)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:367)
>   at 
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:365)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:357)
>   at 
> org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:48)
>   ...
>   Cause: org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second 
> argument must be a constant, but double was passed instead.
>   at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:147)
>   at 
> org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector$lzycompute(hiveUDFs.scala:598)
>   at 
> org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector(hiveUDFs.scala:596)
>   at 
> org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector$lzycompute(hiveUDFs.scala:606)
>   at 
> org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector(hiveUDFs.scala:606)
>   at org.apache.spark.sql.hive.HiveUDAFFunction.(hiveUDFs.scala:654)
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
>

[jira] [Updated] (SPARK-14171) UDAF aggregates argument object inspector not parsed correctly

2016-03-25 Thread Jianfeng Hu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfeng Hu updated SPARK-14171:

Description: 
For example, when using percentile_approx and count distinct together, it 
raises an error complaining the argument is not constant. We have a test case 
to reproduce. Could you help look into a fix of this? This was working in 
previous version (Spark 1.4 + Hive 0.13). Thanks!

{code}--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -148,6 +148,9 @@ class HiveUDFSuite extends QueryTest with TestHiveSingleton 
with SQLTestUtils {

 checkAnswer(sql("SELECT percentile_approx(100.0, array(0.9, 0.9)) FROM src 
LIMIT 1"),
   sql("SELECT array(100, 100) FROM src LIMIT 1").collect().toSeq)
+
+checkAnswer(sql("SELECT percentile_approx(key, 0.9), count(distinct 
key) FROM src LIMIT 1"),
+  sql("SELECT max(key), 1 FROM src LIMIT 1").collect().toSeq)
}

   test("UDFIntegerToString") {
{code}


When running the test suite, we can see this error:

{code}
- Generic UDAF aggregates *** FAILED ***
  org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, 
tree: 
hiveudaffunction(HiveFunctionWrapper(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox,org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox@6e1dc6a7),key#51176,0.9,false,0,0)
  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)
  at org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:357)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode.withNewChildren(TreeNode.scala:238)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter.org$apache$spark$sql$catalyst$analysis$DistinctAggregationRewriter$$patchAggregateFunctionChildren$1(DistinctAggregationRewriter.scala:148)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:192)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:190)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
  ...
  Cause: java.lang.reflect.InvocationTargetException:
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:368)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:367)
  at 
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:365)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:357)
  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:48)
  ...
  Cause: org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second 
argument must be a constant, but double was passed instead.
  at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:147)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector$lzycompute(hiveUDFs.scala:598)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector(hiveUDFs.scala:596)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector$lzycompute(hiveUDFs.scala:606)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector(hiveUDFs.scala:606)
  at org.apache.spark.sql.hive.HiveUDAFFunction.(hiveUDFs.scala:654)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  ...
{code}


  was:
For example, when using percentile_approx and count distinct together, it 
raises an error complaining the argument is not constant. We have a test case 
to reproduce. Could you help look into a fix of this? This was working in 
previous version (Spark 1.4 + Hive 0.13). Thanks!

{{---

[jira] [Updated] (SPARK-14171) UDAF aggregates argument object inspector not parsed correctly

2016-03-25 Thread Jianfeng Hu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfeng Hu updated SPARK-14171:

Description: 
For example, when using percentile_approx and count distinct together, it 
raises an error complaining the argument is not constant. We have a test case 
to reproduce. Could you help look into a fix of this? This was working in 
previous version (Spark 1.4 + Hive 0.13). Thanks!

{{--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -148,6 +148,9 @@ class HiveUDFSuite extends QueryTest with TestHiveSingleton 
with SQLTestUtils {

 checkAnswer(sql("SELECT percentile_approx(100.0, array(0.9, 0.9)) FROM src 
LIMIT 1"),
   sql("SELECT array(100, 100) FROM src LIMIT 1").collect().toSeq)
+
+checkAnswer(sql("SELECT percentile_approx(key, 0.9), count(distinct 
key) FROM src LIMIT 1"),
+  sql("SELECT max(key), 1 FROM src LIMIT 1").collect().toSeq)
}

   test("UDFIntegerToString") {
}}


When running the test suite, we can see this error:

{{
- Generic UDAF aggregates *** FAILED ***
  org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, 
tree: 
hiveudaffunction(HiveFunctionWrapper(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox,org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox@6e1dc6a7),key#51176,0.9,false,0,0)
  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)
  at org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:357)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode.withNewChildren(TreeNode.scala:238)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter.org$apache$spark$sql$catalyst$analysis$DistinctAggregationRewriter$$patchAggregateFunctionChildren$1(DistinctAggregationRewriter.scala:148)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:192)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:190)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
  ...
  Cause: java.lang.reflect.InvocationTargetException:
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:368)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:367)
  at 
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:365)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:357)
  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:48)
  ...
  Cause: org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second 
argument must be a constant, but double was passed instead.
  at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:147)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector$lzycompute(hiveUDFs.scala:598)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector(hiveUDFs.scala:596)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector$lzycompute(hiveUDFs.scala:606)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector(hiveUDFs.scala:606)
  at org.apache.spark.sql.hive.HiveUDAFFunction.(hiveUDFs.scala:654)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  ...
}}


  was:
For example, when using percentile_approx and count distinct together, it 
raises an error complaining the argument is not constant. We have a test case 
to reproduce. Could you help look into a fix of this? This was working in 
previous version (Spark 1.4 + Hive 0.13). Thanks!

```---

[jira] [Created] (SPARK-14171) UDAF aggregates argument object inspector not parsed correctly

2016-03-25 Thread Jianfeng Hu (JIRA) 
Jianfeng Hu created SPARK-14171:
---

 Summary: UDAF aggregates argument object inspector not parsed 
correctly
 Key: SPARK-14171
 URL: https://issues.apache.org/jira/browse/SPARK-14171
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.6.1
Reporter: Jianfeng Hu
Priority: Blocker


For example, when using percentile_approx and count distinct together, it 
raises an error complaining the argument is not constant. We have a test case 
to reproduce. Could you help look into a fix of this? This was working in 
previous version (Spark 1.4 + Hive 0.13). Thanks!

```--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
@@ -148,6 +148,9 @@ class HiveUDFSuite extends QueryTest with TestHiveSingleton 
with SQLTestUtils {

 checkAnswer(sql("SELECT percentile_approx(100.0, array(0.9, 0.9)) FROM src 
LIMIT 1"),
   sql("SELECT array(100, 100) FROM src LIMIT 1").collect().toSeq)
+
+checkAnswer(sql("SELECT percentile_approx(key, 0.9), count(distinct 
key) FROM src LIMIT 1"),
+  sql("SELECT max(key), 1 FROM src LIMIT 1").collect().toSeq)
}

   test("UDFIntegerToString") {```


When running the test suite, we can see this error:

```
- Generic UDAF aggregates *** FAILED ***
  org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, 
tree: 
hiveudaffunction(HiveFunctionWrapper(org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox,org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox@6e1dc6a7),key#51176,0.9,false,0,0)
  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:49)
  at org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:357)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode.withNewChildren(TreeNode.scala:238)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter.org$apache$spark$sql$catalyst$analysis$DistinctAggregationRewriter$$patchAggregateFunctionChildren$1(DistinctAggregationRewriter.scala:148)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:192)
  at 
org.apache.spark.sql.catalyst.analysis.DistinctAggregationRewriter$$anonfun$15.apply(DistinctAggregationRewriter.scala:190)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
  ...
  Cause: java.lang.reflect.InvocationTargetException:
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:368)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1$$anonfun$apply$12.apply(TreeNode.scala:367)
  at 
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:69)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:365)
  at 
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$makeCopy$1.apply(TreeNode.scala:357)
  at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:48)
  ...
  Cause: org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second 
argument must be a constant, but double was passed instead.
  at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:147)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector$lzycompute(hiveUDFs.scala:598)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.functionAndInspector(hiveUDFs.scala:596)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector$lzycompute(hiveUDFs.scala:606)
  at 
org.apache.spark.sql.hive.HiveUDAFFunction.returnInspector(hiveUDFs.scala:606)
  at org.apache.spark.sql.hive.HiveUDAFFunction.(hiveUDFs.scala:654)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  ...
```




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (SPARK-13984) Schema verification always fail when using remote Hive metastore

2016-03-19 Thread Jianfeng Hu (JIRA) 
Jianfeng Hu created SPARK-13984:
---

 Summary: Schema verification always fail when using remote Hive 
metastore
 Key: SPARK-13984
 URL: https://issues.apache.org/jira/browse/SPARK-13984
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.6.1
Reporter: Jianfeng Hu


Launch a Hive metastore Thrift server, and then in hive-site.xml:
- set hive.metastore.uris
- set hive.metastore.schema.verification to true

Run spark-shell, it will fail with:
java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: MetaException(message:Version information not found in metastore.

It might be using the wrong metastore (could possibly the local derby 
metastore) when doing the verification? Not exactly sure on this but maybe 
could someone more familiar with the HiveContext code help investigating?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-13984) Schema verification always fail when using remote Hive metastore

2016-03-18 Thread Jianfeng Hu (JIRA) 

 [ 
https://issues.apache.org/jira/browse/SPARK-13984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jianfeng Hu updated SPARK-13984:

Description: 
Launch a Hive metastore Thrift server, and then in hive-site.xml:
- set hive.metastore.uris
- set hive.metastore.schema.verification to true

Run spark-shell, it will fail with:
java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: MetaException(message:Version information not found in metastore.

It might be using the wrong metastore (could be possibly the local derby 
metastore) when doing the verification? Not exactly sure on this but maybe 
could someone more familiar with the HiveContext code help investigating?


  was:
Launch a Hive metastore Thrift server, and then in hive-site.xml:
- set hive.metastore.uris
- set hive.metastore.schema.verification to true

Run spark-shell, it will fail with:
java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
Caused by: MetaException(message:Version information not found in metastore.

It might be using the wrong metastore (could possibly the local derby 
metastore) when doing the verification? Not exactly sure on this but maybe 
could someone more familiar with the HiveContext code help investigating?



> Schema verification always fail when using remote Hive metastore
> 
>
> Key: SPARK-13984
> URL: https://issues.apache.org/jira/browse/SPARK-13984
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 1.6.1
>Reporter: Jianfeng Hu
>
> Launch a Hive metastore Thrift server, and then in hive-site.xml:
> - set hive.metastore.uris
> - set hive.metastore.schema.verification to true
> Run spark-shell, it will fail with:
> java.lang.RuntimeException: Unable to instantiate 
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> Caused by: MetaException(message:Version information not found in metastore.
> It might be using the wrong metastore (could be possibly the local derby 
> metastore) when doing the verification? Not exactly sure on this but maybe 
> could someone more familiar with the HiveContext code help investigating?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org