I mean I have installed Spark 2.0 in the same environment where Spark 1.6 thrift server was running, then stopped the Spark 1.6 thrift server and started the Spark 2.0 one.
If I’m not mistaken, Spark 2.0 should be still compatible with Hive 1.2.1 and no upgrade procedures are required. The spark-defaults.conf file has not been changed. The following commands issued to the Spark 2.0 thrift server work: create database test; use test; create table tb_1 (id int); insert into table tb_1 select t.id from (select 1 as id) t; While all of these commands return the same error: show databases; show tables; show partitions tb_1; Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 62.0 failed 10 times, most recent failure: Lost task 0.9 in stage 62.0 (TID 540, vertica204): java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow From: Jeff Zhang [mailto:zjf...@gmail.com] Sent: martedì 6 settembre 2016 02:50 To: Campagnola, Francesco <francesco.campagn...@anritsu.com> Cc: user@spark.apache.org Subject: Re: Spark 2.0.0 Thrift Server problem with Hive metastore How do you upgrade to spark 2.0 ? On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco <francesco.campagn...@anritsu.com<mailto:francesco.campagn...@anritsu.com>> wrote: Hi, in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1, with Hive metastore configured on Postgres DB, I have upgraded Spark to the 2.0.0. I have started the thrift server on YARN, then tried to execute from the beeline cli or a jdbc client the following command: SHOW DATABASES; It always gives this error on Spark server side: spark@spark-test[spark] /home/spark> beeline -u jdbc:hive2://$(hostname):10000 -n spark Connecting to jdbc:hive2://spark-test:10000 16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:10000 16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:10000 16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2:// spark-test:10000 Connected to: Spark SQL (version 2.0.0) Driver: Hive JDBC (version 1.2.1.spark2) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1.spark2 by Apache Hive 0: jdbc:hive2:// spark-test:10000> show databases; java.lang.IllegalStateException: Can't overwrite cause with java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow at java.lang.Throwable.initCause(Throwable.java:457) at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236) at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236) at org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197) at org.apache.hive.service.cli.HiveSQLException.<init>(HiveSQLException.java:108) at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256) at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242) at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365) at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:42) at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794) at org.apache.hive.beeline.Commands.execute(Commands.java:860) at org.apache.hive.beeline.Commands.sql(Commands.java:713) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467) Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 10 times, most recent failure: Lost task 0.9 in stage 3.0 (TID 12, vertica204): java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:247) at org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) at org.apache.spark.scheduler.Task.run(Task.scala:85) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Driver stacktrace: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:422) at org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:244) at org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:210) ... 15 more Error: Error retrieving next row (state=,code=0) The same command works when using Spark 1.6, is it a possible issue? Thanks! -- Best Regards Jeff Zhang
