How do you upgrade to spark 2.0 ? On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco < francesco.campagn...@anritsu.com> wrote:
> Hi, > > > > in an already working Spark - Hive environment with Spark 1.6 and Hive > 1.2.1, with Hive metastore configured on Postgres DB, I have upgraded Spark > to the 2.0.0. > > > > I have started the thrift server on YARN, then tried to execute from the > beeline cli or a jdbc client the following command: > > SHOW DATABASES; > > It always gives this error on Spark server side: > > > > spark@spark-test[spark] /home/spark> beeline -u > jdbc:hive2://$(hostname):10000 -n spark > > > > Connecting to jdbc:hive2://spark-test:10000 > > 16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:10000 > > 16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:10000 > > 16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client > transport with JDBC Uri: jdbc:hive2:// spark-test:10000 > > Connected to: Spark SQL (version 2.0.0) > > Driver: Hive JDBC (version 1.2.1.spark2) > > Transaction isolation: TRANSACTION_REPEATABLE_READ > > Beeline version 1.2.1.spark2 by Apache Hive > > > > 0: jdbc:hive2:// spark-test:10000> show databases; > > java.lang.IllegalStateException: Can't overwrite cause with > java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericInternalRow > cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow > > at java.lang.Throwable.initCause(Throwable.java:457) > > at org.apache.hive.service.cli.HiveSQLException.toStackTrace( > HiveSQLException.java:236) > > at org.apache.hive.service.cli.HiveSQLException.toStackTrace( > HiveSQLException.java:236) > > at org.apache.hive.service.cli.HiveSQLException.toCause( > HiveSQLException.java:197) > > at org.apache.hive.service.cli.HiveSQLException.<init>( > HiveSQLException.java:108) > > at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256) > > at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils. > java:242) > > at org.apache.hive.jdbc.HiveQueryResultSet.next( > HiveQueryResultSet.java:365) > > at org.apache.hive.beeline.BufferedRows.<init>( > BufferedRows.java:42) > > at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794) > > at org.apache.hive.beeline.Commands.execute(Commands.java:860) > > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973) > > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813) > > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771) > > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection( > BeeLine.java:484) > > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467) > > Caused by: org.apache.spark.SparkException: Job aborted due to stage > failure: Task 0 in stage 3.0 failed 10 times, most recent failure: Lost > task 0.9 in stage 3.0 (TID 12, vertica204): java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be > cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow > > at org.apache.spark.sql.execution.SparkPlan$$anonfun$ > 4.apply(SparkPlan.scala:247) > > at org.apache.spark.sql.execution.SparkPlan$$anonfun$ > 4.apply(SparkPlan.scala:240) > > at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$ > 1$$anonfun$apply$24.apply(RDD.scala:784) > > at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$ > 1$$anonfun$apply$24.apply(RDD.scala:784) > > at org.apache.spark.rdd.MapPartitionsRDD.compute( > MapPartitionsRDD.scala:38) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask. > scala:70) > > at org.apache.spark.scheduler.Task.run(Task.scala:85) > > at org.apache.spark.executor.Executor$TaskRunner.run( > Executor.scala:274) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > > > Driver stacktrace: > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:62) > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > > at org.apache.hive.service.cli.HiveSQLException.newInstance( > HiveSQLException.java:244) > > at org.apache.hive.service.cli.HiveSQLException.toStackTrace( > HiveSQLException.java:210) > > ... 15 more > > Error: Error retrieving next row (state=,code=0) > > > > The same command works when using Spark 1.6, is it a possible issue? > > > > Thanks! > -- Best Regards Jeff Zhang
