Did anyone use STS of Spark 2.0 on production? For me I still waiting for the compatible in parquet file created by Spark 1.6.1
> On Sep 6, 2016, at 2:46 PM, Campagnola, Francesco > <francesco.campagn...@anritsu.com> wrote: > > I mean I have installed Spark 2.0 in the same environment where Spark 1.6 > thrift server was running, > then stopped the Spark 1.6 thrift server and started the Spark 2.0 one. > > If I’m not mistaken, Spark 2.0 should be still compatible with Hive 1.2.1 and > no upgrade procedures are required. > The spark-defaults.conf file has not been changed. > > The following commands issued to the Spark 2.0 thrift server work: > create database test; > use test; > create table tb_1 (id int); > insert into table tb_1 select t.id from (select 1 as id) t; > > While all of these commands return the same error: > show databases; > show tables; > show partitions tb_1; > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 62.0 failed 10 times, most recent failure: Lost task 0.9 in > stage 62.0 (TID 540, vertica204): java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast > to org.apache.spark.sql.catalyst.expressions.UnsafeRow > > > > > From: Jeff Zhang [mailto:zjf...@gmail.com <mailto:zjf...@gmail.com>] > Sent: martedì 6 settembre 2016 02:50 > To: Campagnola, Francesco <francesco.campagn...@anritsu.com > <mailto:francesco.campagn...@anritsu.com>> > Cc: user@spark.apache.org <mailto:user@spark.apache.org> > Subject: Re: Spark 2.0.0 Thrift Server problem with Hive metastore > > How do you upgrade to spark 2.0 ? > > On Mon, Sep 5, 2016 at 11:25 PM, Campagnola, Francesco > <francesco.campagn...@anritsu.com <mailto:francesco.campagn...@anritsu.com>> > wrote: > Hi, > > in an already working Spark - Hive environment with Spark 1.6 and Hive 1.2.1, > with Hive metastore configured on Postgres DB, I have upgraded Spark to the > 2.0.0. > > I have started the thrift server on YARN, then tried to execute from the > beeline cli or a jdbc client the following command: > SHOW DATABASES; > It always gives this error on Spark server side: > > spark@spark-test[spark] /home/spark> beeline -u > jdbc:hive2://$(hostname):10000 -n spark > > Connecting to jdbc:hive2://spark-test:10000 > 16/09/05 17:41:43 INFO jdbc.Utils: Supplied authorities: spark-test:10000 > 16/09/05 17:41:43 INFO jdbc.Utils: Resolved authority: spark-test:10000 > 16/09/05 17:41:43 INFO jdbc.HiveConnection: Will try to open client transport > with JDBC Uri: jdbc:hive2:// spark-test:10000 > Connected to: Spark SQL (version 2.0.0) > Driver: Hive JDBC (version 1.2.1.spark2) > Transaction isolation: TRANSACTION_REPEATABLE_READ > Beeline version 1.2.1.spark2 by Apache Hive > > 0: jdbc:hive2:// spark-test:10000> show databases; > java.lang.IllegalStateException: Can't overwrite cause with > java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast > to org.apache.spark.sql.catalyst.expressions.UnsafeRow > at java.lang.Throwable.initCause(Throwable.java:457) > at > org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236) > at > org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:236) > at > org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:197) > at > org.apache.hive.service.cli.HiveSQLException.<init>(HiveSQLException.java:108) > at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256) > at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242) > at > org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365) > at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:42) > at org.apache.hive.beeline.BeeLine.print(BeeLine.java:1794) > at org.apache.hive.beeline.Commands.execute(Commands.java:860) > at org.apache.hive.beeline.Commands.sql(Commands.java:713) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:973) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:813) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:771) > at > org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:484) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:467) > Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: > Task 0 in stage 3.0 failed 10 times, most recent failure: Lost task 0.9 in > stage 3.0 (TID 12, vertica204): java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericInternalRow cannot be cast > to org.apache.spark.sql.catalyst.expressions.UnsafeRow > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:247) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$4.apply(SparkPlan.scala:240) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784) > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:784) > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319) > at org.apache.spark.rdd.RDD.iterator(RDD.scala:283) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) > at org.apache.spark.scheduler.Task.run(Task.scala:85) > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Driver stacktrace: > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at > org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:244) > at > org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:210) > ... 15 more > Error: Error retrieving next row (state=,code=0) > > The same command works when using Spark 1.6, is it a possible issue? > > Thanks! > > > > -- > Best Regards > > Jeff Zhang
