Hi Roc,

Spark SQL 1.2.0 can only work with Hive 0.12.0 or Hive 0.13.1 (controlled by compilation flags), versions prior 1.2.0 only works with Hive 0.12.0. So Hive 0.15.0-SNAPSHOT is not an option.

Would like to add that this is due to backwards compatibility issue of Hive metastore, AFAIK there's no popular system works with multiple versions of Hive metastore simultaneously.

Cheng

On 12/24/14 6:31 PM, Roc Chu wrote:
this is my problem. I use mysql to store hive meta data. and i can get what i want when I exec show tables in hive shell. but in the same machine. I use spark-sql to execute same command (show tables), I got errors.
I look at the log of hive metastore find this errors

2014-12-24 05:04:59,874 ERROR [pool-3-thread-2]: server.TThreadPoolServer (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Expected protocol id ffffff82 but got ffffff80 at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:503) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:75) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2014-12-24 05:05:01,883 ERROR [pool-3-thread-3]: server.TThreadPoolServer (TThreadPoolServer.java:run(294)) - Thrift error occurred during processing of message. org.apache.thrift.protocol.TProtocolException: Expected protocol id ffffff82 but got ffffff80 at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:503) at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:75) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

I compiled e spark from soucecode of spark-1.1.1
and my hive was compiled from apache-hive-0.15.0

I put the same hive-site.xml both hive conf and apark conf.

waiting for help
thanks




---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to