Hari Sekhon created HIVE-7594:
---------------------------------
Summary: Hive JDBC client: "out of sequence response" on large
long running query
Key: HIVE-7594
URL: https://issues.apache.org/jira/browse/HIVE-7594
Project: Hive
Issue Type: Improvement
Components: Clients, HiveServer2
Affects Versions: 0.13.0
Environment: HDP2.1
Reporter: Hari Sekhon
When executing a long running query in a JDBC client (Squirrel) to HiveServer2
after several minutes I get this error in the client:
{code}
Error: org.apache.thrift.TApplicationException: ExecuteStatement failed: out of
sequence response
SQLState: 08S01
ErrorCode: 0
{code}
I've seen this before in, and I just did a quick search of other Hive Jiras and
saw the Metastore exception jira, which I have in some older logs but not
corresponding / recent in these recent instances
The query to reproduce is "select count(*) from myTable" where myTable is a 1TB
table of 620 million rows. This happens in both MR and Tez execution engines
running on Yarn.
Here are all the jars I've added to the classpath (taken from Hortonworks doc
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-latest/bk_dataintegration/content/ch_using-hive-2.html,
plus added hadoop-common, hive-exec and slf4j-api to solve class not found
issues on top of that):
commons-codec-1.4.jar
commons-logging-1.1.3.jar
hadoop-common-2.4.0.2.1.3.0-563.jar
hive-exec-0.13.0.2.1.3.0-563.jar
hive-jdbc-0.13.0.2.1.3.0-563.jar
hive-service-0.13.0.2.1.3.0-563.jar
httpclient-4.2.5.jar
httpcore-4.2.5.jar
libthrift-0.9.0.jar
slf4j-api-1.7.5.jar
I am seeing errors like this in the hiveserver2.log:
{code}
2014-08-01 15:04:31,358 ERROR [pool-5-thread-3]: server.TThreadPoolServer
(TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
...
2014-08-01 15:06:31,520 ERROR [pool-5-thread-3]: server.TThreadPoolServer
(TThreadPoolServer.java:run(215)) - Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:189)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
... 4 more
...
2014-08-01 15:05:41,096 ERROR [pool-4-thread-22]: operation.Operation
(SQLOperation.java:run(202)) - Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Illegal Operation state transition
at
org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:94)
at
org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:99)
at
org.apache.hive.service.cli.operation.Operation.setState(Operation.java:109)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:158)
at
org.apache.hive.service.cli.operation.SQLOperation.access$000(SQLOperation.java:68)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:199)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:500)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
Regards,
Hari Sekhon
http://www.linkedin.com/in/harisekhon
--
This message was sent by Atlassian JIRA
(v6.2#6252)
