[
https://issues.apache.org/jira/browse/HIVE-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523378#comment-14523378
]
Alex McLintock commented on HIVE-9865:
--------------------------------------
I am seeing a very similar error on HDP2.2.0 on RedHat6
I can perhaps help with diagnostics if Paul has gone quiet.
> server.TThreadPoolServer - Error occurred during processing of message
> ----------------------------------------------------------------------
>
> Key: HIVE-9865
> URL: https://issues.apache.org/jira/browse/HIVE-9865
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 0.14.0
> Environment: Ubuntu 12.04 x86_64
> Reporter: Paul
> Attachments: hive-env.sh, hive-site.xml
>
>
> 2015-02-22 19:49:31,890 INFO [HiveServer2-Handler-Pool: Thread-42]:
> session.SessionState (SessionState.java:createPath(558)) - Created HDFS
> directory: /tmp/hive/hive/8dda6291-828e-4bbd-a5a3-923617352699/_tmp_space.db
> 2015-02-22 19:49:12,614 INFO [HiveServer2-Handler-Pool: Thread-30]:
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
> 2015-02-22 19:48:53,321 ERROR [HiveServer2-Handler-Pool: Thread-9222]:
> server.TThreadPoolServer (TThreadPoolServer.java:run(215)) - Error occurred
> during processing of message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:190)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.thrift.transport.TTransportException
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
> at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> ... 4 more
> 2015-02-22 19:50:21,784 INFO
> [main-SendThread(ip-172-25-1-227.us-west-2.compute.internal:2181)]:
> zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket
> connection established to
> ip-172-25-1-227.us-west-2.compute.internal/172.25.1.227:2181, initiating
> session
> 2015-02-22 19:50:21,784 INFO [HiveServer2-Handler-Pool: Thread-9220]:
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(232)) - Client
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V6
> 2015-02-22 19:51:22,151 INFO [HiveServer2-Handler-Pool: Thread-34]:
> session.SessionState (SessionState.java:createPath(558)) - Created local
> directory: /tmp/e9c8b31d-a096-4af8-8327-a252c0c3db1e_resources
> 2015-02-22 19:52:17,425 ERROR [BoneCP-pool-watch-thread]:
> bonecp.CustomThreadFactory (CustomThreadFactory.java:uncaughtException(69)) -
> Uncaught Exception in thread BoneCP-pool-watch-thread
> java.lang.OutOfMemoryError: Java heap space
> at java.sql.DriverManager.getConnection(DriverManager.java:570)
> at java.sql.DriverManager.getConnection(DriverManager.java:187)
> at
> com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
> at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269)
> at
> com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:242)
> at
> com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
> at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-02-22 19:51:22,151 INFO [HiveServer2-Handler-Pool: Thread-42]:
> session.SessionState (SessionState.java:start(460)) - No Tez session required
> at this point. hive.execution.engine=mr.
> 2015-02-22 19:51:10,931 ERROR [CuratorFramework-0]: curator.ConnectionState
> (ConnectionState.java:checkTimeouts(201)) - Connection timed out for
> connection string
> (ip-172-25-1-227.us-west-2.compute.internal:2181,ip-172-25-1-230.us-west-2.compute.internal:2181,ip-172-25-1-231.us-west-2.compute.internal:2181)
> and timeout (15000) / elapsed (15114)
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
> org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
> at
> org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
> at
> org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-02-22 19:54:17,609 INFO [HiveServer2-Handler-Pool: Thread-42]:
> session.HiveSessionImpl (HiveSessionImpl.java:setOperationLogSessionDir(218))
> - Operation log session directory is created:
> /home/hive/${system:java.io.tmpdir}/${system:user.name}/operation_logs/8dda6291-828e-4bbd-a5a3-923617352699
> 2015-02-22 19:53:55,164 ERROR [BoneCP-pool-watch-thread]: bonecp.BoneCP
> (BoneCP.java:obtainInternalConnection(292)) - Failed to acquire connection to
> jdbc:mysql://ip-172-25-1-231.us-west-2.compute.internal/hive?createDatabaseIfNotExist=true.
> Sleeping for 7000 ms. Attempts left: 5
> ...
> 2015-02-22 19:52:10,308 ERROR [HiveServer2-Handler-Pool: Thread-30]:
> thrift.ProcessFunction (ProcessFunction.java:process(41)) - Internal error
> processing OpenSession
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.createEntry(HashMap.java:897)
> at java.util.HashMap.putForCreate(HashMap.java:550)
> at java.util.HashMap.putAllForCreate(HashMap.java:555)
> at java.util.HashMap.<init>(HashMap.java:298)
> at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:703)
> at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2372)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.<init>(HiveSessionImpl.java:116)
> at
> org.apache.hive.service.cli.session.HiveSessionImplwithUGI.<init>(HiveSessionImplwithUGI.java:47)
> at
> org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:260)
> at
> org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:175)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:322)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:235)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-02-22 19:51:33,478 INFO
> [main-SendThread(ip-172-25-1-227.us-west-2.compute.internal:2181)]:
> zookeeper.ClientCnxn (ClientCnxn.java:run(1096)) - Client session timed out,
> have not heard from server in 51503ms for sessionid 0x34ba7aa216d0006,
> closing socket connection and attempting reconnect
> 2015-02-22 19:57:26,796 INFO [Thread-6]: server.HiveServer2
> (HiveServer2.java:stop(299)) - Shutting down HiveServer2
> ...
> 2015-02-22 19:51:33,478 INFO
> [main-SendThread(ip-172-25-1-227.us-west-2.compute.internal:2181)]:
> zookeeper.ClientCnxn (ClientCnxn.java:run(1096)) - Client session timed out,
> have not heard from server in 51503ms for sessionid 0x34ba7aa216d0006,
> closing socket connection and attempting reconnect
> 2015-02-22 19:57:26,796 INFO [Thread-6]: server.HiveServer2
> (HiveServer2.java:stop(299)) - Shutting down HiveServer2
> 2015-02-22 19:56:21,460 INFO [HiveServer2-Handler-Pool: Thread-34]:
> session.SessionState (SessionState.java:createPath(558)) - Created HDFS
> directory: /tmp/hive/hive/e9c8b31d-a096-4af8-8327-a252c0c3db1e
> 2015-02-22 19:55:27,236 WARN [CuratorFramework-0]: curator.ConnectionState
> (ConnectionState.java:checkTimeouts(192)) - Connection attempt unsuccessful
> after 579400 (greater than max timeout of 60000). Resetting connection and
> trying again with a new connection.
> 2015-02-22 19:55:26,132 INFO [HiveServer2-Handler-Pool: Thread-38]:
> session.SessionState (SessionState.java:createPath(558)) - Created local
> directory: /tmp/5a28f57a-f4e6-47f1-8c3f-34d86833df7f_resources
> 2015-02-22 19:58:06,068 INFO
> [main-SendThread(ip-172-25-1-231.us-west-2.compute.internal:2181)]:
> zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket
> connection to server
> ip-172-25-1-231.us-west-2.compute.internal/172.25.1.231:2181. Will not
> attempt to authenticate using SASL (unknown error)
> 2015-02-22 19:57:58,802 INFO [Thread-6]: thrift.ThriftCLIService
> (ThriftCLIService.java:stop(137)) - Thrift server has stopped
> 2015-02-22 19:58:36,333 INFO [Thread-3]: server.HiveServer2
> (HiveStringUtils.java:run(679)) - SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down HiveServer2 at
> ip-172-25-1-231.us-west-2.compute.internal/172.25.1.231
> ************************************************************/ 2015-02-22
> 19:58:38,574 INFO
> [main-SendThread(ip-172-25-1-231.us-west-2.compute.internal:2181)]:
> zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket
> connection established to
> ip-172-25-1-231.us-west-2.compute.internal/172.25.1.231:2181, initiating
> session 2015-02-22 19:58:36,334 INFO [Thread-6]: service.AbstractService
> (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped.
> 2015-02-22 19:58:50,858 INFO [Thread-6]: service.AbstractService
> (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
> 2015-02-22 19:58:50,858 INFO [Thread-6]: service.AbstractService
> (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
> 2015-02-22 19:59:02,200 INFO [Thread-6]: service.AbstractService
> (AbstractService.java:stop(125)) - Service:CLIService is stopped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
