gongzh021 opened a new issue #1624: URL: https://github.com/apache/incubator-kyuubi/issues/1624
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug When kyuubi throws the following error after running for a period of time. The client cannot connect at this time. trying to kill spark process can't solve this problem. kyuubi must be restarted to make it run normally. ### Affects Version(s) 2.2.0 ### Kyuubi Server Log Output ```logtalk 2021-12-24 14:59:04.279 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:B4 F9 8B 10 D5 7B 4E 7D A6 CA 2E 4A 10 34 F6 0D, secret:CD 9D 5B F2 41 B8 47 BC 9B 98 63 31 F4 F0 09 16), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side 2021-12-24 14:59:04.394 INFO log.OperationLog: Creating operation log file /usr/local/software/kyuubi-1.4.0-bin-spark-3.0-hadoop2.7/server_operation_logs/14c0013b-6293-4e64-ab42-29a56a52d907/a8a8108d-8983-449a-8264-a0d64de1822c 2021-12-24 14:59:04.394 INFO operation.ExecuteStatement: Processing wuchunjun's query[a8a8108d-8983-449a-8264-a0d64de1822c]: INITIALIZED_STATE -> PENDING_STATE, statement: SELECT current_database() 2021-12-24 14:59:04.396 INFO operation.ExecuteStatement: Processing wuchunjun's query[a8a8108d-8983-449a-8264-a0d64de1822c]: PENDING_STATE -> RUNNING_STATE, statement: SELECT current_database() 2021-12-24 14:59:04.416 INFO operation.ExecuteStatement: Query[a8a8108d-8983-449a-8264-a0d64de1822c] in FINISHED_STATE 2021-12-24 14:59:04.416 INFO operation.ExecuteStatement: Processing wuchunjun's query[a8a8108d-8983-449a-8264-a0d64de1822c]: RUNNING_STATE -> FINISHED_STATE, statement: SELECT current_database(), time taken: 0.02 seconds 2021-12-24 14:59:04.456 INFO operation.ExecuteStatement: Processing wuchunjun's query[a8a8108d-8983-449a-8264-a0d64de1822c]: FINISHED_STATE -> CLOSED_STATE, statement: SELECT current_database() 2021-12-24 14:59:04.456 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:59 3A 58 96 91 BA 48 F0 A5 C2 EB 6B 96 F8 A0 BF, secret:22 E5 49 B6 95 DA 42 6F 9B 13 EC 5E F1 43 9F 40), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side 2021-12-24 14:59:06.722 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:11.722 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:16.723 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:21.724 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:26.724 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:31.725 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:36.725 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:41.726 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:46.727 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:47.429 INFO server.KyuubiThriftBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 2021-12-24 14:59:47.431 INFO operation.LaunchEngine: Processing gongzh's query[c9b0cb47-c082-4ce0-a8c5-2b6bfd7d93af]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE 2021-12-24 14:59:47.431 INFO operation.LaunchEngine: Processing gongzh's query[c9b0cb47-c082-4ce0-a8c5-2b6bfd7d93af]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE 2021-12-24 14:59:47.431 INFO imps.CuratorFrameworkImpl: Starting 2021-12-24 14:59:47.431 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-nn001:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@73010f35 2021-12-24 14:59:47.432 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop-nn001/172.16.9.141:2181. Will not attempt to authenticate using SASL (unknown error) 2021-12-24 14:59:47.432 INFO zookeeper.ClientCnxn: Socket connection established to hadoop-nn001/172.16.9.141:2181, initiating session 2021-12-24 14:59:47.432 INFO server.NIOServerCnxnFactory: Accepted socket connection from /172.16.9.141:57962 2021-12-24 14:59:47.432 INFO server.ZooKeeperServer: Client attempting to establish new session at /172.16.9.141:57962 2021-12-24 14:59:47.433 INFO server.ZooKeeperServer: Established session 0x103c7ce4df3007d with negotiated timeout 60000 for client /172.16.9.141:57962 2021-12-24 14:59:47.433 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop-nn001/172.16.9.141:2181, sessionid = 0x103c7ce4df3007d, negotiated timeout = 60000 2021-12-24 14:59:47.433 INFO state.ConnectionStateManager: State change: CONNECTED 2021-12-24 14:59:47.434 INFO client.ServiceDiscovery: Get service instance:hadoop-nn001:38309 and version:Some(1.4.0-incubating) under /kyuubi_GROUP_SPARK_SQL/GROUP_bigdata_admin/default 2021-12-24 14:59:47.435 INFO session.KyuubiSessionImpl: [gongzh:58.34.219.242] SessionHandle [421dfef3-59c4-43a4-9ed8-386a1849aa52] - Connected to engine [hadoop-nn001:38309] 2021-12-24 14:59:51.727 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 14:59:56.728 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:01.729 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:06.729 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:11.730 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:16.730 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:21.731 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:26.731 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:31.732 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:36.733 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:41.733 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:46.734 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:47.495 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting 2021-12-24 15:00:47.495 INFO server.PrepRequestProcessor: Processed session termination for sessionid: 0x103c7ce4df3007d 2021-12-24 15:00:47.496 INFO zookeeper.ZooKeeper: Session: 0x103c7ce4df3007d closed 2021-12-24 15:00:47.496 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x103c7ce4df3007d 2021-12-24 15:00:47.496 INFO server.NIOServerCnxn: Closed socket connection for client /172.16.9.141:57962 which had sessionid 0x103c7ce4df3007d 2021-12-24 15:00:47.496 INFO operation.LaunchEngine: Processing gongzh's query[c9b0cb47-c082-4ce0-a8c5-2b6bfd7d93af]: RUNNING_STATE -> ERROR_STATE, statement: LAUNCH_ENGINE, time taken: 60.065 seconds 2021-12-24 15:00:47.497 INFO operation.LaunchEngine: Processing gongzh's query[c9b0cb47-c082-4ce0-a8c5-2b6bfd7d93af]: ERROR_STATE -> CLOSED_STATE, statement: LAUNCH_ENGINE 2021-12-24 15:00:47.497 WARN session.KyuubiSessionManager: Error closing session for gongzh client ip: 58.34.219.242 org.apache.kyuubi.KyuubiSQLException: Error while cleaning up the engine resources at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:149) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:75) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftBinaryFrontendService.getSessionHandle(ThriftBinaryFrontendService.scala:199) at org.apache.kyuubi.server.KyuubiThriftBinaryFrontendService.OpenSession(KyuubiThriftBinaryFrontendService.scala:46) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.protocol.TProtocolException: Required field 'sessionHandle' is unset! Struct:TCloseSessionReq(sessionHandle:null) at org.apache.hive.service.rpc.thrift.TCloseSessionReq.validate(TCloseSessionReq.java:294) at org.apache.hive.service.rpc.thrift.TCLIService$CloseSession_args.validate(TCLIService.java:3875) at org.apache.hive.service.rpc.thrift.TCLIService$CloseSession_args$CloseSession_argsStandardScheme.write(TCLIService.java:3932) at org.apache.hive.service.rpc.thrift.TCLIService$CloseSession_args$CloseSession_argsStandardScheme.write(TCLIService.java:3901) at org.apache.hive.service.rpc.thrift.TCLIService$CloseSession_args.write(TCLIService.java:3852) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:71) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_CloseSession(TCLIService.java:185) at org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseSession(TCLIService.java:177) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$1(KyuubiSyncThriftClient.scala:73) at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:49) at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:73) at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:146) ... 13 more 2021-12-24 15:00:47.497 ERROR server.KyuubiThriftBinaryFrontendService: Error opening session: org.apache.kyuubi.KyuubiSQLException: Error opening session for gongzh client ip 58.34.219.242, due to org.apache.kyuubi.KyuubiSQLException: Error operating UNKNOWN_OPERATION: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376) at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168) at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$openSession$1(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:49) at org.apache.kyuubi.client.KyuubiSyncThriftClient.openSession(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:101) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:81) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 25 more at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:80) at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45) at org.apache.kyuubi.service.ThriftBinaryFrontendService.getSessionHandle(ThriftBinaryFrontendService.scala:199) at org.apache.kyuubi.server.KyuubiThriftBinaryFrontendService.OpenSession(KyuubiThriftBinaryFrontendService.scala:46) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: org.apache.kyuubi.KyuubiSQLException: Error operating UNKNOWN_OPERATION: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376) at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168) at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$openSession$1(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:49) at org.apache.kyuubi.client.KyuubiSyncThriftClient.openSession(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:101) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:81) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 25 more at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.kyuubi.operation.LaunchEngine.runInternal(LaunchEngine.scala:60) at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:130) at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:93) at org.apache.kyuubi.session.KyuubiSessionImpl.runOperation(KyuubiSessionImpl.scala:116) at org.apache.kyuubi.session.KyuubiSessionImpl.open(KyuubiSessionImpl.scala:83) at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:62) ... 12 more Caused by: org.apache.kyuubi.KyuubiSQLException: Error operating UNKNOWN_OPERATION: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376) at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168) at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$openSession$1(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:49) at org.apache.kyuubi.client.KyuubiSyncThriftClient.openSession(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:101) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:81) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 25 more at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:69) at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:49) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:53) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376) at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435) at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_OpenSession(TCLIService.java:168) at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:155) at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$openSession$1(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:49) at org.apache.kyuubi.client.KyuubiSyncThriftClient.openSession(KyuubiSyncThriftClient.scala:65) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:101) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:81) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:87) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51) ... 5 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ... 25 more 2021-12-24 15:00:47.738 INFO server.KyuubiThriftBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8 2021-12-24 15:00:47.739 INFO operation.LaunchEngine: Processing gongzh's query[736f05cd-49a2-4af4-ada2-f0a303026dc6]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE 2021-12-24 15:00:47.739 INFO operation.LaunchEngine: Processing gongzh's query[736f05cd-49a2-4af4-ada2-f0a303026dc6]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE 2021-12-24 15:00:47.739 INFO imps.CuratorFrameworkImpl: Starting 2021-12-24 15:00:47.739 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop-nn001:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@7f158a90 2021-12-24 15:00:47.740 INFO zookeeper.ClientCnxn: Opening socket connection to server hadoop-nn001/172.16.9.141:2181. Will not attempt to authenticate using SASL (unknown error) 2021-12-24 15:00:47.740 INFO zookeeper.ClientCnxn: Socket connection established to hadoop-nn001/172.16.9.141:2181, initiating session 2021-12-24 15:00:47.740 INFO server.NIOServerCnxnFactory: Accepted socket connection from /172.16.9.141:58384 2021-12-24 15:00:47.740 INFO server.ZooKeeperServer: Client attempting to establish new session at /172.16.9.141:58384 2021-12-24 15:00:47.741 INFO server.ZooKeeperServer: Established session 0x103c7ce4df3007e with negotiated timeout 60000 for client /172.16.9.141:58384 2021-12-24 15:00:47.741 INFO zookeeper.ClientCnxn: Session establishment complete on server hadoop-nn001/172.16.9.141:2181, sessionid = 0x103c7ce4df3007e, negotiated timeout = 60000 2021-12-24 15:00:47.741 INFO state.ConnectionStateManager: State change: CONNECTED 2021-12-24 15:00:47.742 INFO client.ServiceDiscovery: Get service instance:hadoop-nn001:38309 and version:Some(1.4.0-incubating) under /kyuubi_GROUP_SPARK_SQL/GROUP_bigdata_admin/default 2021-12-24 15:00:47.743 INFO session.KyuubiSessionImpl: [gongzh:58.34.219.242] SessionHandle [87ee6778-993a-4596-94bd-ba1d3b8a3dac] - Connected to engine [hadoop-nn001:38309] 2021-12-24 15:00:51.734 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:00:56.735 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:01:01.735 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:01:06.736 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:01:11.737 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE 2021-12-24 15:01:16.737 INFO operation.ExecuteStatement: Query[b2c48905-ce65-4a67-a759-ce3dc12054b1] in RUNNING_STATE ``` ### Kyuubi Engine Log Output ```logtalk 21/12/22 19:46:12 INFO kyuubi.SQLOperationListener: Query [aacace32-e0d4-4bd9-8ef8-d40a31c627a6]: Job 1 succeeded, 0 active jobs running 21/12/22 19:46:12 INFO operation.GetTypeInfo: Processing gongzh's query[843d94c8-7185-4609-9c5b-0379c529d342]: INITIALIZED_STATE -> RUNNING_STATE, statement: GET_TYPE_INFO 21/12/22 19:46:12 INFO operation.GetTypeInfo: Processing gongzh's query[843d94c8-7185-4609-9c5b-0379c529d342]: RUNNING_STATE -> FINISHED_STATE, statement: GET_TYPE_INFO, time taken: 0.01 seconds 21/12/22 19:46:13 INFO operation.GetCatalogs: Processing gongzh's query[01c68d41-acb4-4473-8618-6cecc296b159]: INITIALIZED_STATE -> RUNNING_STATE, statement: GET_CATALOGS 21/12/22 19:46:13 INFO operation.GetCatalogs: Processing gongzh's query[01c68d41-acb4-4473-8618-6cecc296b159]: RUNNING_STATE -> FINISHED_STATE, statement: GET_CATALOGS, time taken: 0.025 seconds 21/12/22 19:46:13 INFO spark.SparkThriftBinaryFrontendService: Received request of closing SessionHandle [21efc671-e6b7-405a-8698-77226a0746be] 21/12/22 19:46:13 INFO session.SparkSQLSessionManager: SessionHandle [21efc671-e6b7-405a-8698-77226a0746be] is closed, current opening sessions 0 21/12/22 19:46:13 INFO spark.SparkThriftBinaryFrontendService: Finished closing SessionHandle [21efc671-e6b7-405a-8698-77226a0746be] 21/12/22 19:46:13 ERROR server.TThreadPoolServer: Thrift error occurred during processing of message. org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374) at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451) at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433) at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:43) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ### Kyuubi Server Configurations _No response_ ### Kyuubi Engine Configurations _No response_ ### Additional context _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
