[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource
{code:java}
0x0003c0acca40
{code}
{code:java}
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource :
{code:java}
0x0003c0acca40
{code}
,and the code is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocat
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource {code:java}
0x0003c0acca40
{code}
{code:java}
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource :
{code:java}
0x0003c0acca40
{code}
,and the code is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocati
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource
*0x0003c0acca40*
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource *0x0003c0acca40*: ,and the jstack
is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHan
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource
*0x0003c0acca40*
{code:java}
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource *0x0003c0acca40*: ,and the jstack
is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryI
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource
*0x0003c0acca40*
{code:java}
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource *0x0003c0acca40*: ,and the jstack
is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryI
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Attachment: HiveServer2_jstack.txt
> HiveServer2 is blocked at
> org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
>
>
> Key: HIVE-20401
> URL: https://issues.apache.org/jira/browse/HIVE-20401
> Project: Hive
> Issue Type: Bug
> Components: Beeline, HiveServer2
>Affects Versions: 1.2.2
>Reporter: zhangbutao
>Priority: Major
> Attachments: HiveServer2_jstack.txt
>
>
> HiveServer2 process often gets stuck , and jstack shows about one hundred
> thread is blocked at the following code,waiting the resource
> *0x0003c0acca40*
> {code:java}
> "HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
> tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry
> [0x7fd6d6eee000]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
> - waiting to lock <0x0003c0acca40> (a java.lang.Object)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
> at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> {code}
> We find one thread locked the resource *0x0003c0acca40*: ,and the jstack
> is as follow:
> {code:java}
> "HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
> tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> - locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1441)
> at org.apache.hadoop.ipc.Client.call(Client.java:1351)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
>
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource
*0x0003c0acca40*
{code:java}
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource *0x0003c0acca40*: ,and the jstack
is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryI
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource
*0x0003c0acca40*
{code:java}
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource *0x0003c0acca40*: ,and the jstack
is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryI
[jira] [Updated] (HIVE-20401) HiveServer2 is blocked at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
[
https://issues.apache.org/jira/browse/HIVE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangbutao updated HIVE-20401:
--
Description:
HiveServer2 process often gets stuck , and jstack shows about one hundred
thread is blocked at the following code,waiting the resource
*0x0003c0acca40*
{code:java}
"HiveServer2-Handler-Pool: Thread-935985" #935985 prio=5 os_prio=0
tid=0x7fd71470c000 nid=0x3e1f waiting for monitor entry [0x7fd6d6eee000]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1144)
- waiting to lock <0x0003c0acca40> (a java.lang.Object)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1139)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:110)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:181)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:257)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:423)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:410)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy22.executeStatementAsync(Unknown Source)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:275)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:492)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:698)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{code}
We find one thread locked the resource *0x0003c0acca40*: ,and the jstack
is as follow:
{code:java}
"HiveServer2-Handler-Pool: Thread-809399" #809399 prio=5 os_prio=0
tid=0x7fd71433e000 nid=0x37c0 in Object.wait() [0x7fd6d04b8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(AsyncGet.java:59)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
- locked <0x0003d9ab5d80> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy18.getEZForPath(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getEZForPath(ClientNamenodeProtocolTranslatorPB.java:1413)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryI
