weijie.tong created DRILL-5749:
----------------------------------
Summary: Foreman and Netty threads occure deadlock
Key: DRILL-5749
URL: https://issues.apache.org/jira/browse/DRILL-5749
Project: Apache Drill
Issue Type: Bug
Components: Execution - RPC
Affects Versions: 1.11.0, 1.10.0
Reporter: weijie.tong
Priority: Critical
when the cluster was in high concurrency query and the reused control
connection occured exceptoin, the foreman and netty threads both try to acquire
each other's lock then deadlock occured. The netty thread hold the map
(RequestIdMap) lock then try to acquire the ReconnectingConnection lock to send
command, while the foreman thread hold the ReconnectingConnection lock then try
to acquire the RequestIdMap lock. So the deadlock happend.
Below is the jstack dump:
Found one Java-level deadlock:
=============================
"265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
waiting to lock monitor 0x00007f935b721f48 (object 0x0000000656affc40, a
org.apache.drill.exec.rpc.control.ControlConnectionManager),
which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"
"265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
waiting to lock monitor 0x00007f90de3b9648 (object 0x00000006b524d7e8, a
com.carrotsearch.hppc.IntObjectHashMap),
which is held by "BitServer-2"
"BitServer-2":
waiting to lock monitor 0x00007f935b721f48 (object 0x0000000656affc40, a
org.apache.drill.exec.rpc.control.ControlConnectionManager),
which is held by "265aa82f-d8c1-5df0-9946-003a4990db7e:foreman"
Java stack information for the threads listed above:
===================================================
"265aa5cb-e5e2-39ed-9c2f-7658b905372e:foreman":
at
org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:72)
- waiting to lock <0x0000000656affc40> (a
org.apache.drill.exec.rpc.control.ControlConnectionManager)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragments(ControlTunnel.java:66)
at
org.apache.drill.exec.work.foreman.Foreman.sendRemoteFragments(Foreman.java:1210)
at
org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1141)
at
org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1045)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:849)
"265aa82f-d8c1-5df0-9946-003a4990db7e:foreman":
at
org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:87)
- waiting to lock <0x00000006b524d7e8> (a
com.carrotsearch.hppc.IntObjectHashMap)
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:153)
at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:115)
at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:89)
at
org.apache.drill.exec.rpc.control.ControlConnection.send(ControlConnection.java:65)
at
org.apache.drill.exec.rpc.control.ControlTunnel$SendFragment.doRpcCall(ControlTunnel.java:160)
at
org.apache.drill.exec.rpc.control.ControlTunnel$SendFragment.doRpcCall(ControlTunnel.java:150)
at
org.apache.drill.exec.rpc.ListeningCommand.connectionAvailable(ListeningCommand.java:38)
at
org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:75)
- locked <0x0000000656affc40> (a
org.apache.drill.exec.rpc.control.ControlConnectionManager)
at
org.apache.drill.exec.rpc.control.ControlTunnel.sendFragments(ControlTunnel.java:66)
at
org.apache.drill.exec.work.foreman.Foreman.sendRemoteFragments(Foreman.java:1210)
at
org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1141)
at
org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1045)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:849)
"BitServer-2":
at
org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:72)
- waiting to lock <0x0000000656affc40> (a
org.apache.drill.exec.rpc.control.ControlConnectionManager)
at
org.apache.drill.exec.rpc.control.ControlTunnel.cancelFragment(ControlTunnel.java:71)
at
org.apache.drill.exec.work.foreman.QueryManager.cancelExecutingFragments(QueryManager.java:220)
at
org.apache.drill.exec.work.foreman.Foreman.moveToState(Foreman.java:968)
at
org.apache.drill.exec.work.foreman.Foreman.access$2600(Foreman.java:109)
at
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:1020)
at
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.processEvent(Foreman.java:1013)
at
org.apache.drill.common.EventProcessor.processEvents(EventProcessor.java:107)
at
org.apache.drill.common.EventProcessor.sendEvent(EventProcessor.java:65)
at
org.apache.drill.exec.work.foreman.Foreman$StateSwitch.addEvent(Foreman.java:1015)
at
org.apache.drill.exec.work.foreman.Foreman.addToEventQueue(Foreman.java:1033)
at
org.apache.drill.exec.work.foreman.Foreman$FragmentSubmitListener.failed(Foreman.java:1274)
at
org.apache.drill.exec.rpc.ListeningCommand$DeferredRpcOutcome.failed(ListeningCommand.java:50)
at
org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:134)
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:74)
at
org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:64)
at
com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
at
org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:58)
- locked <0x00000006b524d7e8> (a com.carrotsearch.hppc.IntObjectHashMap)
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:183)
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:165)
at
org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:142)
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:204)
at
org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:191)
at
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
at
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
at
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at
io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
at
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1099)
at
io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
at
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
at
io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
at
io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
at
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
at
io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73)
at
io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
at
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
at
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466)
at
org.apache.drill.exec.rpc.RpcExceptionHandler.exceptionCaught(RpcExceptionHandler.java:39)
at
io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
at
io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
at
io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at
io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
at
io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
at
io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at
io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
at
io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:253)
at
io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
at
io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:275)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
