[
https://issues.apache.org/jira/browse/HBASE-19866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341061#comment-16341061
]
Duo Zhang commented on HBASE-19866:
-----------------------------------
There is a dead lock.
NettyRpcServer.stop is synchronized, and in the method, we will wait until all
the connections to be closed.
And one of the event loop thread calls RpcServer.authorize which is also
synchronized, so it is blocked there and never returns back since we have
already locked when calling NettyRpcServer.stop.
The master thread.
{noformat}
Thread 209 (M:0;b10d9a4963f1:49296):
State: WAITING
Blocked count: 164
Waited count: 334
Waiting on
org.apache.hbase.thirdparty.io.netty.channel.group.DefaultChannelGroupFuture@7b59ef17
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:254)
org.apache.hbase.thirdparty.io.netty.channel.group.DefaultChannelGroupFuture.awaitUninterruptibly(DefaultChannelGroupFuture.java:183)
org.apache.hbase.thirdparty.io.netty.channel.group.DefaultChannelGroupFuture.awaitUninterruptibly(DefaultChannelGroupFuture.java:40)
org.apache.hadoop.hbase.ipc.NettyRpcServer.stop(NettyRpcServer.java:148)
org.apache.hadoop.hbase.regionserver.RSRpcServices.stop(RSRpcServices.java:1405)
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1134)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:570)
java.lang.Thread.run(Thread.java:748)
{noformat}
The event loop thread.
{noformat}
Thread 446 (RS-EventLoopGroup-1-4):
State: BLOCKED
Blocked count: 1
Waited count: 0
Blocked on org.apache.hadoop.hbase.ipc.NettyRpcServer@45c829a7
Blocked by 209 (M:0;b10d9a4963f1:49296)
Stack:
org.apache.hadoop.hbase.ipc.RpcServer.authorize(RpcServer.java:547)
org.apache.hadoop.hbase.ipc.ServerRpcConnection.authorizeConnection(ServerRpcConnection.java:485)
org.apache.hadoop.hbase.ipc.ServerRpcConnection.processOneRpc(ServerRpcConnection.java:465)
org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.process(NettyServerRpcConnection.java:94)
org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.process(NettyServerRpcConnection.java:78)
org.apache.hadoop.hbase.ipc.NettyServerRpcConnection.process(NettyServerRpcConnection.java:73)
org.apache.hadoop.hbase.ipc.NettyRpcServerRequestDecoder.channelRead(NettyRpcServerRequestDecoder.java:62)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
{noformat}
> TestRegionServerReportForDuty doesn't timeout
> ---------------------------------------------
>
> Key: HBASE-19866
> URL: https://issues.apache.org/jira/browse/HBASE-19866
> Project: HBase
> Issue Type: Sub-task
> Reporter: Appy
> Assignee: Appy
> Priority: Major
>
> So reading around junit docs
> (https://github.com/junit-team/junit4/wiki/timeout-for-tests), looks like the
> reason is result of these two rules:
> - @Test(timeout=X) applies only on the test function, and not on whole test
> fixture (@After, @Before, etc)
> - Timeout rule applies on whole test fixture
> TestRegionServerReportForDuty just has @Test(timeout=180000) and no Timeout
> rule unlike we have in so many other tests.
> The test method, in the logs I have, runs in less then 60 sec. So it meets
> the timeout specified in @Test annotation.
> However, we get stuck in tearDown, and since there is no Timeout rule, it
> keeps on running until surefire kills the JVM after
> forkedProcessTimeoutInSeconds (set to 900 sec).
> Let use the "Timeout" rule instead of {{@Test(timeout=180000)}}.
> *However, note that this won't solve the root cause of hangup.* It'll just
> make the test fail neatly rather than getting stuck and requiring surefire
> plugin to kill the forked JVMs (see HBASE-19803).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
