[ https://issues.apache.org/jira/browse/HBASE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14707932#comment-14707932 ]
ramkrishna.s.vasudevan commented on HBASE-13480: ------------------------------------------------ In my cluster when ever I try to do scan with YCSB and when the number of client threads are higher I end up in this problem very frequently. As Stack said this is critical in trunk. Good one. > ShortCircuitConnection doesn't short-circuit all calls as expected > ------------------------------------------------------------------ > > Key: HBASE-13480 > URL: https://issues.apache.org/jira/browse/HBASE-13480 > Project: HBase > Issue Type: Bug > Components: Client > Affects Versions: 1.0.0, 2.0.0, 1.1.0 > Reporter: Josh Elser > Assignee: Jingcheng Du > Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-13480-1.patch, HBASE-13480.patch > > > Noticed the following situation in debugging unexpected unit tests failures > in HBASE-13351. > {{ConnectionUtils#createShortCircuitHConnection(Connection, ServerName, > AdminService.BlockingInterface, ClientService.BlockingInterface)}} is > intended to avoid the extra RPC by calling the server's instantiation of the > protobuf rpc stub directly for the AdminService and ClientService. > The problem is that this is insufficient to actually avoid extra "remote" > RPCs as all other calls to the Connection are routed to a "real" Connection > instance. As such, any object created by the "real" Connection (such as an > HTable) will use the real Connection, not the SSC. > The end result is that > {{MasterRpcService#reportRegionStateTransition(RpcController, > ReportRegionStateTransitionRequest)}} will make additional "remote" RPCs over > what it thinks is an SSC through a {{Get}} on {{HTable}} which was > constructed using the SSC, but the {{Get}} itself will use the underlying > real Connection instead of the SSC. With insufficiently sized thread pools, > this has been observed to result in RPC deadlock in the HMaster where an RPC > attempts to make another RPC but there are no more threads available to > service the second RPC so the first RPC blocks indefinitely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)