[ https://issues.apache.org/jira/browse/HBASE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14745457#comment-14745457 ]
stack commented on HBASE-14431: ------------------------------- bq. BTW is there a reason why HBASE-13337 is not committed to master branch? None. Mistake on my part. Fixed. Thanks for noticing [~asamir] > AsyncRpcClient#removeConnection() never removes connection from connections > pool if server fails > ------------------------------------------------------------------------------------------------ > > Key: HBASE-14431 > URL: https://issues.apache.org/jira/browse/HBASE-14431 > Project: HBase > Issue Type: Bug > Components: IPC/RPC > Affects Versions: 2.0.0, 1.0.2, 1.1.2 > Reporter: Samir Ahmic > Assignee: Samir Ahmic > Priority: Critical > > I was playing with master branch in distributed mode (3 rs + master + > backup_master) and notice strange behavior when i was testing this sequence > of events on single rs: /kill/start/run_balancer while client was writing > data to cluster (LoadTestTool). > I have notice that LTT fails with following: > {code} > 2015-09-09 11:05:58,364 INFO [main] client.AsyncProcess: #2, waiting for > some tasks to finish. Expected max=0, tasksInProgress=35 > Exception in thread "main" > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 > action: BindException: 1 time, > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228) > at > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208) > at > org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697) > at > org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211) > {code} > After some digging and adding some more logging in code i have notice that > following condition in {code}AsyncRpcClient.removeConnection(AsyncRpcChannel > connection) {code} is never true: > {code} > if (connectionInPool == connection) { > {code} > causing that {code}AsyncRpcChannel{code} connection is never removed from > {code}connections{code} pool in case rs fails. > After changing this condition to: > {code} > if (connectionInPool.address.equals(connection.address)) { > {code} > issue was resolved and client was removing failed server from connections > pool. > I will attach patch after running some more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)