We are currently running with 0.92.0 (cdh4b1) and have configured HA so that the master will fail over automatically (we will be upgrading to 0.92.1 cdh4u0 soon). In testing failover we've noticed that in our long running processes which hold onto an instance of HTable or HTablePool when the master fails over our long running processes fail when interacting with HBase because of exceptions like the following. Should those instances be resilient to failover? I would at least expect for HTablePool to invalidate bad pooled tables. Is there a way to do this or a better way than catching RetriesExhaustedException and invalidating the pools ourself? Or is holding the HTablePool instance for a long time an anti-pattern?
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions: Mon Jun 11 14:04:26 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:27 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:28 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:29 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:31 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:33 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:37 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:41 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:04:49 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed Mon Jun 11 14:05:05 CDT 2012, org.apache.hadoop.hbase.client.HTable $5@72be6280, java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation@430a14ad closed at org.apache.hadoop.hbase.client.HConnectionManager $HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java: 1345) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:657) Thanks, Micah
