[ https://issues.apache.org/jira/browse/HBASE-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905790#action_12905790 ]
stack commented on HBASE-2952: ------------------------------ @Prakash Sorry, got distracted. So, let me try the reference count you suggest above. In HTable#close, along w/ flush, we'd call HConnectionManager.releaseConnection, a new method. When ref count went to zero, we'd do what the current HCM.deleteConnection does (I think I need to remove this method, or make it private). Let me put up a patch tomorrow. > HConnectionManager's shutdown hook interferes with client's operations > ---------------------------------------------------------------------- > > Key: HBASE-2952 > URL: https://issues.apache.org/jira/browse/HBASE-2952 > Project: HBase > Issue Type: Improvement > Components: client > Affects Versions: 0.20.0 > Reporter: Prakash Khemani > > My HBase client calls incrementColValue() in pairs. If someone kills the > client (SIGINT or SIGTERM) I want my client's increment threads to gracefully > exit. If a thread has already done one of the incrementColValue() then I want > that thread to complete the other incrementColValue() and then exit. > For this purpose I installed my own shutdownHook(). My shitdownHook() thread > 'sugnals' all the threads in my process that it is time to exit and then > waits for them to complete. > The problem is that HConnectionManager's shutdownHook thread also runs and > shuts down all connections and IPC threads. > My increment thread keeps waiting to increment and then times out after 240s. > Two problems with this - the incrementColValiue() didn't go through which > will increase the chances of inconsistency in my HBase data. And it too 240s > to exit. I am pasting some of the messages that the client thread outputs > while it tries contact the HBase server. > Signalled. Exiting ... > 2010-09-01 12:11:14,769 DEBUG [HCM.shutdownHook] > zookeeper.ZooKeeperWrapper(787): > <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Closed > connection with ZooKeeper; /hbase/root-region-server > flushing after 7899 > 2010-09-01 12:11:19,669 DEBUG [Line Processing Thread 0] > client.HConnectionManager$TableServers(903): Cache hit for row <> in > tableName .META.: location server hadoop2205.snc3.facebook.com:60020, > location region name .META.,,1.1028785192 > 2010-09-01 12:11:19,671 INFO [Line Processing Thread 0] > zookeeper.ZooKeeperWrapper(206): Reconnecting to zookeeper > 2010-09-01 12:11:19,671 DEBUG [Line Processing Thread 0] > zookeeper.ZooKeeperWrapper(212): > <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Connected > to zookeeper again > 2010-09-01 12:11:24,679 DEBUG [Line Processing Thread 0] > client.HConnectionManager$TableServers(964): Removed .META.,,1.1028785192 for > tableName=.META. from cache because of content_action_url_metrics,\x080r& > B\xF7\x81_T\x07\x08\x16uOrcom.gigya 429934274290948,99999999999999 > 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] > client.HConnectionManager$TableServers(857): locateRegionInMeta attempt 0 of > 4 failed; retrying after sleep of 5000 because: The client is stopped > 2010-09-01 12:11:24,680 DEBUG [Line Processing Thread 0] > zookeeper.ZooKeeperWrapper(470): > <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying to > read /hbase/root-region-server > 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] > zookeeper.ZooKeeperWrapper(489): > <localhost:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Read > ZNode /hbase/root-region-server got 10.26.119.190:60020 > 2010-09-01 12:11:24,681 DEBUG [Line Processing Thread 0] > client.HConnectionManager$TableServers(1116): Root region location changed. > Sleeping. > === > It might be a good idea to only run the HCM shutdown code when all the > HTables referring to it have been closed. That way the client can control > when the shutdown actually happens. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.