[
https://issues.apache.org/jira/browse/HBASE-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157546#comment-13157546
]
stack commented on HBASE-4729:
------------------------------
Looking at patch again:
{code}
+ return;
+ }
+ } catch (KeeperException ke) {
+ }
{code}
Can we not bury the 'return' down here in the catch block? Its going to be
responsible for someone losing hours trying to follow whats going on in here
(they'll miss this return for sure).
We should also log the exception above if it happens at the least.
I agree w/ what Ted says that fixing alter is a bigger problem than this jira,
that we should do that elsewhere. This issue stops master at least exiting
both for alter table + splitting scenario and for balancer + concurrent
splitting.
I made this issue critical for 0.92. Let me try and work up a test.
> Race between online altering and splitting kills the master
> -----------------------------------------------------------
>
> Key: HBASE-4729
> URL: https://issues.apache.org/jira/browse/HBASE-4729
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.92.0
> Reporter: Jean-Daniel Cryans
> Assignee: ramkrishna.s.vasudevan
> Priority: Critical
> Fix For: 0.92.0, 0.94.0
>
> Attachments: 4729.txt
>
>
> I was running an online alter while regions were splitting, and suddenly the
> master died and left my table half-altered (haven't restarted the master yet).
> What killed the master:
> {quote}
> 2011-11-02 17:06:44,428 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected ZK exception creating node CLOSING
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for /hbase/unassigned/f7e1783e65ea8d621a4bc96ad310f101
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:110)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:459)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:441)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:769)
> at
> org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:568)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1722)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1661)
> at org.apache.hadoop.hbase.master.BulkReOpen$1.run(BulkReOpen.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> A znode was created because the region server was splitting the region 4
> seconds before:
> {quote}
> 2011-11-02 17:06:40,704 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of
> region TestTable,0012469153,1320253135043.f7e1783e65ea8d621a4bc96ad310f101.
> 2011-11-02 17:06:40,704 DEBUG
> org.apache.hadoop.hbase.regionserver.SplitTransaction:
> regionserver:62023-0x132f043bbde0710 Creating ephemeral node for
> f7e1783e65ea8d621a4bc96ad310f101 in SPLITTING state
> 2011-11-02 17:06:40,751 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde0710 Attempting to transition node
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLITTING
> ...
> 2011-11-02 17:06:44,061 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:62023-0x132f043bbde0710 Successfully transitioned node
> f7e1783e65ea8d621a4bc96ad310f101 from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLIT
> 2011-11-02 17:06:44,061 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
> master to process the split for f7e1783e65ea8d621a4bc96ad310f101
> {quote}
> Now that the master is dead the region server is spewing those last two lines
> like mad.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
