[
https://issues.apache.org/jira/browse/HBASE-12166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158375#comment-14158375
]
Jimmy Xiang commented on HBASE-12166:
-------------------------------------
You are right. Not hosting namespace on master can solve the issue. Your fix is
fine with me. I'd like to look into it further to find out the root. Thanks.
> TestDistributedLogSplitting.testMasterStartsUpWithLogReplayWork
> ---------------------------------------------------------------
>
> Key: HBASE-12166
> URL: https://issues.apache.org/jira/browse/HBASE-12166
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: stack
> Assignee: stack
> Fix For: 2.0.0, 0.99.1
>
> Attachments: 12166.txt, log.txt
>
>
> See
> https://builds.apache.org/job/PreCommit-HBASE-Build/11204//testReport/org.apache.hadoop.hbase.master/TestDistributedLogSplitting/testMasterStartsUpWithLogReplayWork/
> The namespace region gets stuck. It is never 'recovered' even though we have
> finished log splitting. Here is the main exception:
> {code}
> 4941 2014-10-03 02:00:36,862 DEBUG
> [B.defaultRpcServer.handler=1,queue=0,port=37113] ipc.CallRunner(111):
> B.defaultRpcServer.handler=1,queue=0,port=37113: callId: 211 service:
> ClientService methodName: Get
> size: 99 connection: 67.195.81.144:44526
> 4942 org.apache.hadoop.hbase.exceptions.RegionInRecoveryException:
> hbase:namespace,,1412301462277.eba5d23de65f2718715eeb22edf7edc2. is recovering
> 4943 at
> org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:6058)
> 4944 at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2086)
> 4945 at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2072)
> 4946 at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:5014)
> 4947 at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4988)
> 4948 at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:1690)
> 4949 at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:30418)
> 4950 at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2020)
> 4951 at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
> 4952 at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
> 4953 at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
> 4954 at java.lang.Thread.run(Thread.java:744)
> {code}
> See how we've finished log splitting long time previous:
> {code}
> 2014-10-03 01:57:48,129 INFO [M_LOG_REPLAY_OPS-asf900:37113-1]
> master.SplitLogManager(294): finished splitting (more than or equal to)
> 197337 bytes in 1 log files in
> [hdfs://localhost:49601/user/jenkins/hbase/WALs/asf900.gq1.ygridcore.net,40732,1412301461887-splitting]
> in 379ms
> {code}
> If I grep for the deleting of znodes on recovery, which is when we set the
> recovering flag to false, I see a bunch of regions but not my namespace one:
> 2014-10-03 01:57:47,330 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66): /hbase/recovering-regions/1588230740
> znode deleted. Region: 1588230740 completes recovery.
> 2014-10-03 01:57:48,119 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66):
> /hbase/recovering-regions/adfdcf958dd958f0e2ce59072ce2209d znode deleted.
> Region: adfdcf958dd958f0e2ce59072ce2209d completes recovery.
> 2014-10-03 01:57:48,121 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66):
> /hbase/recovering-regions/41d438848305831b61d708a406d5ecde znode deleted.
> Region: 41d438848305831b61d708a406d5ecde completes recovery.
> 2014-10-03 01:57:48,122 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66):
> /hbase/recovering-regions/6a7cada80de2ae5d774fe8cd33bd4cda znode deleted.
> Region: 6a7cada80de2ae5d774fe8cd33bd4cda completes recovery.
> 2014-10-03 01:57:48,124 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66):
> /hbase/recovering-regions/65451bd5b38bd16a31e25b62b3305533 znode deleted.
> Region: 65451bd5b38bd16a31e25b62b3305533 completes recovery.
> 2014-10-03 01:57:48,125 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66):
> /hbase/recovering-regions/07afdc3748894cf2b56e0075272a95a0 znode deleted.
> Region: 07afdc3748894cf2b56e0075272a95a0 completes recovery.
> 2014-10-03 01:57:48,126 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66):
> /hbase/recovering-regions/a4337ad2874ee7e599ca2344fce21583 znode deleted.
> Region: a4337ad2874ee7e599ca2344fce21583 completes recovery.
> 2014-10-03 01:57:48,128 INFO [Thread-9216-EventThread]
> zookeeper.RecoveringRegionWatcher(66):
> /hbase/recovering-regions/9d91d6eafe260ce33e8d7d23ccd13192 znode deleted.
> Region: 9d91d6eafe260ce33e8d7d23ccd13192 completes recovery.
> This would seem to indicate that we successfully wrote zk that we are
> recovering:
> {code}
> 2014-10-03 01:57:47,672 DEBUG [MASTER_SERVER_OPERATIONS-asf900:37113-0]
> coordination.ZKSplitLogManagerCoordination(647): Mark region
> eba5d23de65f2718715eeb22edf7edc2 recovering from failed region server
> asf900.gq1.ygridcore.net,42071,1412301461790
> {code}
> As part of the open of namespace, we updated our last flushed id successfully:
> {code}
> 2ae5d774fe8cd33bd4cda/family
> 2014-10-03 01:57:47,698 DEBUG [Thread-9216-EventThread]
> zookeeper.ZooKeeperWatcher(304): master:37113-0x148d3b7d84b020f,
> quorum=localhost:50702, baseZNode=/hbase Received ZooKeeper Event,
> type=NodeDataChanged, state=SyncConnected,
> path=/hbase/recovering-regions/eba5d23de65f2718715eeb22edf7edc2
> 2014-10-03 01:57:47,699 DEBUG
> [PostOpenDeployTasks:eba5d23de65f2718715eeb22edf7edc2]
> regionserver.HRegionServer(2947): Update last flushed sequence id of region
> eba5d23de65f2718715eeb22edf7edc2 for
> asf900.gq1.ygridcore.net,42071,1412301461790
> {code}
> Not sure why this node is up in zk.
> Let me dig again later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
