Jeffrey Zhong created HBASE-9665: ------------------------------------ Summary: Region gets lost when balancer & SSH both trying to assign Key: HBASE-9665 URL: https://issues.apache.org/jira/browse/HBASE-9665 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 0.96.0 Reporter: Jeffrey Zhong Priority: Critical
In summary, a server dies and its regions are re-assigned. While right before SSH, balancer is starting assign one region on the server to somewhere. The balancer assignment got preempted by the SSH assignment: {code} 2013-09-25 11:55:32,854 INFO Priority.RpcServer.handler=7,port=60020 regionserver.HRegionServer: Received CLOSE for the region:6deb1bfefe8cbdb443084efe919fdeb7 , which we are already trying to OPEN. Cancelling OPENING. {code} The SSH assignment(by GeneralBulkAssigner) failed too due to: {code} 2013-09-25 11:55:32,927 WARN [RS_OPEN_REGION-hor15n09:60020-2] zookeeper.ZKAssign: regionserver:60020-0x14153d449d30ad0 Attempt to transition the unassigned node for 6deb1bfefe8cbdb443084efe919fdeb7 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the server that tried to transition was hor15n09.gq1.ygridcore.net,60020,1380109280320 not the expected hor15n07.gq1.ygridcore.net,60020,1380109890414 {code} In the end, the region 6deb1bfefe8cbdb443084efe919fdeb7 is lost. Below is the master log, you can see both balancer and SSH try to assign the region around the same time: {code} 2013-09-25 11:55:32,731 INFO [MASTER_SERVER_OPERATIONS-hor15n05:60000-4] master.RegionStates: Transitioning {6deb1bfefe8cbdb443084efe919fdeb7 state=PENDING_CLOSE, ts=1380110132710, server=hor15n12.gq1.ygridcore.net,60020,1380109596307} will be handled by SSH for hor15n12.gq1.ygridcore.net,60020,1380109596307 ... 2013-09-25 11:55:32,849 INFO [hor15n05.gq1.ygridcore.net,60000,1380108611483-BalancerChore] master.RegionStates: Transitioned {6deb1bfefe8cbdb443084efe919fdeb7 state=OFFLINE, ts=1380110132768, server=null} to {6deb1bfefe8cbdb443084efe919fdeb7 state=PENDING_OPEN, ts=1380110132849, server=hor15n07.gq1.ygridcore.net,60020,1380109890414} ... 2013-09-25 11:55:32,898 INFO [hor15n05.gq1.ygridcore.net,60000,1380108611483-GeneralBulkAssigner-1] master.RegionStates: Transitioned {6deb1bfefe8cbdb443084efe919fdeb7 state=OFFLINE, ts=1380110132861, server=null} to {6deb1bfefe8cbdb443084efe919fdeb7 state=PENDING_OPEN, ts=1380110132898, server=hor15n09.gq1.ygridcore.net,60020,1380109280320} {code} Since SSH force region assignment while it doesn't recreate offline znode, the later region opening would fail with the following error. I'm suggesting to recreate offline znode when we force a region assignment(forceNewPlan=true) with low impact. {code} 2013-09-25 11:55:32,927 WARN [RS_OPEN_REGION-hor15n09:60020-2] zookeeper.ZKAssign: regionserver:60020-0x14153d449d30ad0 Attempt to transition the unassigned node for 6deb1bfefe8cbdb443084efe919fdeb7 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the server that tried to transition was hor15n09.gq1.ygridcore.net,60020,1380109280320 not the expected hor15n07.gq1.ygridcore.net,60020,1380109890414 {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira