[ https://issues.apache.org/jira/browse/HBASE-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578298#comment-14578298 ]
Jerry He commented on HBASE-13845: ---------------------------------- ok. The master branch goes totally ZK-less region assignment. The problem does not show up under ZK-less assignment. We always call am.regionOffline(HRegionInfo.FIRST_META_REGIONINFO) before verifyAndAssignMetaWithRetries() when processing the meta server shutdown/crash: {code} // Assign meta if we were carrying it. // Check again: region may be assigned to other where because of RIT // timeout if (am.isCarryingMeta(serverName)) { LOG.info("Server " + serverName + " was carrying META. Trying to assign."); am.regionOffline(HRegionInfo.FIRST_META_REGIONINFO); verifyAndAssignMetaWithRetries(); {code} When using ZK-less assignment, am.regionOffline(HRegionInfo.FIRST_META_REGIONINFO) ==> RegionStateStore.updateRegionState() sets OFFLINE state on meta region server znode. {code} // meta state stored in zk. if (hri.isMetaRegion()) { // persist meta state in MetaTableLocator (which in turn is zk storage currently) try { MetaTableLocator.setMetaLocation(server.getZooKeeper(), newState.getServerName(), hri.getReplicaId(), newState.getState()); return; // Done } {code} Then MetaTableLocator.getMetaRegionLocation() will always return null. {code} public ServerName getMetaRegionLocation(final ZooKeeperWatcher zkw, int replicaId) { try { RegionState state = getMetaRegionState(zkw, replicaId); return state.isOpened() ? state.getServerName() : null; } catch (KeeperException ke) { return null; } } {code} So we will never try to re-use existing meta region server location on ZK. In the master branch code: {code} private void verifyAndAssignMeta(final MasterProcedureEnv env) throws InterruptedException, IOException, KeeperException { MasterServices services = env.getMasterServices(); if (!isMetaAssignedQuickTest(env)) { services.getAssignmentManager().assignMeta(HRegionInfo.FIRST_META_REGIONINFO); } else if (serverName.equals(services.getMetaTableLocator(). getMetaRegionLocation(services.getZooKeeper()))) { throw new IOException("hbase:meta is onlined on the dead server " + this.serverName); } else { LOG.info("Skip assigning hbase:meta because it is online at " + services.getMetaTableLocator().getMetaRegionLocation(services.getZooKeeper())); } } {code} isMetaAssignedQuickTest() will always return false. > Expire of one region server carrying meta can bring down the master > ------------------------------------------------------------------- > > Key: HBASE-13845 > URL: https://issues.apache.org/jira/browse/HBASE-13845 > Project: HBase > Issue Type: Bug > Components: master > Affects Versions: 2.0.0, 1.1.0, 1.2.0 > Reporter: Jerry He > Assignee: Jerry He > Fix For: 2.0.0, 1.2.0, 1.1.1 > > Attachments: HBASE-13845-branch-1.1.patch > > > There seems to be a code bug that can cause expiration of one region server > carrying meta to bring down the master under certain case. > Here is the sequence of event. > a) The master detects the expiration of a region server on ZK, and starts to > expire the region server. > b) Since the failed region server carries meta, the shutdown handler will > call verifyAndAssignMetaWithRetries() during processing the expired rs. > c) In verifyAndAssignMeta(), there is a logic to verifyMetaRegionLocation > {code} > (!server.getMetaTableLocator().verifyMetaRegionLocation(server.getConnection(), > this.server.getZooKeeper(), timeout)) { > this.services.getAssignmentManager().assignMeta > (HRegionInfo.FIRST_META_REGIONINFO); > } else if > (serverName.equals(server.getMetaTableLocator().getMetaRegionLocation( > this.server.getZooKeeper()))) { > throw new IOException("hbase:meta is onlined on the dead server " > + serverName); > {code} > If we see the meta region is still alive on the expired rs, we throw an > exception. > We do some retries (default 10x1000ms) for verifyAndAssignMeta. > If we still get the exception after retries, we abort the master. > {code} > 2015-05-27 06:58:30,156 FATAL > [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] master.HMaster: Master > server abort: loaded coprocessors are: [] > 2015-05-27 06:58:30,156 FATAL > [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] master.HMaster: > verifyAndAssignMeta failed after10 times retries, aborting > java.io.IOException: hbase:meta is onlined on the dead server > bdvs1164.svl.ibm.com,16020,1432681743203 > at > org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMeta(MetaServerShutdownHandler.java:162) > at > org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignMetaWithRetries(MetaServerShutdownHandler.java:184) > at > org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:93) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2015-05-27 06:58:30,156 INFO > [MASTER_META_SERVER_OPERATIONS-bdvs1163:60000-0] regionserver.HRegionServer: > STOPPED: verifyAndAssignMeta failed after10 times retries, aborting > {code} > The problem happens when the expired is slow processing its own expiration or > has a slow death, and is still able to respond to master's meta verification > in the meantime -- This message was sent by Atlassian JIRA (v6.3.4#6332)