[ https://issues.apache.org/jira/browse/HBASE-11988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135057#comment-14135057 ]
Anoop Sam John commented on HBASE-11988: ---------------------------------------- {noformat} 2014-09-15 20:33:29,003 DEBUG [PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3] master.AssignmentManager(2676): Got transition OPENED for {f364c75f82c175a3b5c2c7d7cf810ce3 state=PENDING_OPEN, ts=1410813208967, server=asf901.gq1.ygridcore.net,44530,1410813206686} from asf901.gq1.ygridcore.net,44530,1410813206686 2014-09-15 20:33:29,003 INFO [PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3] master.RegionStates(949): Transition {f364c75f82c175a3b5c2c7d7cf810ce3 state=PENDING_OPEN, ts=1410813208967, server=asf901.gq1.ygridcore.net,44530,1410813206686} to {f364c75f82c175a3b5c2c7d7cf810ce3 state=OPEN, ts=1410813209003, server=asf901.gq1.ygridcore.net,44530,1410813206686} 2014-09-15 20:33:29,003 INFO [PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3] master.RegionStateStore(207): Updating row hbase:namespace,,1410813208374.f364c75f82c175a3b5c2c7d7cf810ce3. with state=OPEN&openSeqNum=2&server=asf901.gq1.ygridcore.net,44530,1410813206686 2014-09-15 20:33:29,006 INFO [PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3] master.RegionStates(381): Onlined f364c75f82c175a3b5c2c7d7cf810ce3 on asf901.gq1.ygridcore.net,44530,1410813206686 2014-09-15 20:33:29,007 DEBUG [PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3] regionserver.HRegionServer(1735): Finished post open deploy task for hbase:namespace,,1410813208374.f364c75f82c175a3b5c2c7d7cf810ce3. 2014-09-15 20:33:29,007 DEBUG [RS_OPEN_REGION-asf901:44530-0] handler.OpenRegionHandler(122): Opened hbase:namespace,,1410813208374.f364c75f82c175a3b5c2c7d7cf810ce3. on asf901.gq1.ygridcore.net,44530,1410813206686 2014-09-15 20:33:29,009 INFO [ActiveMasterManager] master.HMaster(621): Master has completed initialization 2014-09-15 20:33:29,025 ERROR [ActiveMasterManager] master.HMaster(633): Coprocessor postStartMaster() hook failed java.io.IOException: Table Namespace Manager not ready yet, try again later at org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1669) at org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:1852) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1096) at org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:186) at org.apache.hadoop.hbase.master.MasterCoprocessorHost$58.call(MasterCoprocessorHost.java:692) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:861) at org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:688) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:631) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:155) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1244) at java.lang.Thread.run(Thread.java:744) {noformat} So the ns table create was over and available before the VC try creating labels table. {code} void checkNamespaceManagerReady() throws IOException { checkInitialized(); if (tableNamespaceManager == null || !tableNamespaceManager.isTableAvailableAndInitialized()) { throw new IOException("Table Namespace Manager not ready yet, try again later"); } } {code} The check test whether ns table is available and *active*. The active state check will fail as we can see in the log that in master the state changed after this exception log {noformat} 2014-09-15 20:33:29,410 DEBUG [MASTER_TABLE_OPERATIONS-asf901:44530-0] util.FSTableDescriptors(658): Deleted table descriptor at hdfs://localhost:34588/user/jenkins/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2014-09-15 20:33:29,410 INFO [MASTER_TABLE_OPERATIONS-asf901:44530-0] util.FSTableDescriptors(624): Updated tableinfo=hdfs://localhost:34588/user/jenkins/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000002 2014-09-15 20:33:29,412 DEBUG [MASTER_TABLE_OPERATIONS-asf901:44530-0] master.TableStateManager(197): Table hbase:namespace written descriptor for state ENABLED 2014-09-15 20:33:29,412 DEBUG [MASTER_TABLE_OPERATIONS-asf901:44530-0] master.TableStateManager(199): Table hbase:namespace updated state to ENABLED {noformat} Now my Q is this. During master init we create the ns table, if it is not there, and wait in a while loop until the ns table is available (region opened) ( See TableNamespaceManager#start()) Can this loop check for the ns table active state also? > AC/VC system table create on postStartMaster fails too often in test > -------------------------------------------------------------------- > > Key: HBASE-11988 > URL: https://issues.apache.org/jira/browse/HBASE-11988 > Project: HBase > Issue Type: Bug > Reporter: Anoop Sam John > Assignee: Anoop Sam John > > See for example > {noformat} > 2014-09-16 04:02:08,833 ERROR [ActiveMasterManager] master.HMaster(633): > Coprocessor postStartMaster() hook failed > java.io.IOException: Table Namespace Manager not ready yet, try again later > at > org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1669) > at > org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:1852) > at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1096) > at > org.apache.hadoop.hbase.security.access.AccessControlLists.init(AccessControlLists.java:143) > at > org.apache.hadoop.hbase.security.access.AccessController.postStartMaster(AccessController.java:1059) > at > org.apache.hadoop.hbase.master.MasterCoprocessorHost$58.call(MasterCoprocessorHost.java:692) > at > org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:861) > at > org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:688) > at > org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:631) > at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:155) > at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1244) > at java.lang.Thread.run(Thread.java:744) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)