[
https://issues.apache.org/jira/browse/HBASE-11988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135057#comment-14135057
]
Anoop Sam John commented on HBASE-11988:
----------------------------------------
{noformat}
2014-09-15 20:33:29,003 DEBUG
[PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3]
master.AssignmentManager(2676): Got transition OPENED for
{f364c75f82c175a3b5c2c7d7cf810ce3 state=PENDING_OPEN, ts=1410813208967,
server=asf901.gq1.ygridcore.net,44530,1410813206686} from
asf901.gq1.ygridcore.net,44530,1410813206686
2014-09-15 20:33:29,003 INFO
[PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3]
master.RegionStates(949): Transition {f364c75f82c175a3b5c2c7d7cf810ce3
state=PENDING_OPEN, ts=1410813208967,
server=asf901.gq1.ygridcore.net,44530,1410813206686} to
{f364c75f82c175a3b5c2c7d7cf810ce3 state=OPEN, ts=1410813209003,
server=asf901.gq1.ygridcore.net,44530,1410813206686}
2014-09-15 20:33:29,003 INFO
[PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3]
master.RegionStateStore(207): Updating row
hbase:namespace,,1410813208374.f364c75f82c175a3b5c2c7d7cf810ce3. with
state=OPEN&openSeqNum=2&server=asf901.gq1.ygridcore.net,44530,1410813206686
2014-09-15 20:33:29,006 INFO
[PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3]
master.RegionStates(381): Onlined f364c75f82c175a3b5c2c7d7cf810ce3 on
asf901.gq1.ygridcore.net,44530,1410813206686
2014-09-15 20:33:29,007 DEBUG
[PostOpenDeployTasks:f364c75f82c175a3b5c2c7d7cf810ce3]
regionserver.HRegionServer(1735): Finished post open deploy task for
hbase:namespace,,1410813208374.f364c75f82c175a3b5c2c7d7cf810ce3.
2014-09-15 20:33:29,007 DEBUG [RS_OPEN_REGION-asf901:44530-0]
handler.OpenRegionHandler(122): Opened
hbase:namespace,,1410813208374.f364c75f82c175a3b5c2c7d7cf810ce3. on
asf901.gq1.ygridcore.net,44530,1410813206686
2014-09-15 20:33:29,009 INFO [ActiveMasterManager] master.HMaster(621): Master
has completed initialization
2014-09-15 20:33:29,025 ERROR [ActiveMasterManager] master.HMaster(633):
Coprocessor postStartMaster() hook failed
java.io.IOException: Table Namespace Manager not ready yet, try again later
at
org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1669)
at
org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:1852)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1096)
at
org.apache.hadoop.hbase.security.visibility.VisibilityController.postStartMaster(VisibilityController.java:186)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost$58.call(MasterCoprocessorHost.java:692)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:861)
at
org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:688)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:631)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:155)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1244)
at java.lang.Thread.run(Thread.java:744)
{noformat}
So the ns table create was over and available before the VC try creating labels
table.
{code}
void checkNamespaceManagerReady() throws IOException {
checkInitialized();
if (tableNamespaceManager == null ||
!tableNamespaceManager.isTableAvailableAndInitialized()) {
throw new IOException("Table Namespace Manager not ready yet, try again
later");
}
}
{code}
The check test whether ns table is available and *active*. The active state
check will fail as we can see in the log that in master the state changed after
this exception log
{noformat}
2014-09-15 20:33:29,410 DEBUG [MASTER_TABLE_OPERATIONS-asf901:44530-0]
util.FSTableDescriptors(658): Deleted table descriptor at
hdfs://localhost:34588/user/jenkins/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001
2014-09-15 20:33:29,410 INFO [MASTER_TABLE_OPERATIONS-asf901:44530-0]
util.FSTableDescriptors(624): Updated
tableinfo=hdfs://localhost:34588/user/jenkins/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000002
2014-09-15 20:33:29,412 DEBUG [MASTER_TABLE_OPERATIONS-asf901:44530-0]
master.TableStateManager(197): Table hbase:namespace written descriptor for
state ENABLED
2014-09-15 20:33:29,412 DEBUG [MASTER_TABLE_OPERATIONS-asf901:44530-0]
master.TableStateManager(199): Table hbase:namespace updated state to ENABLED
{noformat}
Now my Q is this.
During master init we create the ns table, if it is not there, and wait in a
while loop until the ns table is available (region opened) ( See
TableNamespaceManager#start()) Can this loop check for the ns table active
state also?
> AC/VC system table create on postStartMaster fails too often in test
> --------------------------------------------------------------------
>
> Key: HBASE-11988
> URL: https://issues.apache.org/jira/browse/HBASE-11988
> Project: HBase
> Issue Type: Bug
> Reporter: Anoop Sam John
> Assignee: Anoop Sam John
>
> See for example
> {noformat}
> 2014-09-16 04:02:08,833 ERROR [ActiveMasterManager] master.HMaster(633):
> Coprocessor postStartMaster() hook failed
> java.io.IOException: Table Namespace Manager not ready yet, try again later
> at
> org.apache.hadoop.hbase.master.HMaster.checkNamespaceManagerReady(HMaster.java:1669)
> at
> org.apache.hadoop.hbase.master.HMaster.getNamespaceDescriptor(HMaster.java:1852)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1096)
> at
> org.apache.hadoop.hbase.security.access.AccessControlLists.init(AccessControlLists.java:143)
> at
> org.apache.hadoop.hbase.security.access.AccessController.postStartMaster(AccessController.java:1059)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$58.call(MasterCoprocessorHost.java:692)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:861)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.postStartMaster(MasterCoprocessorHost.java:688)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:631)
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:155)
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1244)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
