Yongle Zhang created HBASE-25259: ------------------------------------ Summary: After upgrading HBase from 2.1.1 to 2.2.0, table cannot be read and HMaster crashes Key: HBASE-25259 URL: https://issues.apache.org/jira/browse/HBASE-25259 Project: HBase Issue Type: Bug Affects Versions: 2.1.1, 2.2.0 Reporter: Yongle Zhang
When we upgraded HBASE cluster from 2.1.1 to 2.2.0, we failed to read data by using HBase shell command, and later HMaster crashes. Commandline error message: {code:java} HBase ShellUse "help" to get list of supported commands.Use "exit" to quit this interactive shell.For Reference, please visit: http://hbase.apache.org/2.0/book.html#shellVersion 2.2.0, r3ec693241f2edb0cf78e71739874f2323a86bf1e, Tue Nov 3 02:12:43 UTC 2020Took 0.0043 secondshbase(main):001:0> listTABLETestTable1 row(s)Took 0.5663 seconds=> ["TestTable"]hbase(main):002:0> scan 'TestTable'ROW COLUMN+CELL ERROR: Unknown table TestTable! For usage try 'help "scan"' Took 0.1510 secondshbase(main):003:0> {code} Log file: {code:java} 2020-11-07 04:03:57,346 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLEDat org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)... 4 more2020-11-07 04:03:57,347 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: []2020-11-07 04:03:57,347 ERROR [master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master bff37223dfc6,16000,1604721496292: Unhandled exception. Starting shutdown. *****java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLEDat org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)... 4 more2020-11-07 04:03:57,348 INFO [master/bff37223dfc6:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server 'bff37223dfc6,16000,1604721496292' *****2020-11-07 stopping=bff37223dfc6,16000,1604721496292; zookeeper connection closed.2020-11-07 04:03:57,612 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x1014e7bdcf800002020-11-07 04:03:57,620 ERROR [main] master.HMasterCommandLine: Master exitingjava.lang.RuntimeException: HMaster Abortedat org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:244)at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198) {code} Steps to reproduce: # ** Start up a cluster of version 2.1.1 with 3 nodes # Use hbase pe to write data. {code:java} /hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100 sequentialWrite 1{code} # Stop the cluster by: ## run “bash stop-hbase.sh” to shutdown the master. ## Set the property hbase.procedure.upgrade-to-2-2 to true in hbase-site.xml ## Start the master again and wait until the Master quits # Upgrade the node to 2.2.0 # After upgrading, try to read the data TestTable, the error message shows. Then, after a while, Master crashes. -- This message was sent by Atlassian Jira (v8.3.4#803005)