Yongle Zhang created HBASE-25259:
------------------------------------
Summary: After upgrading HBase from 2.1.1 to 2.2.0, table cannot
be read and HMaster crashes
Key: HBASE-25259
URL: https://issues.apache.org/jira/browse/HBASE-25259
Project: HBase
Issue Type: Bug
Affects Versions: 2.1.1, 2.2.0
Reporter: Yongle Zhang
When we upgraded HBASE cluster from 2.1.1 to 2.2.0, we failed to read data by
using HBase shell command, and later HMaster crashes.
Commandline error message:
{code:java}
HBase ShellUse "help" to get list of supported commands.Use "exit" to quit this
interactive shell.For Reference, please visit:
http://hbase.apache.org/2.0/book.html#shellVersion 2.2.0,
r3ec693241f2edb0cf78e71739874f2323a86bf1e, Tue Nov 3 02:12:43 UTC 2020Took
0.0043 secondshbase(main):001:0> listTABLETestTable1 row(s)Took 0.5663
seconds=> ["TestTable"]hbase(main):002:0> scan 'TestTable'ROW
COLUMN+CELL
ERROR: Unknown table TestTable!
For usage try 'help "scan"'
Took 0.1510 secondshbase(main):003:0>
{code}
Log file:
{code:java}
2020-11-07 04:03:57,346 ERROR [master/bff37223dfc6:16000:becomeActiveMaster]
master.HMaster: Failed to become active masterjava.lang.IllegalStateException:
Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the
service has FAILEDat
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at
org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at
java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout
300000ms waiting for namespace table to be assigned and enabled:
tableName=hbase:namespace, state=ENABLEDat
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)...
4 more2020-11-07 04:03:57,347 ERROR
[master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: Master server
abort: loaded coprocessors are: []2020-11-07 04:03:57,347 ERROR
[master/bff37223dfc6:16000:becomeActiveMaster] master.HMaster: ***** ABORTING
master bff37223dfc6,16000,1604721496292: Unhandled exception. Starting
shutdown. *****java.lang.IllegalStateException: Expected the service
ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:366)at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:312)at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1344)at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1122)at
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2353)at
org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:598)at
java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Timedout
300000ms waiting for namespace table to be assigned and enabled:
tableName=hbase:namespace, state=ENABLEDat
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)at
org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)at
org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:248)at
org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1342)...
4 more2020-11-07 04:03:57,348 INFO
[master/bff37223dfc6:16000:becomeActiveMaster] regionserver.HRegionServer:
***** STOPPING region server 'bff37223dfc6,16000,1604721496292' *****2020-11-07
stopping=bff37223dfc6,16000,1604721496292; zookeeper connection
closed.2020-11-07 04:03:57,612 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down for session: 0x1014e7bdcf800002020-11-07 04:03:57,620
ERROR [main] master.HMasterCommandLine: Master
exitingjava.lang.RuntimeException: HMaster Abortedat
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:244)at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)at
org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3198)
{code}
Steps to reproduce:
# ** Start up a cluster of version 2.1.1 with 3 nodes
# Use hbase pe to write data.
{code:java}
/hbase/bin/hbase pe --nomapred --oneCon=true --valueSize=10 --rows=100
sequentialWrite 1{code}
# Stop the cluster by:
## run “bash stop-hbase.sh” to shutdown the master.
## Set the property hbase.procedure.upgrade-to-2-2 to true in hbase-site.xml
## Start the master again and wait until the Master quits
# Upgrade the node to 2.2.0
# After upgrading, try to read the data TestTable, the error message shows.
Then, after a while, Master crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
