Hi all,
We encountered a problem about region not onlining. A region is
splitted by a closing RS and then this RS down. It seems master has
known this split but it doesn't tried to make it online. Log from
master
2011-06-30 22:58:52,945 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
and split region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
checking daughter presence
2011-06-30 22:58:52,946 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING,
server=hadoop01.sh.intel.com,50820,1309421825940,
region=ed60ec735e30db1d99290995eb1cd2d7
2011-06-30 22:58:53,005 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
present
2011-06-30 22:58:53,065 DEBUG
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
present
Log from RS is:
2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 73 on 50820 caught:
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 73 on 50820: exiting
2011-06-30 22:57:05,767 INFO
org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
leases
2011-06-30 22:57:05,768 INFO
org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
leases
2011-06-30 22:57:05,768 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x130ba69074900b4
2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
0x130ba69074900b4 closed
2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
EventThread shut down
2011-06-30 22:57:05,857 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
2011-06-30 22:57:05,863 DEBUG
org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
2011-06-30 22:57:05,911 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
in META
2011-06-30 22:57:05,942 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
in region .META.,,1, serverInfo=null
2011-06-30 22:57:05,943 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
because stopping=false, stopped=true
2011-06-30 22:57:05,950 INFO
org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
in region .META.,,1, serverInfo=null
2011-06-30 22:57:05,950 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
daughter
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
because stopping=false, stopped=true
2011-06-30 22:57:06,004 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
new regions:
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
Split took 1mins, 12sec
2011-06-30 22:57:06,004 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
Split Thread to finish...
2011-06-30 22:57:06,004 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
Large Compaction Thread to finish...
2011-06-30 22:57:06,004 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
Small Compaction Thread to finish...
2011-06-30 22:57:06,004 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
exiting
2011-06-30 22:57:06,090 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
starting; hbase.shutdown.hook=true;
fsShutdownHook=Thread[Thread-15,5,main]
2011-06-30 22:57:06,090 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
hook
2011-06-30 22:57:06,090 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
shutdown hook thread.
2011-06-30 22:57:06,196 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
finished.
Thanks
Weihua
