Both daughter regions are not online. Thanks Weihua
2011/7/4 Ted Yu <[email protected]>: > In the future, please direct questions on cdh releases to > [email protected] > You may cc [email protected] > > There is more than one minute difference between master and RS logs. > Which one of the daughter regions didn't come online ? > > Cheers > > On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <[email protected]> wrote: > >> The HBase version we are using is CDH3U0. >> >> Thanks >> Weihua >> >> 2011/7/4 Weihua JIANG <[email protected]>: >> > Hi all, >> > >> > We encountered a problem about region not onlining. A region is >> > splitted by a closing RS and then this RS down. It seems master has >> > known this split but it doesn't tried to make it online. Log from >> > master >> > 2011-06-30 22:58:52,945 DEBUG >> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined >> > and split region >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.; >> > checking daughter presence >> > 2011-06-30 22:58:52,946 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: Handling >> > transition=RS_ZK_REGION_OPENING, >> > server=hadoop01.sh.intel.com,50820,1309421825940, >> > region=ed60ec735e30db1d99290995eb1cd2d7 >> > 2011-06-30 22:58:53,005 DEBUG >> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. >> > present >> > 2011-06-30 22:58:53,065 DEBUG >> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. >> > present >> > >> > Log from RS is: >> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC >> > Server handler 73 on 50820 caught: >> > java.nio.channels.ClosedChannelException >> > at >> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) >> > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) >> > at >> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342) >> > at >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727) >> > at >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792) >> > at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083) >> > >> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC >> > Server handler 73 on 50820: exiting >> > 2011-06-30 22:57:05,767 INFO >> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing >> > leases >> > 2011-06-30 22:57:05,768 INFO >> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed >> > leases >> > 2011-06-30 22:57:05,768 INFO >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >> > Closed zookeeper sessionid=0x130ba69074900b4 >> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session: >> > 0x130ba69074900b4 closed >> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn: >> > EventThread shut down >> > 2011-06-30 22:57:05,857 DEBUG >> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. >> > 2011-06-30 22:57:05,863 DEBUG >> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. >> > 2011-06-30 22:57:05,911 INFO >> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd. >> > in META >> > 2011-06-30 22:57:05,942 INFO >> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. >> > in region .META.,,1, serverInfo=null >> > 2011-06-30 22:57:05,943 INFO >> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening >> > daughter >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e. >> > because stopping=false, stopped=true >> > 2011-06-30 22:57:05,950 INFO >> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. >> > in region .META.,,1, serverInfo=null >> > 2011-06-30 22:57:05,950 INFO >> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening >> > daughter >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294. >> > because stopping=false, stopped=true >> > 2011-06-30 22:57:06,004 INFO >> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META >> > updated, and report to master. >> > >> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd., >> > new regions: >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e., >> > >> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.. >> > Split took 1mins, 12sec >> > 2011-06-30 22:57:06,004 DEBUG >> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for >> > Split Thread to finish... >> > 2011-06-30 22:57:06,004 DEBUG >> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for >> > Large Compaction Thread to finish... >> > 2011-06-30 22:57:06,004 DEBUG >> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for >> > Small Compaction Thread to finish... >> > 2011-06-30 22:57:06,004 INFO >> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820 >> > exiting >> > 2011-06-30 22:57:06,090 INFO >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook >> > starting; hbase.shutdown.hook=true; >> > fsShutdownHook=Thread[Thread-15,5,main] >> > 2011-06-30 22:57:06,090 INFO >> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown >> > hook >> > 2011-06-30 22:57:06,090 INFO >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs >> > shutdown hook thread. >> > 2011-06-30 22:57:06,196 INFO >> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook >> > finished. >> > >> > >> > Thanks >> > Weihua >> > >> >
