Regionserver is crashed frequently these days.,but It worked fine many months before these days . Some logs of one RS's log is as follows:
2011-04-01 19:13:40,413 WARN org.apache.hadoop.hbase.regionserver.Store: Failed open of hdfs:// master.uc.uuwatch.com:9000/hbase/cjjHTML/1494733632/page/5173469199902346167.1864097884; presumption is that file was corrupted at flush and lost edits picked up by commit log replay. Verify! java.io.IOException: Cannot open filename /hbase/cjjHTML/1864097884/page/5173469199902346167 ...... 2011-04-01 19:17:22,716 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: cjjHTML,http://news.ifeng.com/gundong/detail_2011_03/15/515 4913_0.shtml,1300245193111: Overloaded 2011-04-01 19:17:22,716 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: cjjHTML,http://news.ifeng.com/gundong/detail_2011_0 3/15/5154913_0.shtml,1300245193111: Overloaded 2011-04-01 19:17:22,716 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed cjjHTML, http://news.ifeng.com/gundong/detail_2011_03/15/5154913_0.shtml,1300245193111 2011-04-01 19:17:22,716 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE: cjjHTML,http://news.39.net/kyfx/2010124/1561005.html,1298035330808: Overloaded 2011-04-01 19:17:22,716 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_CLOSE: cjjHTML, http://news.39.net/kyfx/2010124/1561005.html,1298035330808: Overloaded ...... 2011-04-01 22:01:49,212 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x942f0f7ae13d0000 to sun.nio.ch.SelectionKeyImpl@34819c89 java.io.IOException: TIMED OUT at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906) 2011-04-01 22:01:49,213 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x982f0f7ae0960001 to sun.nio.ch.SelectionKeyImpl@7e9ca589 java.io.IOException: TIMED OUT at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906) 2011-04-01 22:01:49,310 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Attempt=1 org.apache.hadoop.hbase.Leases$LeaseStillHeldException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94) at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48) at org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:549) at java.lang.Thread.run(Thread.java:619) 2011-04-01 22:01:49,313 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Disconnected, type: None, path: null 2011-04-01 22:01:49,589 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server /192.168.5.155:2181 2011-04-01 22:01:49,589 INFO org.apache.zookeeper.ClientCnxn: Priming connection to java.nio.channels.SocketChannel[connected local=/ 192.168.5.149:33651 remote=/192.168.5.155:2181] 2011-04-01 22:01:49,590 INFO org.apache.zookeeper.ClientCnxn: Server connection successful 2011-04-01 22:01:49,592 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x982f0f7ae0960001 to sun.nio.ch.SelectionKeyImpl@a5858e9 java.io.IOException: Session Expired at org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) 2011-04-01 22:01:49,592 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Expired, type: None, path: null 2011-04-01 22:01:49,592 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session expired 2011-04-01 22:01:49,602 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=0.0, regions=941, stores=1721, storefiles=1439, storefileIndexSize=228, memstoreSize=558, compactionQueueSize=12, usedHeap=2896, maxHeap=3995, blockCacheSize=640319368, blockCacheFree=197676344, blockCacheCount=8309, blockCacheHitRatio=93, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0 2011-04-01 22:01:49,853 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server /192.168.5.147:2181 2011-04-01 22:01:49,854 INFO org.apache.zookeeper.ClientCnxn: Priming connection to java.nio.channels.SocketChannel[connected local=/ 192.168.5.149:50014 remote=/192.168.5.147:2181] 2011-04-01 22:01:49,854 INFO org.apache.zookeeper.ClientCnxn: Server connection successful 2011-04-01 22:01:49,856 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x942f0f7ae13d0000 to sun.nio.ch.SelectionKeyImpl@35c591b7 java.io.IOException: Session Expired at org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589) at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) 2011-04-01 22:01:49,856 INFO org.apache.zookeeper.ZooKeeper: Closing session: 0x942f0f7ae13d0000 2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ClientCnxn: Closing ClientCnxn for session: 0x942f0f7ae13d0000 2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ClientCnxn: Disconnecting ClientCnxn for session: 0x942f0f7ae13d0000 2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ZooKeeper: Session: 0x942f0f7ae13d0000 closed 2011-04-01 22:01:49,857 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2011-04-01 22:01:50,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020 2011-04-01 22:01:50,311 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting 2011-04-01 22:01:50,311 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020: exiting ...... -- Thanks & Best regards jiajun
