Hey guys, Yesterday our Hbase cluster had 4 of 11 regionserver don't work well, that the numberOfOnlineRegions= 0 . And when we restart the cluster,not only 4 but all of our regionservers this occurs. Here is the hbase master's log.Except the exception of the log ,we also find few zookeeper's exception and log splitting exception.We can't find the real cause.
Hope that helps and forgive my poor English : ) Thanks Lee 2014-06-26 16:00:20,220 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:21,220 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:22,220 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:23,222 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:24,222 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:25,224 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:26,224 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:27,225 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:28,227 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:29,227 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:30,229 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:31,231 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:32,231 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:33,233 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:34,234 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:35,235 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:36,235 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:37,235 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:38,237 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:39,238 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:40,238 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:41,240 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:42,240 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:43,241 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:44,243 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager: Skipping resubmissions of task /hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05. light.soufun.com%252C60020%252C1403110267692.1403720357924 because threshold 3 reached 2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager: Skipping resubmissions of task /hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05. light.soufun.com%252C60020%252C1403110267692.1403723402640 because threshold 3 reached 2014-06-26 16:00:45,244 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:46,245 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 0 2014-06-26 16:00:46,972 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000: readAndProcess threw exception java.io.IOException: Connection reset by peer. Count of bytes read: 0 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198) at sun.nio.ch.IOUtil.read(IOUtil.java:171) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245) at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796) at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServe r.java:1179) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748 ) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServe r.java:539) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java :514) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) at java.lang.Thread.run(Thread.java:662)