Hey guys,
Yesterday our Hbase cluster had 4 of 11 regionserver don't work well, that
the numberOfOnlineRegions= 0 .
And when we restart the cluster,not only 4 but all of our regionservers this
occurs.
Here is the hbase master's log.Except the exception of the log ,we also find
few zookeeper's exception and log splitting exception.We can't find the real
cause.
Hope that helps and forgive my poor English : )
Thanks
Lee
2014-06-26 16:00:20,220 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:21,220 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:22,220 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:23,222 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:24,222 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:25,224 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:26,224 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:27,225 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:28,227 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:29,227 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:30,229 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:31,231 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:32,231 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:33,233 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:34,234 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:35,235 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:36,235 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:37,235 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:38,237 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:39,238 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:40,238 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:41,240 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:42,240 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:43,241 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:44,243 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager:
Skipping resubmissions of task
/hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h
adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05.
light.soufun.com%252C60020%252C1403110267692.1403720357924 because threshold
3 reached
2014-06-26 16:00:45,244 INFO org.apache.hadoop.hbase.master.SplitLogManager:
Skipping resubmissions of task
/hbase/splitlog/hdfs%3A%2F%2Fjyw-o-hadoop00%3A9000%2Fhbase%2F.logs%2Fjyw-o-h
adoop05.light.soufun.com%2C60020%2C1403110267692-splitting%2Fjyw-o-hadoop05.
light.soufun.com%252C60020%252C1403110267692.1403723402640 because threshold
3 reached
2014-06-26 16:00:45,244 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:46,245 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned =
0
2014-06-26 16:00:46,972 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60000: readAndProcess threw exception java.io.IOException:
Connection reset by peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServe
r.java:1179)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748
)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServe
r.java:539)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:514)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)
at java.lang.Thread.run(Thread.java:662)
