Re: hbase region servers refuse connection

[Rural Hunter]

Hi Tian Qiang,

Good to hear that. It seems the jira is fixed. I will seek time to build 
the latest 0.96 branch and test it.

于 2014/7/14 18:06, Qiang Tian 写道:
Hi YuMing, :)
yes. several iterations of jstack on the problem regionserver could help
identify the problem

Rural,
you probably hit hbase11277(and probably YuMin as well) - the reader 14
loops again and again in
below stack(high cpu usage) and listener 12 is blocked and cannot
accept new connections.



    1. Thread 12 (RpcServer.listener,port=60020):
    2.   State: BLOCKED
    3.   Blocked count: 123264191
    4.   Waited count: 0
    5.   Blocked on
    org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@77f87716
    6.   Blocked by 14 (RpcServer.reader=1,port=60020)
    7.   Stack:
    8.
    
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:598)
    9.
    org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:755)
    10.
    org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:673)
    11. Thread 24 (RpcServer.responder):



    1. Thread 14 (RpcServer.reader=1,port=60020):
    2.   State: RUNNABLE
    3.   Blocked count: 12510492
    4.   Waited count: 12826560
    5.   Stack:
    6.     sun.nio.ch.FileDispatcher.read0(Native Method)
    7.     sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    8.     sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
    9.     sun.nio.ch.IOUtil.read(IOUtil.java:224)
    10.     sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
    11.
    org.apache.hadoop.hbase.ipc.RpcServer.channelIO(RpcServer.java:2438)
    12.
    org.apache.hadoop.hbase.ipc.RpcServer.channelRead(RpcServer.java:2404)
    13.
    
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1498)
    14.
    org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:780)
    15.
    
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:568)
    16.
    
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:543)
    17.
    
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    18.
    
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    19.     java.lang.Thread.run(Thread.java:701)
    20. Thread 13 (RpcServer.reader=0,port=60020):
    21.



    1. 2014-07-10 14:13:49,614 WARN  [RpcServer.reader=7,port=60020]
    ipc.RpcServer: RpcServer.listener,port=60020: count of bytes read: 0
    2. java.io.IOException: Connection reset by peer
    3.         at sun.nio.ch.FileDispatcher.read0(Native Method)
    4.         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    5.         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
    6.         at sun.nio.ch.IOUtil.read(IOUtil.java:224)
    7.         at
    sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
    8.         at
    org.apache.hadoop.hbase.ipc.RpcServer.channelRead(RpcServer.java:2404)
    9.         at
    
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1425)
    10.         at
    org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:780)
    11.         at
    
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:568)
    12.         at
    
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:543)
    13.         at
    
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    14.         at
    
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    15.         at java.lang.Thread.run(Thread.java:701)