[ https://issues.apache.org/jira/browse/HBASE-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Victor Xu updated HBASE-14062: ------------------------------ Description: We saw these blocked info in our jstack output: {noformat} "RpcServer.listener,port=60020" daemon prio=10 tid=0x00007f158097b800 nid=0x2cd05 waiting for monitor entry [0x0000000046374000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:833) - waiting to lock <0x00000002bb094ac8> (a java.util.Collections$SynchronizedList) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:748) {noformat} And the owner of the lock is LinkedList.remove: {noformat} "RpcServer.reader=9,port=60020" daemon prio=10 tid=0x00007f1580394000 nid=0x2cc19 runnable [0x0000000043b4c000] java.lang.Thread.State: RUNNABLE at java.util.LinkedList.remove(LinkedList.java:363) at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) - locked <0x00000002bb094ac8> (a java.util.Collections$SynchronizedList) at org.apache.hadoop.hbase.ipc.RpcServer.closeConnection(RpcServer.java:1992) - locked <0x00000002bb094ac8> (a java.util.Collections$SynchronizedList) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:867) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645) - locked <0x00000002bae09a30> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) {noformat} This issue blocked RS once in a while and I had to restart it whenever it happens. It seems like a bug. Any suggestions? was: We saw these blocked info in our jstack output: {noformat} "RpcServer.listener,port=60020" daemon prio=10 tid=0x00007f158097b800 nid=0x2cd05 waiting for monitor entry [0x0000000046374000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:833) - waiting to lock <0x00000002bb094ac8> (a java.util.Collections$SynchronizedList) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:748) {noformat} And the owner of the lock is LinkedList.remove: {noformat} "RpcServer.reader=9,port=60020" daemon prio=10 tid=0x00007f1580394000 nid=0x2cc19 runnable [0x0000000043b4c000] java.lang.Thread.State: RUNNABLE at java.util.LinkedList.remove(LinkedList.java:363) at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) - locked <0x00000002bb094ac8> (a java.util.Collections$SynchronizedList) at org.apache.hadoop.hbase.ipc.RpcServer.closeConnection(RpcServer.java:1992) - locked <0x00000002bb094ac8> (a java.util.Collections$SynchronizedList) at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:867) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645) - locked <0x00000002bae09a30> (a org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader) at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) {noformat} This issue blocked RS once in a while and I had to restart it whenever it happens. Any suggestions? > RpcServer.Listener.doAccept get blocked by LinkedList.remove > ------------------------------------------------------------ > > Key: HBASE-14062 > URL: https://issues.apache.org/jira/browse/HBASE-14062 > Project: HBase > Issue Type: Bug > Components: IPC/RPC > Affects Versions: 0.98.12 > Reporter: Victor Xu > Attachments: jstack.log > > > We saw these blocked info in our jstack output: > {noformat} > "RpcServer.listener,port=60020" daemon prio=10 tid=0x00007f158097b800 > nid=0x2cd05 waiting for monitor entry [0x0000000046374000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:833) > - waiting to lock <0x00000002bb094ac8> (a > java.util.Collections$SynchronizedList) > at > org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:748) > {noformat} > And the owner of the lock is LinkedList.remove: > {noformat} > "RpcServer.reader=9,port=60020" daemon prio=10 tid=0x00007f1580394000 > nid=0x2cc19 runnable [0x0000000043b4c000] > java.lang.Thread.State: RUNNABLE > at java.util.LinkedList.remove(LinkedList.java:363) > at > java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - locked <0x00000002bb094ac8> (a > java.util.Collections$SynchronizedList) > at > org.apache.hadoop.hbase.ipc.RpcServer.closeConnection(RpcServer.java:1992) > - locked <0x00000002bb094ac8> (a > java.util.Collections$SynchronizedList) > at > org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:867) > at > org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645) > - locked <0x00000002bae09a30> (a > org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > {noformat} > This issue blocked RS once in a while and I had to restart it whenever it > happens. It seems like a bug. Any suggestions? -- This message was sent by Atlassian JIRA (v6.3.4#6332)