[
https://issues.apache.org/jira/browse/HBASE-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014719#comment-14014719
]
Andrew Purtell commented on HBASE-11277:
----------------------------------------
Agreed with your points [~lhofhansl]. The other side is not disconnected and
there is space in the buffer. My understanding of the code after this change is
now in the data phase as long as we have read > 0 we will loop back to do
another immediate read call, but otherwise we will exit handling for the
connection and move on. We won't come back to the connection again until it is
selectable for reading again. In this sense I think the fix is correct. Without
the fix when a channel is selectable we go to read from it, and spin on reads
until the entire message is transmitted no matter how many times we spin in a
hot loop. With the fix when a channel is selectable we go to read from it but
then set it aside when read returns 0 for useful work.
> RPCServer threads can wedge under high load
> -------------------------------------------
>
> Key: HBASE-11277
> URL: https://issues.apache.org/jira/browse/HBASE-11277
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.96.2, 0.98.3
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Critical
> Fix For: 0.99.0, 0.96.3, 0.98.3
>
> Attachments: HBASE-11277.patch
>
>
> This is with 0.98.0 in an insecure setup with 7u55 and 7u60. Under high load,
> RPCServer threads can wedge, fail to make progess, and consume 100% CPU time
> on a core indefinitely.
> Dumping threads, all threads are in BLOCKED or IN_NATIVE state. The IN_NATIVE
> threads are mostly in EPollArrayWrapper.epollWait or
> FileDispatcherImpl.read0. The number of threads found in
> FileDispatcherImpl.read0 correspond to the number of runaway threads expected
> based on looking at 'top' output. These look like:
> {noformat}
> Thread 64758: (state = IN_NATIVE)
> - sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int)
> @bci=0 (Compiled frame; information may be imprecise)
> - sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int)
> @bci=4, line=39 (Compiled frame)
> - sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor,
> java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223
> (Compil
> ed frame)
> - sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long,
> sun.nio.ch.NativeDispatcher) @bci=48, line=197 (Compiled frame)
> - sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379
> (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer.channelRead(java.nio.channels.ReadableByteChannel,
> java.nio.ByteBuffer) @bci=12, line=2224 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess()
> @bci=509, line=1488 (Compiled frame)
> -
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(java.nio.channels.SelectionKey)
> @bci=23, line=790 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop() @bci=97,
> line=581 (Compiled frame)
> - org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run() @bci=1,
> line=556 (Interpreted frame)
> -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=95, line=1145 (Interpreted frame)
> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
> (Interpreted frame)
> - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
