liuguanghua created HDFS-18036:
----------------------------------
Summary: Interrupt the IPC parameter sending thread when a
connection is marked closed
Key: HDFS-18036
URL: https://issues.apache.org/jira/browse/HDFS-18036
Project: Hadoop HDFS
Issue Type: Bug
Reporter: liuguanghua
A production DFSRouter died with a native OOM while creating a new thread:
--------------- T H R E A D ---------------
Current thread (0x00007ee90de5d800): JavaThread "IPC Client (1604275183)
connection to NN:8020 from userA" daemon [_thread_new, id=65241,
stack(0x00007ee1e9442000,0x00007ee1e9543000)]
Stack: [0x00007ee1e9442000,0x00007ee1e9543000], sp=0x00007ee1e9541a20, free
space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xad21aa] VMError::report_and_die()+0x2ba
V [libjvm.so+0x5022db] report_vm_out_of_memory(char const*, int, unsigned long,
VMErrorType, char const*)+0x8b
V [libjvm.so+0x92fe33] os::Linux::commit_memory_impl(char*, unsigned long,
bool)+0x123
V [libjvm.so+0x92fefc] os::pd_commit_memory(char*, unsigned long, bool)+0xc
V [libjvm.so+0x927f8a] os::commit_memory(char*, unsigned long, bool)+0x2a
V [libjvm.so+0x92bc4f] os::pd_create_stack_guard_pages(char*, unsigned
long)+0x7f
V [libjvm.so+0xa771ce] JavaThread::create_stack_guard_pages()+0x5e
V [libjvm.so+0xa81454] JavaThread::run()+0x34
V [libjvm.so+0x930198] java_start(Thread*)+0x108
C [libpthread.so.0+0x7ea5] start_thread+0xc5
The failure was in create_stack_guard_pages() — the JVM could not mmap the
stack for yet another thread. The thread dump shows why:
||Thread kind||Count||
|{{IPC Parameter Sending Thread for ...}} (blocked)|*25,191*|
|{{IPC Client (...) connection to ...}}|2,319|
|Total Java threads|32,449|
Almost all threads target the same subcluster NameNode from the router. The
sending threads outnumber the actual connections by ~11:1, each one holding a
~1MB native stack — enough to exhaust the process's address space /
vm.max_map_count.
Root cause
When an IPC Connection is marked closed (idle timeout, I/O error, or a FATAL
response), its RpcRequestSender thread (IPC Parameter Sending Thread for ...)
is left blocked in rpcRequestQueue.poll(maxIdleTime). It only notices the close
after the poll times out, so the thread — and its stack / mapped memory —
lingers for up to ipc.client.connection.maxidletime after the connection is
already gone.
markClosed() calls notifyAll(), which only wakes the Connection.run() thread
(blocked in waitForWork()'s wait(timeout)). The sender thread blocks inside
SynchronousQueue.poll(timeout), which does not wait on the connection monitor,
so it is never woken and only exits once the poll times out. The connection
thread exits on close; the sender thread stays behind and accumulates. Under
heavy connection churn (e.g. a DFSRouter recycling pooled connections to its
subcluster NameNodes), these lingering threads pile up until the JVM can no
longer create new threads.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]