[
https://issues.apache.org/jira/browse/HDFS-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liuguanghua resolved HDFS-18035.
--------------------------------
Resolution: Invalid
> Interrupt the IPC parameter sending thread when a connection is marked closed
> -----------------------------------------------------------------------------
>
> Key: HDFS-18035
> URL: https://issues.apache.org/jira/browse/HDFS-18035
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: common
> Reporter: liuguanghua
> Assignee: liuguanghua
> Priority: Major
>
> A production DFSRouter died with a native OOM while creating a new thread:
> --------------- T H R E A D ---------------
> Current thread (0x00007ee90de5d800): JavaThread "IPC Client (1604275183)
> connection to NN:8020 from userA" daemon [_thread_new, id=65241,
> stack(0x00007ee1e9442000,0x00007ee1e9543000)]
> Stack: [0x00007ee1e9442000,0x00007ee1e9543000], sp=0x00007ee1e9541a20, free
> space=1022k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> V [libjvm.so+0xad21aa] VMError::report_and_die()+0x2ba
> V [libjvm.so+0x5022db] report_vm_out_of_memory(char const*, int, unsigned
> long, VMErrorType, char const*)+0x8b
> V [libjvm.so+0x92fe33] os::Linux::commit_memory_impl(char*, unsigned long,
> bool)+0x123
> V [libjvm.so+0x92fefc] os::pd_commit_memory(char*, unsigned long, bool)+0xc
> V [libjvm.so+0x927f8a] os::commit_memory(char*, unsigned long, bool)+0x2a
> V [libjvm.so+0x92bc4f] os::pd_create_stack_guard_pages(char*, unsigned
> long)+0x7f
> V [libjvm.so+0xa771ce] JavaThread::create_stack_guard_pages()+0x5e
> V [libjvm.so+0xa81454] JavaThread::run()+0x34
> V [libjvm.so+0x930198] java_start(Thread*)+0x108
> C [libpthread.so.0+0x7ea5] start_thread+0xc5
>
> The failure was in create_stack_guard_pages() — the JVM could not mmap the
> stack for yet another thread. The thread dump shows why:
> ||Thread kind||Count||
> |{{IPC Parameter Sending Thread for ...}} (blocked)|*25,191*|
> |{{IPC Client (...) connection to ...}}|2,319|
> |Total Java threads|32,449|
>
> Almost all threads target the same subcluster NameNode from the router. The
> sending threads outnumber the actual connections by ~11:1, each one holding a
> ~1MB native stack — enough to exhaust the process's address space /
> vm.max_map_count.
> Root cause
> When an IPC Connection is marked closed (idle timeout, I/O error, or a FATAL
> response), its RpcRequestSender thread (IPC Parameter Sending Thread for ...)
> is left blocked in rpcRequestQueue.poll(maxIdleTime). It only notices the
> close after the poll times out, so the thread — and its stack / mapped memory
> — lingers for up to ipc.client.connection.maxidletime after the connection is
> already gone.
> markClosed() calls notifyAll(), which only wakes the Connection.run() thread
> (blocked in waitForWork()'s wait(timeout)). The sender thread blocks inside
> SynchronousQueue.poll(timeout), which does not wait on the connection
> monitor, so it is never woken and only exits once the poll times out. The
> connection thread exits on close; the sender thread stays behind and
> accumulates. Under heavy connection churn (e.g. a DFSRouter recycling pooled
> connections to its subcluster NameNodes), these lingering threads pile up
> until the JVM can no longer create new threads.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]