[ 
https://issues.apache.org/jira/browse/HDFS-18035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liuguanghua resolved HDFS-18035.
--------------------------------
    Resolution: Invalid

> Interrupt the IPC parameter sending thread when a connection is marked closed
> -----------------------------------------------------------------------------
>
>                 Key: HDFS-18035
>                 URL: https://issues.apache.org/jira/browse/HDFS-18035
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: common
>            Reporter: liuguanghua
>            Assignee: liuguanghua
>            Priority: Major
>
> A production DFSRouter died with a native OOM while creating a new thread:
> --------------- T H R E A D ---------------
> Current thread (0x00007ee90de5d800): JavaThread "IPC Client (1604275183) 
> connection to NN:8020 from userA" daemon [_thread_new, id=65241, 
> stack(0x00007ee1e9442000,0x00007ee1e9543000)]
> Stack: [0x00007ee1e9442000,0x00007ee1e9543000], sp=0x00007ee1e9541a20, free 
> space=1022k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native 
> code)
> V [libjvm.so+0xad21aa] VMError::report_and_die()+0x2ba
> V [libjvm.so+0x5022db] report_vm_out_of_memory(char const*, int, unsigned 
> long, VMErrorType, char const*)+0x8b
> V [libjvm.so+0x92fe33] os::Linux::commit_memory_impl(char*, unsigned long, 
> bool)+0x123
> V [libjvm.so+0x92fefc] os::pd_commit_memory(char*, unsigned long, bool)+0xc
> V [libjvm.so+0x927f8a] os::commit_memory(char*, unsigned long, bool)+0x2a
> V [libjvm.so+0x92bc4f] os::pd_create_stack_guard_pages(char*, unsigned 
> long)+0x7f
> V [libjvm.so+0xa771ce] JavaThread::create_stack_guard_pages()+0x5e
> V [libjvm.so+0xa81454] JavaThread::run()+0x34
> V [libjvm.so+0x930198] java_start(Thread*)+0x108
> C [libpthread.so.0+0x7ea5] start_thread+0xc5
>  
> The failure was in create_stack_guard_pages() — the JVM could not mmap the 
> stack for yet another thread. The thread dump shows why:
> ||Thread kind||Count||
> |{{IPC Parameter Sending Thread for ...}} (blocked)|*25,191*|
> |{{IPC Client (...) connection to ...}}|2,319|
> |Total Java threads|32,449|
>  
> Almost all threads target the same subcluster NameNode from the router. The 
> sending threads outnumber the actual connections by ~11:1, each one holding a 
> ~1MB native stack — enough to exhaust the process's address space / 
> vm.max_map_count.
> Root cause
> When an IPC Connection is marked closed (idle timeout, I/O error, or a FATAL 
> response), its RpcRequestSender thread (IPC Parameter Sending Thread for ...) 
> is left blocked in rpcRequestQueue.poll(maxIdleTime). It only notices the 
> close after the poll times out, so the thread — and its stack / mapped memory 
> — lingers for up to ipc.client.connection.maxidletime after the connection is 
> already gone.
> markClosed() calls notifyAll(), which only wakes the Connection.run() thread 
> (blocked in waitForWork()'s wait(timeout)). The sender thread blocks inside 
> SynchronousQueue.poll(timeout), which does not wait on the connection 
> monitor, so it is never woken and only exits once the poll times out. The 
> connection thread exits on close; the sender thread stays behind and 
> accumulates. Under heavy connection churn (e.g. a DFSRouter recycling pooled 
> connections to its subcluster NameNodes), these lingering threads pile up 
> until the JVM can no longer create new threads.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to