Hello Everyone,
We often find many child processes in datanodes, which have already
finished for long time. And following are the jstack log:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):
"DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable
[0x000000004100a000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1)
- locked <0x00002aaab9b5f710> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaab9b5f680> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65)
at
org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting on
condition [0x0000000041604000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab9b61620> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Attach Listener" daemon prio=10 tid=0x0000000043a22800 nid=0x2608 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"pool-3-thread-1" prio=10 tid=0x00000000438ad000 nid=0x243f waiting on
condition [0x0000000042575000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaab9ae8770> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
"Thread for syncLogs" daemon prio=10 tid=0x00002aaac4446800 nid=0x2437
waiting on condition [0x0000000040f09000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
"Low Memory Detector" daemon prio=10 tid=0x0000000043771800 nid=0x242d
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x000000004376e800 nid=0x242c waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x000000004376a800 nid=0x242b waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x0000000043768800 nid=0x242a
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x0000000043746000 nid=0x2429 in
Object.wait() [0x0000000042171000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab9a8afe0> (a
java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00002aaab9a8afe0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x0000000043744000 nid=0x2428 in
Object.wait() [0x0000000042070000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x000000004373d800 nid=0x2427 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000436eb800 nid=0x2423
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000436ed800 nid=0x2424
runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000436ef800 nid=0x2425
runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000436f1000 nid=0x2426
runnable
"VM Periodic Task Thread" prio=10 tid=0x0000000043774000 nid=0x242e waiting
on condition
JNI global references: 871
any suggestion?
best regards,
LvZheng.
