any idea? 2010/2/11 Zheng Lv <lvzheng19800...@gmail.com>
> Hello Everyone, > We often find many child processes in datanodes, which have already > finished for long time. And following are the jstack log: > Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode): > "DestroyJavaVM" prio=10 tid=0x00002aaac8019800 nid=0x2422 waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "NioProcessor-31" prio=10 tid=0x00000000439fa000 nid=0x2826 runnable > [0x000000004100a000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked <0x00002aaab9b5f6f8> (a sun.nio.ch.Util$1) > - locked <0x00002aaab9b5f710> (a > java.util.Collections$UnmodifiableSet) > - locked <0x00002aaab9b5f680> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > org.apache.mina.transport.socket.nio.NioProcessor.select(NioProcessor.java:65) > at > org.apache.mina.common.AbstractPollingIoProcessor$Worker.run(AbstractPollingIoProcessor.java:672) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > "pool-15-thread-1" prio=10 tid=0x00002aaac802d000 nid=0x2825 waiting on > condition [0x0000000041604000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab9b61620> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > "Attach Listener" daemon prio=10 tid=0x0000000043a22800 nid=0x2608 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "pool-3-thread-1" prio=10 tid=0x00000000438ad000 nid=0x243f waiting on > condition [0x0000000042575000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00002aaab9ae8770> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) > at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > "Thread for syncLogs" daemon prio=10 tid=0x00002aaac4446800 nid=0x2437 > waiting on condition [0x0000000040f09000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.mapred.Child$2.run(Child.java:87) > "Low Memory Detector" daemon prio=10 tid=0x0000000043771800 nid=0x242d > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "CompilerThread1" daemon prio=10 tid=0x000000004376e800 nid=0x242c waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "CompilerThread0" daemon prio=10 tid=0x000000004376a800 nid=0x242b waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x0000000043768800 nid=0x242a > runnable [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x0000000043746000 nid=0x2429 in > Object.wait() [0x0000000042171000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab9a8afe0> (a > java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) > - locked <0x00002aaab9a8afe0> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > "Reference Handler" daemon prio=10 tid=0x0000000043744000 nid=0x2428 in > Object.wait() [0x0000000042070000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:485) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x00002aaab96b13a0> (a java.lang.ref.Reference$Lock) > "VM Thread" prio=10 tid=0x000000004373d800 nid=0x2427 runnable > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000436eb800 nid=0x2423 > runnable > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000436ed800 nid=0x2424 > runnable > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000436ef800 nid=0x2425 > runnable > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000436f1000 nid=0x2426 > runnable > "VM Periodic Task Thread" prio=10 tid=0x0000000043774000 nid=0x242e waiting > on condition > JNI global references: 871 > any suggestion? > best regards, > LvZheng. >
