[ https://issues.apache.org/jira/browse/APEXCORE-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419719#comment-15419719 ]
Pramod Immaneni commented on APEXCORE-505: ------------------------------------------ Taking a look at it > setup and activate calls in operator block heartbeat loop in container > ---------------------------------------------------------------------- > > Key: APEXCORE-505 > URL: https://issues.apache.org/jira/browse/APEXCORE-505 > Project: Apache Apex Core > Issue Type: Bug > Reporter: Ashwin Chandra Putta > > The setup and activate calls in the operators block the StreamingContainer > heartbeat loop to send heartbeats to application master. As a result, if > activation/setup takes more than the heartbeat timeout for any given operator > within a container, the app master ends up killing the container for > heartbeat timeout even though container is active. > To test this: I created simple test application and added sleep for 40 > seconds in setup or activate calls. The application master shows the heart > beat timeout message and kills the container with operator. Please find the > stack trace on the container while it was active as follows: > sleep in activate callback > =================== > {code} > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode): > "Attach Listener" daemon prio=10 tid=0x00007f8060ac0000 nid=0x6d2f waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "1/randomGenerator:RandomNumberGenerator" prio=10 tid=0x00007f8060b9b800 > nid=0x65a2 waiting on condition [0x00007f8050bc8000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at com.datatorrent.sample.MyOperator.activate(MyOperator.java:63) > at com.datatorrent.stram.engine.Node.activate(Node.java:619) > at > com.datatorrent.stram.engine.GenericNode.activate(GenericNode.java:205) > at > com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1336) > at > com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130) > at > com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1396) > "ProcessWideEventLoop" prio=10 tid=0x00007f8060afb800 nid=0x658b runnable > [0x00007f8050cc9000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007d186a380> (a > com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet) > - locked <0x00000007d185a4a0> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007d185a070> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) > at > com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:185) > at > com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157) > at > com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156) > at java.lang.Thread.run(Thread.java:745) > "Dispatcher-0" daemon prio=10 tid=0x00007f8060ae8000 nid=0x6586 waiting on > condition [0x00007f8050dca000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dc5fa118> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSyncAsyncMessageBus.java:51) > at java.lang.Thread.run(Thread.java:745) > "org.apache.hadoop.hdfs.PeerCache@389ed39" daemon prio=10 > tid=0x00007f8060abf800 nid=0x6577 waiting on condition [0x00007f8050fcc000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) > at java.lang.Thread.run(Thread.java:745) > "IPC Parameter Sending Thread #0" daemon prio=10 tid=0x00007f8060a2f800 > nid=0x6572 waiting on condition [0x00007f80510cd000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d511c0c0> (a > java.util.concurrent.SynchronousQueue$TransferStack) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thread-2" daemon prio=10 tid=0x00007f80609dd000 nid=0x656e runnable > [0x00007f80516f2000] > java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) > at > org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52) > at > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:474) > at java.lang.Thread.run(Thread.java:745) > "process reaper" daemon prio=10 tid=0x00007f80607c3000 nid=0x655d waiting on > condition [0x00007f80580df000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d2697c90> (a > java.util.concurrent.SynchronousQueue$TransferStack) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=10 tid=0x00007f80600bd800 nid=0x6554 runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=10 tid=0x00007f80600bb000 nid=0x6553 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=10 tid=0x00007f80600b8000 nid=0x6552 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x00007f80600b6800 nid=0x6551 runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x00007f8060096800 nid=0x6550 in Object.wait() > [0x00007f8052dec000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=10 tid=0x00007f8060094800 nid=0x654f in > Object.wait() [0x00007f8052eed000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > "main" prio=10 tid=0x00007f8060012000 nid=0x6538 waiting on condition > [0x00007f806487d000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded02ed0> (a > java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) > at > com.datatorrent.stram.engine.StreamingContainer.activate(StreamingContainer.java:1474) > - locked <0x00000007dc538160> (a > com.datatorrent.stram.engine.StreamingContainer) > at > com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:874) > - locked <0x00000007dc538160> (a > com.datatorrent.stram.engine.StreamingContainer) > at > com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:814) > at > com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:707) > at > com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:313) > "VM Thread" prio=10 tid=0x00007f8060090800 nid=0x654e runnable > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f8060028000 nid=0x653a > runnable > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f806002a000 nid=0x653b > runnable > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f806002c000 nid=0x653c > runnable > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f806002d800 nid=0x653d > runnable > "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f806002f800 nid=0x653e > runnable > "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f8060031800 nid=0x653f > runnable > "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f8060033000 nid=0x6540 > runnable > "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f8060035000 nid=0x6541 > runnable > "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f8060037000 nid=0x6542 > runnable > "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f8060039000 nid=0x6543 > runnable > "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f806003a800 nid=0x6544 > runnable > "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f806003c800 nid=0x6545 > runnable > "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f806003e800 nid=0x6546 > runnable > "GC task thread#13 (ParallelGC)" prio=10 tid=0x00007f8060040000 nid=0x6547 > runnable > "GC task thread#14 (ParallelGC)" prio=10 tid=0x00007f8060042000 nid=0x6548 > runnable > "GC task thread#15 (ParallelGC)" prio=10 tid=0x00007f8060044000 nid=0x6549 > runnable > "GC task thread#16 (ParallelGC)" prio=10 tid=0x00007f8060045800 nid=0x654a > runnable > "GC task thread#17 (ParallelGC)" prio=10 tid=0x00007f8060047800 nid=0x654b > runnable > "VM Periodic Task Thread" prio=10 tid=0x00007f80600c8800 nid=0x6555 waiting > on condition > JNI global references: 277 > {code} > sleep in setup callback > =================== > {code} > 2016-08-11 17:59:20 > Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode): > "Attach Listener" daemon prio=10 tid=0x00007fda00ab8000 nid=0x22ea waiting on > condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "1/randomGenerator:RandomNumberGenerator" prio=10 tid=0x00007fda00c08800 > nid=0x1ffc waiting on condition [0x00007fd9f06a2000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at com.datatorrent.sample.MyOperator.setup(MyOperator.java:64) > at com.datatorrent.sample.MyOperator.setup(MyOperator.java:25) > at com.datatorrent.stram.engine.Node.setup(Node.java:187) > at > com.datatorrent.stram.engine.StreamingContainer.setupNode(StreamingContainer.java:1309) > at > com.datatorrent.stram.engine.StreamingContainer.access$100(StreamingContainer.java:130) > at > com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1396) > "ProcessWideEventLoop" prio=10 tid=0x00007fda00af5800 nid=0x1fdb runnable > [0x00007fd9f07a3000] > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) > - locked <0x00000007d186a6e8> (a > com.datatorrent.netlet.OptimizedEventLoop$SelectedSelectionKeySet) > - locked <0x00000007d185a808> (a java.util.Collections$UnmodifiableSet) > - locked <0x00000007d185a3d8> (a sun.nio.ch.EPollSelectorImpl) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) > at > com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:185) > at > com.datatorrent.netlet.OptimizedEventLoop.runEventLoop(OptimizedEventLoop.java:157) > at > com.datatorrent.netlet.DefaultEventLoop.run(DefaultEventLoop.java:156) > at java.lang.Thread.run(Thread.java:745) > "Dispatcher-0" daemon prio=10 tid=0x00007fda00ae0000 nid=0x1fda waiting on > condition [0x00007fd9f08a4000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007dc5fc5b0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > net.engio.mbassy.bus.AbstractSyncAsyncMessageBus$1.run(AbstractSyncAsyncMessageBus.java:51) > at java.lang.Thread.run(Thread.java:745) > "org.apache.hadoop.hdfs.PeerCache@762eefed" daemon prio=10 > tid=0x00007fda00ab7800 nid=0x1fd7 waiting on condition [0x00007fd9f0aa6000] > java.lang.Thread.State: TIMED_WAITING (sleeping) > at java.lang.Thread.sleep(Native Method) > at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:255) > at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) > at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) > at java.lang.Thread.run(Thread.java:745) > "IPC Parameter Sending Thread #0" daemon prio=10 tid=0x00007fda00a27800 > nid=0x1fd5 waiting on condition [0x00007fd9f0ba7000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d511c610> (a > java.util.concurrent.SynchronousQueue$TransferStack) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Thread-2" daemon prio=10 tid=0x00007fda009d4800 nid=0x1fd3 runnable > [0x00007fd9f11cc000] > java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method) > at > org.apache.hadoop.net.unix.DomainSocketWatcher.access$900(DomainSocketWatcher.java:52) > at > org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:474) > at java.lang.Thread.run(Thread.java:745) > "process reaper" daemon prio=10 tid=0x00007fda007b3000 nid=0x1fc5 waiting on > condition [0x00007fd9f8090000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007d2698208> (a > java.util.concurrent.SynchronousQueue$TransferStack) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) > at > java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "Service Thread" daemon prio=10 tid=0x00007fda000bd800 nid=0x1fb2 runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread1" daemon prio=10 tid=0x00007fda000bb000 nid=0x1fb1 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "C2 CompilerThread0" daemon prio=10 tid=0x00007fda000b8000 nid=0x1fb0 waiting > on condition [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Signal Dispatcher" daemon prio=10 tid=0x00007fda000b6800 nid=0x1faf runnable > [0x0000000000000000] > java.lang.Thread.State: RUNNABLE > "Finalizer" daemon prio=10 tid=0x00007fda00096800 nid=0x1fad in Object.wait() > [0x00007fd9f28c6000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > - locked <0x00000007d0005608> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > "Reference Handler" daemon prio=10 tid=0x00007fda00094800 nid=0x1fac in > Object.wait() [0x00007fd9f29c7000] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:503) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x00000007d0005190> (a java.lang.ref.Reference$Lock) > "main" prio=10 tid=0x00007fda00012000 nid=0x1f98 waiting on condition > [0x00007fda042f0000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007ded06778> (a > java.util.concurrent.CountDownLatch$Sync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) > at > com.datatorrent.stram.engine.StreamingContainer.activate(StreamingContainer.java:1474) > - locked <0x00000007dc53a5f8> (a > com.datatorrent.stram.engine.StreamingContainer) > at > com.datatorrent.stram.engine.StreamingContainer.deploy(StreamingContainer.java:874) > - locked <0x00000007dc53a5f8> (a > com.datatorrent.stram.engine.StreamingContainer) > at > com.datatorrent.stram.engine.StreamingContainer.processHeartbeatResponse(StreamingContainer.java:814) > at > com.datatorrent.stram.engine.StreamingContainer.heartbeatLoop(StreamingContainer.java:707) > at > com.datatorrent.stram.engine.StreamingContainer.main(StreamingContainer.java:313) > "VM Thread" prio=10 tid=0x00007fda00090800 nid=0x1fab runnable > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fda00028000 nid=0x1f99 > runnable > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fda0002a000 nid=0x1f9a > runnable > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fda0002c000 nid=0x1f9b > runnable > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fda0002d800 nid=0x1f9c > runnable > "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007fda0002f800 nid=0x1f9d > runnable > "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007fda00031800 nid=0x1f9e > runnable > "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007fda00033000 nid=0x1f9f > runnable > "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007fda00035000 nid=0x1fa0 > runnable > "GC task thread#8 (ParallelGC)" prio=10 tid=0x00007fda00037000 nid=0x1fa1 > runnable > "GC task thread#9 (ParallelGC)" prio=10 tid=0x00007fda00039000 nid=0x1fa2 > runnable > "GC task thread#10 (ParallelGC)" prio=10 tid=0x00007fda0003a800 nid=0x1fa3 > runnable > "GC task thread#11 (ParallelGC)" prio=10 tid=0x00007fda0003c800 nid=0x1fa4 > runnable > "GC task thread#12 (ParallelGC)" prio=10 tid=0x00007fda0003e800 nid=0x1fa5 > runnable > "GC task thread#13 (ParallelGC)" prio=10 tid=0x00007fda00040000 nid=0x1fa6 > runnable > "GC task thread#14 (ParallelGC)" prio=10 tid=0x00007fda00042000 nid=0x1fa7 > runnable > "GC task thread#15 (ParallelGC)" prio=10 tid=0x00007fda00044000 nid=0x1fa8 > runnable > "GC task thread#16 (ParallelGC)" prio=10 tid=0x00007fda00045800 nid=0x1fa9 > runnable > "GC task thread#17 (ParallelGC)" prio=10 tid=0x00007fda00047800 nid=0x1faa > runnable > "VM Periodic Task Thread" prio=10 tid=0x00007fda000d0800 nid=0x1fb3 waiting > on condition > JNI global references: 237 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)