Hi Shanti! Thanks for your script! I ran it (with minor
modifications), and it turns out that the garbage
collection threads are indeed taking up the majority of
CPU (see below). What would you recommend I do to
tune/troubleshoot GC further? My server has 32G or RAM,
JVM has heap size of 26G
Thanks,
Kirill
PID CPU Mem JStack Info
0x253d 80 90.7 "GC task thread#3 (ParallelGC)"
prio=10 tid=0x0000000001938800 nid=0x253d runnable
0x2540 76 90.7 "GC task thread#6 (ParallelGC)"
prio=10 tid=0x000000000193e000 nid=0x2540 runnable
0x253b 74 90.7 "GC task thread#1 (ParallelGC)"
prio=10 tid=0x0000000001934800 nid=0x253b runnable
0x253e 74 90.7 "GC task thread#4 (ParallelGC)"
prio=10 tid=0x000000000193a000 nid=0x253e runnable
0x2541 74 90.7 "GC task thread#7 (ParallelGC)"
prio=10 tid=0x000000000193f800 nid=0x2541 runnable
0x253c 66 90.7 "GC task thread#2 (ParallelGC)"
prio=10 tid=0x0000000001936800 nid=0x253c runnable
0x253a 47 90.7 "GC task thread#0 (ParallelGC)"
prio=10 tid=0x0000000001933000 nid=0x253a runnable
0x253f 39 90.7 "GC task thread#5 (ParallelGC)"
prio=10 tid=0x000000000193c000 nid=0x253f runnable
0x2542 19 90.7 "VM Thread" prio=10
tid=0x00000000019b6000 nid=0x2542 runnable
0x2539 0 90.7 "main" prio=10
tid=0x0000000001925000 nid=0x2539 runnable
[0x00007fee9d444000]
0x2543 0 90.7 "Reference Handler" daemon prio=10
tid=0x00000000019bd800 nid=0x2543 runnable
[0x00007fee500d8000]
0x2544 0 90.7 "Finalizer" daemon prio=10
tid=0x00000000019c5800 nid=0x2544 in Object.wait()
[0x00007fee4ffd7000]
0x2545 0 90.7 "Signal Dispatcher" daemon prio=10
tid=0x0000000001a12000 nid=0x2545 runnable
[0x0000000000000000]
0x2546 0 90.7 "C2 CompilerThread0" daemon prio=10
tid=0x00007fee48001000 nid=0x2546 waiting on condition
[0x0000000000000000]
0x2547 0 90.7 "C2 CompilerThread1" daemon prio=10
tid=0x00007fee48004000 nid=0x2547 waiting on condition
[0x0000000000000000]
0x2548 0 90.7 "Service Thread" daemon prio=10
tid=0x0000000001a14800 nid=0x2548 runnable
[0x0000000000000000]
0x2549 0 90.7 "VM Periodic Task Thread" prio=10
tid=0x0000000001a1f800 nid=0x2549 waiting on condition
0x254a 0 90.7 "GC Daemon" daemon prio=10
tid=0x00007fee404b3000 nid=0x254a in Object.wait()
[0x00007fee4e1bd000]
0x254d 0 90.7
"ContainerBackgroundProcessor[StandardEngine[Catalina]]"
daemon prio=10 tid=0x00007fee40389800 nid=0x254d runnable
[0x00007fee4d820000]
0x254e 0 90.7 "http-apr-8080-Poller-0" daemon
prio=10 tid=0x00007fee40378000 nid=0x254e in Object.wait()
[0x00007fee4d661000]
0x254f 0 90.7 "http-apr-8080-CometPoller-0"
daemon prio=10 tid=0x00007fee40395000 nid=0x254f in
Object.wait() [0x00007fee4d4a2000]
0x2550 0 90.7 "http-apr-8080-Sendfile-0" daemon
prio=10 tid=0x00007fee40397000 nid=0x2550 in Object.wait()
[0x00007fee4d2b0000]
0x2551 0 90.7 "http-apr-8080-Acceptor-0" daemon
prio=10 tid=0x00007fee4072c000 nid=0x2551 runnable
[0x00007fee4d1af000]
0x2552 0 90.7 "http-apr-8080-AsyncTimeout" daemon
prio=10 tid=0x00007fee4072e000 nid=0x2552
sleeping[0x00007fee4d0ac000]
0x2553 0 90.7 "http-apr-8443-Poller-0" daemon
prio=10 tid=0x00007fee40730000 nid=0x2553 in Object.wait()
[0x00007fee4cfab000]
0x2554 0 90.7 "http-apr-8443-CometPoller-0"
daemon prio=10 tid=0x00007fee40732000 nid=0x2554 in
Object.wait() [0x00007fee4cdee000]
0x2555 0 90.7 "http-apr-8443-Acceptor-0" daemon
prio=10 tid=0x00007fee40734800 nid=0x2555 runnable
[0x00007fee4cced000]
0x2556 0 90.7 "http-apr-8443-AsyncTimeout" daemon
prio=10 tid=0x00007fee4073b800 nid=0x2556
sleeping[0x00007fee4cbec000]
0x2557 0 90.7 "ajp-apr-8009-Poller-0" daemon
prio=10 tid=0x00007fee4073d800 nid=0x2557 runnable
[0x00007fee4ca2f000]
0x2558 0 90.7 "ajp-apr-8009-CometPoller-0" daemon
prio=10 tid=0x00007fee4073f800 nid=0x2558 in Object.wait()
[0x00007fee4c872000]
0x2559 0 90.7 "ajp-apr-8009-Acceptor-0" daemon
prio=10 tid=0x00007fee40741800 nid=0x2559 runnable
[0x00007fee4c771000]
0x255a 0 90.7 "ajp-apr-8009-AsyncTimeout" daemon
prio=10 tid=0x00007fee40743800 nid=0x255a
sleeping[0x00007fee4c670000]
0x255f 0 90.7 "ajp-apr-8009-exec-5" daemon
prio=10 tid=0x0000000001cef000 nid=0x255f runnable
[0x00007fee4c157000]
0x2560 0 90.7 "ajp-apr-8009-exec-6" daemon
prio=10 tid=0x0000000001e93800 nid=0x2560 runnable
[0x00007fee47ffd000]
0x2561 0 90.7 "ajp-apr-8009-exec-7" daemon
prio=10 tid=0x0000000001c1e800 nid=0x2561 runnable
[0x00007fee4db39000]
0x2562 0 90.7 "ajp-apr-8009-exec-8" daemon
prio=10 tid=0x0000000001d41000 nid=0x2562 runnable
[0x00007fee4dc3a000]
0x2563 0 90.7 "ajp-apr-8009-exec-9" daemon
prio=10 tid=0x0000000001bf9800 nid=0x2563 runnable
[0x00007fee47efb000]
0x2564 0 90.7 "ajp-apr-8009-exec-10" daemon
prio=10 tid=0x0000000001cfe800 nid=0x2564 waiting on
condition [0x00007fee47dfa000]
0x2586 0 90.7 "ajp-apr-8009-exec-12" daemon
prio=10 tid=0x0000000001e85000 nid=0x2586 waiting on
condition [0x00007fee47bf6000]
0x2587 0 90.7 "pool-1-thread-1" prio=10
tid=0x0000000001fe4000 nid=0x2587 waiting on condition
[0x00007fee47af3000]
0x2635 0 90.7 "ajp-apr-8009-exec-15" daemon
prio=10 tid=0x000000000232d800 nid=0x2635 runnable
[0x00007fee38991000]
0x2637 0 90.7 "ajp-apr-8009-exec-17" daemon
prio=10 tid=0x0000000002392800 nid=0x2637 waiting on
condition [0x00007fee38790000]
0x2638 0 90.7 "ajp-apr-8009-exec-18" daemon
prio=10 tid=0x0000000002395000 nid=0x2638 waiting on
condition [0x00007fee3868f000]
0x263a 0 90.7 "ajp-apr-8009-exec-20" daemon
prio=10 tid=0x0000000002312800 nid=0x263a waiting on
condition [0x00007fee3848b000]
0x263d 0 90.7 "ajp-apr-8009-exec-23" daemon
prio=10 tid=0x0000000002324800 nid=0x263d runnable
[0x00007fee38187000]
0x50b 0 90.7 "http-apr-8443-exec-1" daemon
prio=10 tid=0x0000000002209000 nid=0x50b waiting on
condition [0x00007fee232b1000]
0x50c 0 90.7 "http-apr-8443-exec-2" daemon
prio=10 tid=0x000000000213c000 nid=0x50c waiting on
condition [0x00007fee233b2000]
0x50d 0 90.7 "http-apr-8443-exec-3" daemon
prio=10 tid=0x000000000239e000 nid=0x50d waiting on
condition [0x00007fee22baa000]
0x50e 0 90.7 "http-apr-8443-exec-4" daemon
prio=10 tid=0x0000000002aeb800 nid=0x50e waiting on
condition [0x00007fee3939d000]
0x50f 0 90.7 "http-apr-8443-exec-5" daemon
prio=10 tid=0x0000000002ad3000 nid=0x50f waiting on
condition [0x00007fee38f99000]
0x510 0 90.7 "http-apr-8443-exec-6" daemon
prio=10 tid=0x00000000025a1800 nid=0x510 waiting on
condition [0x00007fee397a1000]
0x511 0 90.7 "http-apr-8443-exec-7" daemon
prio=10 tid=0x0000000002188800 nid=0x511 waiting on
condition [0x00007fee45db8000]
0x512 0 90.7 "http-apr-8443-exec-8" daemon
prio=10 tid=0x00000000023b3800 nid=0x512 waiting on
condition [0x00007fee44aa5000]
0x513 0 90.7 "http-apr-8443-exec-9" daemon
prio=10 tid=0x0000000002164800 nid=0x513 waiting on
condition [0x00007fee3dae4000]
0x514 0 90.7 "http-apr-8443-exec-10" daemon
prio=10 tid=0x0000000002af6800 nid=0x514 waiting on
condition [0x00007fee279f8000]
0x19a7 0 90.7 "http-apr-8080-exec-1" daemon
prio=10 tid=0x0000000001ac2800 nid=0x19a7 waiting on
condition [0x00007fee3747b000]
0x19a8 0 90.7 "http-apr-8080-exec-2" daemon
prio=10 tid=0x000000000257d000 nid=0x19a8 waiting on
condition [0x00007fee3767d000]
0x28b0 0 90.7 "http-apr-8080-exec-3" daemon
prio=10 tid=0x000000000274c000 nid=0x28b0 waiting on
condition [0x00007fee3f8f7000]
0x28b1 0 90.7 "http-apr-8080-exec-4" daemon
prio=10 tid=0x00000000021c8800 nid=0x28b1 waiting on
condition [0x00007fee3adb7000]
0x46e7 0 90.7 "Keep-Alive-Timer" daemon prio=10
tid=0x0000000002160800 nid=0x46e7 runnable
[0x00007fee3d5df000]
0x29c5 0 90.7 "Keep-Alive-SocketCleaner" daemon
prio=10 tid=0x000000000242b000 nid=0x29c5 in Object.wait()
[0x00007fee1f877000]
0x6ef8 0 90.7 "http-apr-8080-exec-5" daemon
prio=10 tid=0x0000000002a56000 nid=0x6ef8 waiting on
condition [0x00007fee454af000]
0x6ef9 0 90.7 "http-apr-8080-exec-6" daemon
prio=10 tid=0x00000000022de800 nid=0x6ef9 waiting on
condition [0x00007fee45ab5000]
0x3d38 0 90.7 "http-apr-8080-exec-7" daemon
prio=10 tid=0x0000000002233000 nid=0x3d38 waiting on
condition [0x00007fee37980000]
0x3d39 0 90.7 "http-apr-8080-exec-8" daemon
prio=10 tid=0x00000000021f7800 nid=0x3d39 waiting on
condition [0x00007fee38087000]
0x64a3 0 90.7 "ajp-apr-8009-exec-36" daemon
prio=10 tid=0x00000000022dd000 nid=0x64a3 waiting on
condition [0x00007fee35158000]
0x64a6 0 90.7 "ajp-apr-8009-exec-39" daemon
prio=10 tid=0x0000000002add800 nid=0x64a6 runnable
[0x00007fee3bdc7000]
0x64a7 0 90.7 "ajp-apr-8009-exec-40" daemon
prio=10 tid=0x00000000029c0800 nid=0x64a7 waiting on
condition [0x00007fee24fce000]
0x64ab 0 90.7 "ajp-apr-8009-exec-44" daemon
prio=10 tid=0x0000000002924000 nid=0x64ab runnable
[0x00007fee20f8d000]
0x64ac 0 90.7 "ajp-apr-8009-exec-45" daemon
prio=10 tid=0x0000000002555800 nid=0x64ac waiting on
condition [0x00007fee256d5000]
0x64af 0 90.7 "ajp-apr-8009-exec-48" daemon
prio=10 tid=0x0000000001f8a800 nid=0x64af waiting on
condition [0x00007fee24ccb000]
0x64b0 0 90.7 "ajp-apr-8009-exec-49" daemon
prio=10 tid=0x000000000257b000 nid=0x64b0 runnable
[0x00007fee36d73000]
0x64b1 0 90.7 "ajp-apr-8009-exec-50" daemon
prio=10 tid=0x0000000001bb3000 nid=0x64b1 runnable
[0x00007fee36b71000]
0x64b2 0 90.7 "ajp-apr-8009-exec-51" daemon
prio=10 tid=0x000000000224a000 nid=0x64b2 runnable
[0x00007fee270ee000]
0x64b3 0 90.7 "ajp-apr-8009-exec-52" daemon
prio=10 tid=0x000000000216e800 nid=0x64b3 runnable
[0x00007fee20c8a000]
0xf1b 0 90.7 "http-apr-8080-exec-9" daemon
prio=10 tid=0x0000000001c9e800 nid=0xf1b waiting on
condition [0x00007fee3929c000]
0xf1c 0 90.7 "http-apr-8080-exec-10" daemon
prio=10 tid=0x0000000001e4c000 nid=0xf1c waiting on
condition [0x00007fee3d8e2000]
0x3eda 0 90.7 "ajp-apr-8009-exec-55" daemon
prio=10 tid=0x0000000002155000 nid=0x3eda runnable
[0x00007fee27cfb000]
0x3edb 0 90.7 "ajp-apr-8009-exec-56" daemon
prio=10 tid=0x000000000232f800 nid=0x3edb runnable
[0x00007fee27bfa000]
0x51c7 0 90.7 "ajp-apr-8009-exec-57" daemon
prio=10 tid=0x000000000211c800 nid=0x51c7 waiting on
condition [0x00007fee37c83000]
On 9/27/12 12:17 PM, Shanti Suresh wrote: