I have a tomcat instance hanging up (and currently still hung up) and would
really appreciate pointers on how to debug further.
A belated thanks to Konstantin for the info on kill -QUIT pid to retrieve
thread dumps But I was only able to pull thread dumps on the hung process
using jvisualvm. Process failed to respond to kill -QUIT for unknown
reasons.
Environment:
Tomcat 6.0.29 - using native APR, running on 64-bit RHEL 5.5 - 32 GB ram
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
I pulled 4 thread dumps in a 10 minute span on hung tomcat process after it
was reported to be non-responsive. Unfortunately not any differences
between thread dumps and to my great dismay jstack is unable to retrieve any
stacktraces on any of the threads reporting the following error:
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException:
sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152)
at
sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466)
at
sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
at
sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92)
............................................
>From my googling this appears to be an unfixed bug with jstack, I sure would
like to be wrong about that so I could get some stacktrace info on the
threads of my hung process.
Any thoughts on how to possibly get stacktrace info on these threads with
jstack broken(?) on my environment and any ideas on how to proceed with
debugging this situation would be greatly appreciated. Thanks - Jason
Thread dump from hung tomcat process
Thread 27817 "catalina-exec-foo-20": (state = BLOCKED)
Thread 27267 "catalina-exec-foo-19": (state = BLOCKED)
Thread 27082 "catalina-exec-foo-18": (state = BLOCKED)
Thread 26591 "catalina-exec-foo-17": (state = BLOCKED)
Thread 25899 "catalina-exec-foo-16": (state = BLOCKED)
Thread 25590 "catalina-exec-foo-15": (state = BLOCKED)
Thread 25589 "catalina-exec-foo-14": (state = BLOCKED)
Thread 25586 "catalina-exec-foo-13": (state = BLOCKED)
Thread 25578 "catalina-exec-foo-12": (state = BLOCKED)
Thread 25565 "catalina-exec-foo-11": (state = BLOCKED)
Thread 17427 "Thread-40": (state = BLOCKED)
Thread 9128 "Thread-39": (state = IN_NATIVE)
Thread 16811 "catalina-exec-foo-10": (state = BLOCKED)
Thread 16810 "catalina-exec-foo-9": (state = BLOCKED)
Thread 16809 "catalina-exec-foo-8": (state = IN_NATIVE)
Thread 16808 "catalina-exec-foo-7": (state = BLOCKED)
Thread 16807 "catalina-exec-foo-6": (state = IN_NATIVE)
Thread 16806 "catalina-exec-foo-5": (state = BLOCKED)
Thread 16805 "Thread-8": (state = BLOCKED)
Thread 16804 "catalina-exec-foo-4": (state = BLOCKED)
Thread 16803 "catalina-exec-foo-3": (state = BLOCKED)
Thread 16802 "catalina-exec-foo-2": (state = IN_NATIVE)
Thread 16801 "catalina-exec-foo-1": (state = IN_NATIVE)
Thread 16791 "ajp-9009-Acceptor-0": (state = IN_NATIVE)
Thread 16790 "ajp-9009-CometPoller-0": (state = BLOCKED)
Thread 16789 "ajp-9009-Poller-0": (state = IN_NATIVE)
Thread 16788 "http-8080-Acceptor-0": (state = IN_NATIVE)
Thread 16787 "http-8080-Sendfile-0": (state = BLOCKED)
Thread 16786 "http-8080-CometPoller-0": (state = BLOCKED)
Thread 16785 "http-8080-Poller-0": (state = BLOCKED)
Thread 16784 "ContainerBackgroundProcessor[StandardEngine[Catalina]]":
(state = BLOCKED)
Thread 16783 "Thread-6": (state = BLOCKED)
Thread 16782 "Thread-5": (state = BLOCKED)
Thread 16781 "Thread-4": (state = BLOCKED)
Thread 16780 "Thread-3": (state = BLOCKED)
Thread 16778 "Thread-2": (state = BLOCKED)
Thread 16777 "GC Daemon": (state = BLOCKED)
VM Thread 16775 "Low Memory Detector": (state = BLOCKED)
VM Thread 16774 "CompilerThread1": (state = BLOCKED)
VM Thread 16773 "CompilerThread0": (state = BLOCKED)
Thread 16772 "Signal Dispatcher": (state = BLOCKED)
Thread 16771 "Finalizer": (state = BLOCKED)
Thread 16770 "Reference Handler": (state = BLOCKED)
Thread 16764 "main": (state = IN_NATIVE)
