On 20/10/2010 21:37, Jason Britton wrote: > I have a tomcat instance hanging up (and currently still hung up) and would > really appreciate pointers on how to debug further. > A belated thanks to Konstantin for the info on kill -QUIT pid to retrieve > thread dumps But I was only able to pull thread dumps on the hung process > using jvisualvm. Process failed to respond to kill -QUIT for unknown > reasons. > > Environment: > Tomcat 6.0.29 - using native APR, running on 64-bit RHEL 5.5 - 32 GB ram > java version "1.6.0_21" > Java(TM) SE Runtime Environment (build 1.6.0_21-b06) > Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode) > > I pulled 4 thread dumps in a 10 minute span on hung tomcat process after it > was reported to be non-responsive. Unfortunately not any differences > between thread dumps and to my great dismay jstack is unable to retrieve any > stacktraces on any of the threads reporting the following error: > > Error occurred during stack walking: > sun.jvm.hotspot.debugger.DebuggerException: > sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp > at > sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:152) > at > sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:466) > at > sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65) > at > sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:92) > ............................................ > > From my googling this appears to be an unfixed bug with jstack, I sure would > like to be wrong about that so I could get some stacktrace info on the > threads of my hung process. > > Any thoughts on how to possibly get stacktrace info on these threads with > jstack broken(?) on my environment and any ideas on how to proceed with > debugging this situation would be greatly appreciated. Thanks - Jason > > Thread dump from hung tomcat process > Thread 27817 "catalina-exec-foo-20": (state = BLOCKED) > Thread 27267 "catalina-exec-foo-19": (state = BLOCKED) > Thread 27082 "catalina-exec-foo-18": (state = BLOCKED) > Thread 26591 "catalina-exec-foo-17": (state = BLOCKED) > Thread 25899 "catalina-exec-foo-16": (state = BLOCKED) > Thread 25590 "catalina-exec-foo-15": (state = BLOCKED) > Thread 25589 "catalina-exec-foo-14": (state = BLOCKED) > Thread 25586 "catalina-exec-foo-13": (state = BLOCKED) > Thread 25578 "catalina-exec-foo-12": (state = BLOCKED) > Thread 25565 "catalina-exec-foo-11": (state = BLOCKED) > Thread 17427 "Thread-40": (state = BLOCKED) > Thread 9128 "Thread-39": (state = IN_NATIVE) > Thread 16811 "catalina-exec-foo-10": (state = BLOCKED) > Thread 16810 "catalina-exec-foo-9": (state = BLOCKED) > Thread 16809 "catalina-exec-foo-8": (state = IN_NATIVE) > Thread 16808 "catalina-exec-foo-7": (state = BLOCKED) > Thread 16807 "catalina-exec-foo-6": (state = IN_NATIVE) > Thread 16806 "catalina-exec-foo-5": (state = BLOCKED) > Thread 16805 "Thread-8": (state = BLOCKED) > Thread 16804 "catalina-exec-foo-4": (state = BLOCKED) > Thread 16803 "catalina-exec-foo-3": (state = BLOCKED) > Thread 16802 "catalina-exec-foo-2": (state = IN_NATIVE) > Thread 16801 "catalina-exec-foo-1": (state = IN_NATIVE) > Thread 16791 "ajp-9009-Acceptor-0": (state = IN_NATIVE) > Thread 16790 "ajp-9009-CometPoller-0": (state = BLOCKED) > Thread 16789 "ajp-9009-Poller-0": (state = IN_NATIVE) > Thread 16788 "http-8080-Acceptor-0": (state = IN_NATIVE) > Thread 16787 "http-8080-Sendfile-0": (state = BLOCKED) > Thread 16786 "http-8080-CometPoller-0": (state = BLOCKED) > Thread 16785 "http-8080-Poller-0": (state = BLOCKED) > Thread 16784 "ContainerBackgroundProcessor[StandardEngine[Catalina]]": > (state = BLOCKED) > Thread 16783 "Thread-6": (state = BLOCKED) > Thread 16782 "Thread-5": (state = BLOCKED) > Thread 16781 "Thread-4": (state = BLOCKED) > Thread 16780 "Thread-3": (state = BLOCKED) > Thread 16778 "Thread-2": (state = BLOCKED) > Thread 16777 "GC Daemon": (state = BLOCKED) > VM Thread 16775 "Low Memory Detector": (state = BLOCKED) > VM Thread 16774 "CompilerThread1": (state = BLOCKED) > VM Thread 16773 "CompilerThread0": (state = BLOCKED) > Thread 16772 "Signal Dispatcher": (state = BLOCKED) > Thread 16771 "Finalizer": (state = BLOCKED) > Thread 16770 "Reference Handler": (state = BLOCKED) > Thread 16764 "main": (state = IN_NATIVE)
Lot of stuff blocked there. What is it blocked on? Can you get a strack trace for each thread? p
0x62590808.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
