Re: Tomcat hung
On 18/11/2010 16:23, Caldarale, Charles R wrote: > Yes, it was identifying it just as 10.0-b19 that was confusing. Lightbulb: this was the Hotspot VM version number. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: EXTERNAL: Re: Tomcat hung
In your full production trace, is there an indication which thread has the "<0xd50244e8>" lock? I've debugged similar situations where threads were "blocked waiting for monitor <0x.>" (getting java.sql.Connections, as it turned out) -- identifying & examining the thread that had locked that monitor (with a message something like "grabbed monitor <0x>") helped quite a bit w/debugging. On Thu, Nov 18, 2010 at 10:35 AM, Robillard, Greg L wrote: > Production, as of yet, strangely, I have not been able to create this problem > testing. > > Here is the top of the stack without a profiler. > > 2010-10-04 10:35:50 > Full thread dump Java HotSpot(TM) Server VM (10.0-b19 mixed mode): > > "Attach Listener" daemon prio=10 tid=0x08170400 nid=0x716b waiting on > condition [0x..0x] > java.lang.Thread.State: RUNNABLE > > "http-8080-200" daemon prio=10 tid=0xcbca9800 nid=0xb5e waiting on condition > [0xc5dbc000..0xc5dbcea0] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xd50244e8> (a > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) > at java.util.concurrent.locks.LockSupport.park(Unknown Source) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown > Source) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown > Source) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown > Source) > at > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(Unknown > Source) > at > com.lmco.fltwinds.wxserver.wcs.ScanningCoverage.getOffering(ScanningCoverage.java:286) > at > com.lmco.fltwinds.wxserver.wcs.ScanningCoverage.getCoverageOffering(ScanningCoverage.java:282) > at com.luciad.ogc.wcs.server.f.a(SourceFile:124) > at com.luciad.ogc.wcs.server.f.a(SourceFile:14) > at com.luciad.ogc.wcs.server.f$c.handleRequest(SourceFile:155) > at com.luciad.ogc.common.f.a(LuciadMap:41) > at com.luciad.ogc.wcs.TLcdWCSServlet.a(LuciadMap:68) > at com.luciad.ogc.wcs.TLcdWCSServlet.doGet(LuciadMap:99) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Unknown Source) > > "http-8080-199" daemon prio=10 tid=0xcbca8400 nid=0xb5d waiting on condition > [0xc5e0d000..0xc5e0de20] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xd50244e8> (a > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) > at java.util.concurrent.locks.LockSupport.park(Unknown Source) > > I can include more if required. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat hung
2010/11/18 Robillard, Greg L : > I continually get tomcat in this state and can only recover by restarting > tomcat from the command line. Here is the stack trace that I gathered while > getting into this state, but it does not make any sense to me. > > Tomcat version apache-tomcat-6.0.26 > > OS linux > > Jstack trace on tomcat hang. > Tomcat is not responding, nor any webapps. Profiler will not attach > > Attaching to process ID 21989, please wait... > Debugger attached successfully. > Server compiler detected. > JVM version is 10.0-b19 > Deadlock Detection: > > No deadlocks found. > > Thread 3343: (state = BLOCKED) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(char, > char, int, long) @bci=0, line=218 (Compiled frame) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(java.lang.Object, > char) @bci=158, line=278 (Compiled frame) > - java.lang.String.valueOf(char[]) @bci=12, line=2841 (Interpreted frame) > - java.lang.Thread.getName() @bci=4, line=1061 (Interpreted frame) > - > org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, > org.apache.coyote.Response) @bci=120, line=295 (Interpreted frame) > - org.apache.coyote.http11.Http11Processor.process(java.net.Socket) @bci=468, > line=852 (Interpreted frame) > - > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(java.net.Socket) > @bci=82, line=588 (Interpreted frame) > - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=489 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) > The above, and Thread 12244: It looks that String.valueOf() call in Thread 3343 (called by Thread.getName()) results in performing some network IO and being stuck there? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat hung
thx -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Thursday, November 18, 2010 10:23 AM To: Tomcat Users List Subject: EXTERNAL: RE: Tomcat hung > From: Robillard, Greg L [mailto:greg.l.robill...@lmco.com] > Subject: RE: Tomcat hung > Java(TM) SE Runtime Environment (build 1.6.0_05-b13) Java HotSpot(TM) > Server VM (build 10.0-b19, mixed mode) > Last I checked, this is a real VM. Yes, it was identifying it just as 10.0-b19 that was confusing. You might want to upgrade, since 1.6.0_05 is quite old and lots of fixes have gone in since then. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat hung
> From: Robillard, Greg L [mailto:greg.l.robill...@lmco.com] > Subject: RE: Tomcat hung > Java(TM) SE Runtime Environment (build 1.6.0_05-b13) > Java HotSpot(TM) Server VM (build 10.0-b19, mixed mode) > Last I checked, this is a real VM. Yes, it was identifying it just as 10.0-b19 that was confusing. You might want to upgrade, since 1.6.0_05 is quite old and lots of fixes have gone in since then. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat hung
java -version java version "1.6.0_05" Java(TM) SE Runtime Environment (build 1.6.0_05-b13) Java HotSpot(TM) Server VM (build 10.0-b19, mixed mode) Last I checked, this is a real VM. I will investigate the weather code. -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: Wednesday, November 17, 2010 3:58 PM To: Tomcat Users List Subject: EXTERNAL: RE: Tomcat hung > From: Robillard, Greg L [mailto:greg.l.robill...@lmco.com] > Subject: Tomcat hung > JVM version is 10.0-b19 No idea what JVM that is; strongly suggest you install a real one (e.g., 6u22) and see if the problem goes away. What happens if you run Tomcat directly, not under control of the IDE? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: EXTERNAL: Re: Tomcat hung
> From: Robillard, Greg L [mailto:greg.l.robill...@lmco.com] > Subject: RE: EXTERNAL: Re: Tomcat hung > "http-8080-200" daemon prio=10 tid=0xcbca9800 nid=0xb5e waiting on condition > [0xc5dbc000..0xc5dbcea0] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xd50244e8> (a > java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) > at java.util.concurrent.locks.LockSupport.park(Unknown Source) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown > Source) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown > Source) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown > Source) > at > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(Unknown > Source) > at > com.lmco.fltwinds.wxserver.wcs.ScanningCoverage.getOffering(ScanningCoverage.java:286) > at > com.lmco.fltwinds.wxserver.wcs.ScanningCoverage.getCoverageOffering(ScanningCoverage.java:282) > at com.luciad.ogc.wcs.server.f.a(SourceFile:124) > at com.luciad.ogc.wcs.server.f.a(SourceFile:14) > at com.luciad.ogc.wcs.server.f$c.handleRequest(SourceFile:155) > at com.luciad.ogc.common.f.a(LuciadMap:41) > at com.luciad.ogc.wcs.TLcdWCSServlet.a(LuciadMap:68) > at com.luciad.ogc.wcs.TLcdWCSServlet.doGet(LuciadMap:99) That's all your code, not Tomcat's. Fix your webapp. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: EXTERNAL: Re: Tomcat hung
Production, as of yet, strangely, I have not been able to create this problem testing. Here is the top of the stack without a profiler. 2010-10-04 10:35:50 Full thread dump Java HotSpot(TM) Server VM (10.0-b19 mixed mode): "Attach Listener" daemon prio=10 tid=0x08170400 nid=0x716b waiting on condition [0x..0x] java.lang.Thread.State: RUNNABLE "http-8080-200" daemon prio=10 tid=0xcbca9800 nid=0xb5e waiting on condition [0xc5dbc000..0xc5dbcea0] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd50244e8> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown Source) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(Unknown Source) at com.lmco.fltwinds.wxserver.wcs.ScanningCoverage.getOffering(ScanningCoverage.java:286) at com.lmco.fltwinds.wxserver.wcs.ScanningCoverage.getCoverageOffering(ScanningCoverage.java:282) at com.luciad.ogc.wcs.server.f.a(SourceFile:124) at com.luciad.ogc.wcs.server.f.a(SourceFile:14) at com.luciad.ogc.wcs.server.f$c.handleRequest(SourceFile:155) at com.luciad.ogc.common.f.a(LuciadMap:41) at com.luciad.ogc.wcs.TLcdWCSServlet.a(LuciadMap:68) at com.luciad.ogc.wcs.TLcdWCSServlet.doGet(LuciadMap:99) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Unknown Source) "http-8080-199" daemon prio=10 tid=0xcbca8400 nid=0xb5d waiting on condition [0xc5e0d000..0xc5e0de20] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0xd50244e8> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) at java.util.concurrent.locks.LockSupport.park(Unknown Source) I can include more if required. -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Wednesday, November 17, 2010 3:57 PM To: Tomcat Users List Subject: EXTERNAL: Re: Tomcat hung On 17/11/2010 21:50, Robillard, Greg L wrote: > I continually get tomcat in this state and can only recover by restarting > tomcat from the command line. Here is the stack trace that I gathered while > getting into this state, but it does not make any sense to me. Production or testing? Are these the only threads? What happens if you jstack without attempting to connect a profiler? p > Tomcat version apache-tomcat-6.0.26 > > OS linux > > Jstack trace on tomcat hang. > Tomcat is not responding, nor any webapps. Profiler will not attach > > Attaching to process ID 21989, please wait... > Debugger attached successfully. > Server compiler detected. > JVM version is 10.0-b19 > Deadlock Detection: > > No deadlocks found. > > Thread 15911: (state = BLOCKED) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurre > ntStackTrace(char, char, int, long) @bci=0, line=218 (Compiled frame) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAl > loc(java.lang.Object, char) @bci=158, line=278 (Compiled frame) > - > java.io.ObjectInputStream$BlockDataInputStream.(java.io.ObjectIn > putStream, java.io.InputStream) @bci=91, line=2359 (Interpreted frame) > - java.io.ObjectInputStream.(java.io.InputStream) @bci=25, > line=276 (Interpreted frame) > - sun.rmi.server.MarshalInputStream.(j
RE: Tomcat hung
> From: Robillard, Greg L [mailto:greg.l.robill...@lmco.com] > Subject: Tomcat hung > JVM version is 10.0-b19 No idea what JVM that is; strongly suggest you install a real one (e.g., 6u22) and see if the problem goes away. What happens if you run Tomcat directly, not under control of the IDE? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat hung
On 17/11/2010 21:50, Robillard, Greg L wrote: > I continually get tomcat in this state and can only recover by restarting > tomcat from the command line. Here is the stack trace that I gathered while > getting into this state, but it does not make any sense to me. Production or testing? Are these the only threads? What happens if you jstack without attempting to connect a profiler? p > Tomcat version apache-tomcat-6.0.26 > > OS linux > > Jstack trace on tomcat hang. > Tomcat is not responding, nor any webapps. Profiler will not attach > > Attaching to process ID 21989, please wait... > Debugger attached successfully. > Server compiler detected. > JVM version is 10.0-b19 > Deadlock Detection: > > No deadlocks found. > > Thread 15911: (state = BLOCKED) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(char, > char, int, long) @bci=0, line=218 (Compiled frame) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(java.lang.Object, > char) @bci=158, line=278 (Compiled frame) > - > java.io.ObjectInputStream$BlockDataInputStream.(java.io.ObjectInputStream, > java.io.InputStream) @bci=91, line=2359 (Interpreted frame) > - java.io.ObjectInputStream.(java.io.InputStream) @bci=25, line=276 > (Interpreted frame) > - sun.rmi.server.MarshalInputStream.(java.io.InputStream) @bci=2, > line=107 (Interpreted frame) > - sun.rmi.transport.ConnectionInputStream.(java.io.InputStream) @bci=2, > line=38 (Interpreted frame) > - sun.rmi.transport.StreamRemoteCall.getInputStream() @bci=32, line=115 > (Interpreted frame) > - sun.rmi.transport.Transport.serviceCall(java.rmi.server.RemoteCall) @bci=1, > line=124 (Interpreted frame) > - > sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, > boolean) @bci=217, line=541 (Compiled frame) > - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=821, > line=810 (Interpreted frame) > - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=58, > line=652 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) > @bci=59, line=885 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=907 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) > > > Thread 12244: (state = BLOCKED) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(char, > char, int, long) @bci=0, line=218 (Compiled frame) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(java.lang.Object, > char) @bci=158, line=278 (Compiled frame) > - > java.io.ObjectInputStream$BlockDataInputStream.(java.io.ObjectInputStream, > java.io.InputStream) @bci=91, line=2359 (Interpreted frame) > - java.io.ObjectInputStream.(java.io.InputStream) @bci=25, line=276 > (Interpreted frame) > - sun.rmi.server.MarshalInputStream.(java.io.InputStream) @bci=2, > line=107 (Interpreted frame) > - sun.rmi.transport.ConnectionInputStream.(java.io.InputStream) @bci=2, > line=38 (Interpreted frame) > - sun.rmi.transport.StreamRemoteCall.getInputStream() @bci=32, line=115 > (Interpreted frame) > - sun.rmi.transport.Transport.serviceCall(java.rmi.server.RemoteCall) @bci=1, > line=124 (Interpreted frame) > - > sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, > boolean) @bci=217, line=541 (Compiled frame) > - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=821, > line=810 (Interpreted frame) > - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=58, > line=652 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) > @bci=59, line=885 (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=907 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) > > > Thread 22760: (state = BLOCKED) > - java.lang.Object.wait(long) @bci=0 (Interpreted frame) > - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) > - java.util.TimerThread.mainLoop() @bci=28, line=483 (Interpreted frame) > - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) > > > Thread 3343: (state = BLOCKED) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(char, > char, int, long) @bci=0, line=218 (Compiled frame) > - > org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(java.lang.Object, > char) @bci=158, line=278 (Compiled frame) > - java.lang.String.valueOf(char[]) @bci=12, line=2841 (Interpreted frame) > - java.lang.Thread.getName() @bci=4, line=1061 (Interpreted frame) > - > org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, > org.apache.coyote.Response) @bci=120, line=295 (Interpreted frame) > - org.apache.coyote.http11.Http11Processor.process(java.net.Socket) @bci=468, > line=852 (
Tomcat hung
I continually get tomcat in this state and can only recover by restarting tomcat from the command line. Here is the stack trace that I gathered while getting into this state, but it does not make any sense to me. Tomcat version apache-tomcat-6.0.26 OS linux Jstack trace on tomcat hang. Tomcat is not responding, nor any webapps. Profiler will not attach Attaching to process ID 21989, please wait... Debugger attached successfully. Server compiler detected. JVM version is 10.0-b19 Deadlock Detection: No deadlocks found. Thread 15911: (state = BLOCKED) - org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(char, char, int, long) @bci=0, line=218 (Compiled frame) - org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(java.lang.Object, char) @bci=158, line=278 (Compiled frame) - java.io.ObjectInputStream$BlockDataInputStream.(java.io.ObjectInputStream, java.io.InputStream) @bci=91, line=2359 (Interpreted frame) - java.io.ObjectInputStream.(java.io.InputStream) @bci=25, line=276 (Interpreted frame) - sun.rmi.server.MarshalInputStream.(java.io.InputStream) @bci=2, line=107 (Interpreted frame) - sun.rmi.transport.ConnectionInputStream.(java.io.InputStream) @bci=2, line=38 (Interpreted frame) - sun.rmi.transport.StreamRemoteCall.getInputStream() @bci=32, line=115 (Interpreted frame) - sun.rmi.transport.Transport.serviceCall(java.rmi.server.RemoteCall) @bci=1, line=124 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, boolean) @bci=217, line=541 (Compiled frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=821, line=810 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=58, line=652 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=885 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=907 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 12244: (state = BLOCKED) - org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(char, char, int, long) @bci=0, line=218 (Compiled frame) - org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(java.lang.Object, char) @bci=158, line=278 (Compiled frame) - java.io.ObjectInputStream$BlockDataInputStream.(java.io.ObjectInputStream, java.io.InputStream) @bci=91, line=2359 (Interpreted frame) - java.io.ObjectInputStream.(java.io.InputStream) @bci=25, line=276 (Interpreted frame) - sun.rmi.server.MarshalInputStream.(java.io.InputStream) @bci=2, line=107 (Interpreted frame) - sun.rmi.transport.ConnectionInputStream.(java.io.InputStream) @bci=2, line=38 (Interpreted frame) - sun.rmi.transport.StreamRemoteCall.getInputStream() @bci=32, line=115 (Interpreted frame) - sun.rmi.transport.Transport.serviceCall(java.rmi.server.RemoteCall) @bci=1, line=124 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, boolean) @bci=217, line=541 (Compiled frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=821, line=810 (Interpreted frame) - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=58, line=652 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=885 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=907 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 22760: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=28, line=483 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 3343: (state = BLOCKED) - org.netbeans.lib.profiler.server.ProfilerRuntimeMemory.getAndSendCurrentStackTrace(char, char, int, long) @bci=0, line=218 (Compiled frame) - org.netbeans.lib.profiler.server.ProfilerRuntimeObjLiveness.traceObjAlloc(java.lang.Object, char) @bci=158, line=278 (Compiled frame) - java.lang.String.valueOf(char[]) @bci=12, line=2841 (Interpreted frame) - java.lang.Thread.getName() @bci=4, line=1061 (Interpreted frame) - org.apache.catalina.connector.CoyoteAdapter.service(org.apache.coyote.Request, org.apache.coyote.Response) @bci=120, line=295 (Interpreted frame) - org.apache.coyote.http11.Http11Processor.process(java.net.Socket) @bci=468, line=852 (Interpreted frame) - org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(java.net.Socket) @bci=82, line=588 (Interpreted frame) - org.apache.tomcat.util.net.JIoEndpoint$Worker.run() @bci=41, line=489 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 3255: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interprete
Re: Tomcat hung - still processing a request that has yet to finish
2010/10/12 Jason Britton : > (..) I can > kill -9 it but didn't know if there was any additional information I could > gather before stopping it. > Take a thread dump (or better three dumps in a row). You will know what Servlet / JSP page is busy and what it is doing. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat hung - still processing a request that has yet to finish
Hi all, Tomcat 6.0.29 on 64bit RHEL 5.5. This particular tomcat instance had been running fine for several days, today hung, stopped responding to requests, no interesting log messages appeared until calling shutdown.sh. Then the following appeared: (appears multiple times at end of log) ERROR 10/12/2010 10:49:49:225 AM main apache.catalina.loader.WebappClassLoader The web application [] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation. ERROR 10/12/2010 10:49:49:247 AM main apache.catalina.loader.WebappClassLoader The web application [] appears to have started a thread named [Thread-35] but has failed to stop it. This is very likely to create a memory leak. ERROR 10/12/2010 10:49:49:248 AM main apache.catalina.loader.WebappClassLoader The web application [] appears to have started a thread named [Thread-46] but has failed to stop it. This is very likely to create a memory leak. The shutdown.sh script did not stop Tomcat and the process lives on. I can kill -9 it but didn't know if there was any additional information I could gather before stopping it. Suggestions on how best to debug and figure out exactly what requests are hanging up tomcat would be greatly appreciated. Thanks a lot for any suggestions, Jason
Re: httpd/JK/Tomcat hung connections (2009 edition)
On 10.10.2009 16:45, darinpope wrote:
>
> I'm having the same issue as documented in:
>
> http://www.nabble.com/httpd-JK-Tomcat-hung-connections-td10403182.html
>
> For our situation, we haven't hit the "server reached MaxClients setting"
> issue yet, but we easily could.
Use thread dumps to see, what those threads are doing.
> We also have this same config running on a Windows cluster and that
> environment seems to clean up after itself without any issue.
>
> Does anyone see any obvious misconfigurations below?
>
> Also, in a probably not related issue, when I look at the jkmanager page, I
> see lots of garbage characters in the RR and CD columns, but only in certain
> rows.
>
> It looks something like:
>
> !%���E�� * ��{ZQ�` l��=� �j��8�U���4_~GT��V
>
> When I installed mod_jk, I tried the binary as well as compiling the source
> myself. Both options still showed the garbage characters.
Likely not related. Do the garbage characters only appear after graceful
restarts?
Regards,
Rainer
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
httpd/JK/Tomcat hung connections (2009 edition)
I'm having the same issue as documented in:
http://www.nabble.com/httpd-JK-Tomcat-hung-connections-td10403182.html
For our situation, we haven't hit the "server reached MaxClients setting"
issue yet, but we easily could.
We also have this same config running on a Windows cluster and that
environment seems to clean up after itself without any issue.
Does anyone see any obvious misconfigurations below?
Also, in a probably not related issue, when I look at the jkmanager page, I
see lots of garbage characters in the RR and CD columns, but only in certain
rows.
It looks something like:
!%���E�� * ��{ZQ�` l��=� �j��8�U���4_~GT��V
When I installed mod_jk, I tried the binary as well as compiling the source
myself. Both options still showed the garbage characters.
Thanks,
Darin Pope
4 Apaches
12 Tomcats
CentOS 5.3 x86_64 on all boxes
SELinux disabled
iptables enabled
mod_jk = 1.2.28
JDK = 1.6.0_12
Tomcat = 6.0.18
server.xml:
workers.properties:
worker.template-worker.type=ajp13
worker.template-worker.lbfactor=1
worker.template-worker.ping_timeout=1000
worker.template-worker.ping_mode=A
worker.template-worker.socket_timeout=300
worker.template-worker.socket_keepalive=1
worker.template-worker.connection_pool_timeout=600
StartServers 8
MinSpareServers5
MaxSpareServers 20
ServerLimit 1024
MaxClients 1024
MaxRequestsPerChild 4000
Server version: Apache/2.2.3
Server built: Jan 21 2009 22:00:55
Server's Module Magic Number: 20051115:3
Server loaded: APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
--
View this message in context:
http://www.nabble.com/httpd-JK-Tomcat-hung-connections-%282009-edition%29-tp25834852p25834852.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: httpd/JK/Tomcat hung connections
Rainer, Thanks for all the info. Hopefully we can get this thing whipped into shape. Brantley Rainer Jung wrote: Brantley Hobbs wrote: Rainer, Thanks for the reply! You're certainly right on the JK version. Can you be more specific on the "add some more timeouts" comment? Do you mean a higher number on connection_pool_timeout? Have a look at connect_timeout and prepost_timeout. Usually I also use reply_timeout, but this one not with a very low limit. You connection_pool_timeout looks OK, you might want to increase it for efficiency, but that's not critical. Also, should the relationship between httpd's MaxClients and the connector's maxThreads be 1:1 or what? The httpd server does do additional work besides simply front-ending Tomcat, so I'd assume that we'd want MaxClients to be at least the same as MaxThreads, and probably much more depending on how much additional work the httpd server does. Is this reasonable? I'm just trying to get a handle on the relationships. You are exactly right. Usually the extra work done by Apache is high concerning request counts (e.g. if Apache serves all the static contents), but most of the extra work is done very quickly. So in case you've got 1 Apache and 1 Tomcat, and Apache only serves additional small static content, you can keep the numbers in sync. If there is a n:1 or 1:n relationship (n>1), you might need to adjust. If Apache serves long running downloads or scripts, you might also need to give it more allowed parallelity than tomcat. MaxThreads bigger than MaxClients is mostly the case, if you have 2 Apache, 2 Tomcat and each Apache can use both Tomcats. Then you would choose MaxThreads close to 2*MaxClients, so that in case you need to shut down one of your tomcats, the other one will still be able to accept enough connections from both Apaches. Thanks, Brantley Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: httpd/JK/Tomcat hung connections
Brantley Hobbs wrote: Rainer, Thanks for the reply! You're certainly right on the JK version. Can you be more specific on the "add some more timeouts" comment? Do you mean a higher number on connection_pool_timeout? Have a look at connect_timeout and prepost_timeout. Usually I also use reply_timeout, but this one not with a very low limit. You connection_pool_timeout looks OK, you might want to increase it for efficiency, but that's not critical. Also, should the relationship between httpd's MaxClients and the connector's maxThreads be 1:1 or what? The httpd server does do additional work besides simply front-ending Tomcat, so I'd assume that we'd want MaxClients to be at least the same as MaxThreads, and probably much more depending on how much additional work the httpd server does. Is this reasonable? I'm just trying to get a handle on the relationships. You are exactly right. Usually the extra work done by Apache is high concerning request counts (e.g. if Apache serves all the static contents), but most of the extra work is done very quickly. So in case you've got 1 Apache and 1 Tomcat, and Apache only serves additional small static content, you can keep the numbers in sync. If there is a n:1 or 1:n relationship (n>1), you might need to adjust. If Apache serves long running downloads or scripts, you might also need to give it more allowed parallelity than tomcat. MaxThreads bigger than MaxClients is mostly the case, if you have 2 Apache, 2 Tomcat and each Apache can use both Tomcats. Then you would choose MaxThreads close to 2*MaxClients, so that in case you need to shut down one of your tomcats, the other one will still be able to accept enough connections from both Apaches. Thanks, Brantley Regards, Rainer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: httpd/JK/Tomcat hung connections
Rainer, Thanks for the reply! You're certainly right on the JK version. Can you be more specific on the "add some more timeouts" comment? Do you mean a higher number on connection_pool_timeout? Also, should the relationship between httpd's MaxClients and the connector's maxThreads be 1:1 or what? The httpd server does do additional work besides simply front-ending Tomcat, so I'd assume that we'd want MaxClients to be at least the same as MaxThreads, and probably much more depending on how much additional work the httpd server does. Is this reasonable? I'm just trying to get a handle on the relationships. Thanks, Brantley Rainer Jung wrote: Please update mod_jk. 1.2.6 is *very* outdated. We are now at 1.2.22 and a lot of things have improved. After upgrading, check your configuration against the reference guide in the docs, especially the pages for the worker properties and Apache directives. You might want to add some more timeouts. BTW: Maximum Apache parallelity of 150 does not really fit to a maximum tomcat parallelity of 1500 unless your tomcat does serious extra work . Regards, Rainer Brantley Hobbs wrote: All, I have a web application that appears to run just fine at low loads, but when we ramp up to high load levels, strange things start happening. The symptoms are a *ton* of apparently hung threads on the tomcat status page for my JK connector. They're in stage "S", with 0KB sent and 0KB recv and they never die. Eventually, we reach a point on httpd where we get: [error] server reached MaxClients setting, consider raising the MaxClients setting And when we reach this point, the entire httpd server stops responding. I post this to the tomcat list because this same server serves PHP under similar (or worse) loads with no problems. Here's my worker properties file: worker.lbJ2EE.balanced_workers=web1 worker.web1.type=ajp13 worker.web1.host=128.192.100.14 worker.web1.port=8009 worker.web1.lbfactor=1 worker.web1.retries=5 worker.web1.connection_pool_timeout=60 Here's my AJP connector's configuration: protocol="AJP/1.3" maxThreads="1500" backlog="300" connectionTimeout="6"/> Here's my httpd worker.c configuration StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 Tomcat 5.5.23 (Sun jvm 1.6.0-b105) Apache 2.0.52 mod_jk 1.2.6 All running on RHEL4 Any help appreciated! I don't have a great deal of Tomcat load tuning experience. Brantley Hobbs - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: httpd/JK/Tomcat hung connections
Please update mod_jk. 1.2.6 is *very* outdated. We are now at 1.2.22 and a lot of things have improved. After upgrading, check your configuration against the reference guide in the docs, especially the pages for the worker properties and Apache directives. You might want to add some more timeouts. BTW: Maximum Apache parallelity of 150 does not really fit to a maximum tomcat parallelity of 1500 unless your tomcat does serious extra work . Regards, Rainer Brantley Hobbs wrote: All, I have a web application that appears to run just fine at low loads, but when we ramp up to high load levels, strange things start happening. The symptoms are a *ton* of apparently hung threads on the tomcat status page for my JK connector. They're in stage "S", with 0KB sent and 0KB recv and they never die. Eventually, we reach a point on httpd where we get: [error] server reached MaxClients setting, consider raising the MaxClients setting And when we reach this point, the entire httpd server stops responding. I post this to the tomcat list because this same server serves PHP under similar (or worse) loads with no problems. Here's my worker properties file: worker.lbJ2EE.balanced_workers=web1 worker.web1.type=ajp13 worker.web1.host=128.192.100.14 worker.web1.port=8009 worker.web1.lbfactor=1 worker.web1.retries=5 worker.web1.connection_pool_timeout=60 Here's my AJP connector's configuration: protocol="AJP/1.3" maxThreads="1500" backlog="300" connectionTimeout="6"/> Here's my httpd worker.c configuration StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 Tomcat 5.5.23 (Sun jvm 1.6.0-b105) Apache 2.0.52 mod_jk 1.2.6 All running on RHEL4 Any help appreciated! I don't have a great deal of Tomcat load tuning experience. Brantley Hobbs - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
httpd/JK/Tomcat hung connections
All, I have a web application that appears to run just fine at low loads, but when we ramp up to high load levels, strange things start happening. The symptoms are a *ton* of apparently hung threads on the tomcat status page for my JK connector. They're in stage "S", with 0KB sent and 0KB recv and they never die. Eventually, we reach a point on httpd where we get: [error] server reached MaxClients setting, consider raising the MaxClients setting And when we reach this point, the entire httpd server stops responding. I post this to the tomcat list because this same server serves PHP under similar (or worse) loads with no problems. Here's my worker properties file: worker.lbJ2EE.balanced_workers=web1 worker.web1.type=ajp13 worker.web1.host=128.192.100.14 worker.web1.port=8009 worker.web1.lbfactor=1 worker.web1.retries=5 worker.web1.connection_pool_timeout=60 Here's my AJP connector's configuration: protocol="AJP/1.3" maxThreads="1500" backlog="300" connectionTimeout="6"/> Here's my httpd worker.c configuration StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 Tomcat 5.5.23 (Sun jvm 1.6.0-b105) Apache 2.0.52 mod_jk 1.2.6 All running on RHEL4 Any help appreciated! I don't have a great deal of Tomcat load tuning experience. Brantley Hobbs - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
