Between random intervals our website running with Liferay + Tomcat 6.0.14 gets
Stuck with all http-worker threads runnable, but hanging in
java.net.SocketOutputStream.socketWrite0.
I have also taken a netstat with 30s interval, and we see trend of growing
number of connections in
SYN_RECV and CLOSE_WAIT state until Tomcat is restarted.
Is anybody able to tell, whether this is a bug in Tomcat, JVM, in our
Configuration, network or have I just diagnosed the problem incorrectly?
Something to workaround this problem?
CPU utilization is close to nothing. After getting a thread dump from the
running java process, a following stacktraces are seen in all of the
http-threads:
---
Version information of used JDK and Tomcat, as reported by "catalina.sh
version"
------------------
Using JRE_HOME: /usr/java/jdk1.5.0_13/
Server version: Apache Tomcat/6.0.14
Server built: Jul 20 2007 04:17:30
Server number: 6.0.14.0
OS Name: Linux
OS Version: 2.6.18-53.1.4.el5
Architecture: amd64
JVM Version: 1.5.0_13-b05
JVM Vendor: Sun Microsystems Inc.
---
- - -
"http-9095-28" daemon prio=1 tid=0x0000000044743190 nid=0x3bc runnable
[0x000000005124a000..0x000000005124be10]
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:737)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:434)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:761)
at
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:118)
at
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:570)
at org.apache.coyote.Response.doWrite(Response.java:560)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:381)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:370)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
- locked <0x00002aaaf85f8ac0> (a java.io.BufferedOutputStream)
at
com.liferay.util.servlet.ServletResponseUtil.write(ServletResponseUtil.java:194)
at com.liferay.filters.strip.StripFilter.doFilter(StripFilter.java:249)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.liferay.filters.secure.SecureFilter.doFilter(SecureFilter.java:144)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.liferay.filters.compression.CompressionFilter.doFilter(CompressionFilter.java:122)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.liferay.filters.doubleclick.DoubleClickFilter.doFilter(DoubleClickFilter.java:135)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.liferay.portal.servlet.filters.layoutcache.LayoutCacheFilter.doFilter(LayoutCacheFilter.java:167)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.liferay.portal.servlet.filters.autologin.AutoLoginFilter.doFilter(AutoLoginFilter.java:106)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:147)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:654)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:445)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:292)
at
com.liferay.portal.servlet.filters.virtualhost.VirtualHostFilter.doFilter(VirtualHostFilter.java:197)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:584)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)
Ville Hartikainen
Senior Technical Architect
Digital Media
_________________________________________
Logica - Releasing your potential
Karvaamokuja 2
P.O. Box 38
00381 Helsinki
Finland
T: +358 (0) 10 302 010
M: +358 (0) 50 390 5963
E: [EMAIL PROTECTED]
www.logica.fi<http://www.logicacmg.fi>
