-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Jamie,
On 10/9/15 10:03 AM, Jamie Jackson wrote: > Here's the stack trace dump: > https://gist.github.com/jamiejackson/ca2a49d2c8afac496067 > > FWIW, I've been trying to come up with a reliable test case to > trigger the problem, but I haven't nailed it yet. I've suspected > that it's related to file (large or slow) HTTP file uploads, and > that's what I was running at the time, which helps to explain the > java.net.SocketInputStream.socketRead0 traces. Your server looks mostly idle. It also looks like it's running both Tomcat and Jetty in the same instance. What's going on? > "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE at > java.net.SocketInputStream.socketRead0(Native Method) at > java.net.SocketInputStream.read(SocketInputStream.java:152) at > java.net.SocketInputStream.read(SocketInputStream.java:122) at > org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535) > > at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) > at > org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566) > > at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137) > at > org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348) > > at org.apache.coyote.Request.doRead(Request.java:422) > at > org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) > > at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390) > at > org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304) > > at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106) > at > com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123) > > at java.io.InputStream.read(InputStream.java:179) > at > org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) > > at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - > locked java.io.BufferedInputStream@6e81abfc at > java.io.FilterInputStream.read(FilterInputStream.java:107) Obviously, this thread is reading a file upload. Also these: "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native) java.lang.Thread.State: RUNNABLE "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native) java.lang.Thread.State: RUNNABLE This thread is making an outgoing HTTP request, which could be hanging: > "Thread-27550" Id=27623 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE at > java.net.SocketInputStream.socketRead0(Native Method) at > java.net.SocketInputStream.read(SocketInputStream.java:152) at > java.net.SocketInputStream.read(SocketInputStream.java:122) at > sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at > sun.security.ssl.InputRecord.read(InputRecord.java:480) at > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) - > locked java.lang.Object@659eed19 at > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891) > > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) > - locked sun.security.ssl.AppInputStream@7c98d0ab at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) > at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) > at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) > at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) > at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) > > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) > at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) > > at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) > > at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > at > lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416) > > at lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252) > at > lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112) > > at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86) > at > lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108) > > at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58) Same with this one: "Thread-27551" Id=27624 RUNNABLE (in native) java.lang.Thread.State: RUNNABLE But neither of the above threads making HTTP requests are request-processing threads: they are independent and shouldn't block incoming requests. You have 12 total request-processing threads: the ones named "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just waiting on a new request to come in. You have an AJP connector defined, but it looks like it's not being used. Initially, you said that you used mod_proxy from httpd. Are you expecting to use mod_proxy_http or mod_proxy_ajp? It certainly looks like you are using the HTTP flavor of mod_proxy since your file-uploads are going to the HTTP connector on port 8888 instead of the AJP connector on port 8009. If you don't need the AJP connector, you might want to disable it and get a small amount of resources back. When you try to make a connection during these incidents, do you get any errors on the httpd side? I know mod_jk will complain if it can't make a connection or if there is a timeout... I suspect mod_proxy_http will do the same. Can you post your <Connector> configuration from conf/server.xml? Remember to remove any sensitive information that you might have in there. Also, what do your ProxyPass/ProxyPassReverse configuration directives look like in httpd.conf? - -chris -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0 Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg 3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls +0KkHZDkU8C3MP3MCySA =5Mun -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
