About the appearance of Jetty in the stack trace dump: It's part of FusionReactor (the JVM monitor)--it uses Jetty to serve its interface.
Thanks, Jamie On Fri, Oct 16, 2015 at 3:12 PM, Jamie Jackson <jamieja...@gmail.com> wrote: > > > On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> Jamie, >> >> On 10/9/15 10:03 AM, Jamie Jackson wrote: >> > Here's the stack trace dump: >> > https://gist.github.com/jamiejackson/ca2a49d2c8afac496067 >> > >> > FWIW, I've been trying to come up with a reliable test case to >> > trigger the problem, but I haven't nailed it yet. I've suspected >> > that it's related to file (large or slow) HTTP file uploads, and >> > that's what I was running at the time, which helps to explain the >> > java.net.SocketInputStream.socketRead0 traces. >> >> Your server looks mostly idle. It also looks like it's running both >> Tomcat and Jetty in the same instance. What's going on? > > > I'm not sure yet. I'm asking around, and I'll let you know. > > >> > "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native) >> > java.lang.Thread.State: RUNNABLE at >> > java.net.SocketInputStream.socketRead0(Native Method) at >> > java.net.SocketInputStream.read(SocketInputStream.java:152) at >> > java.net.SocketInputStream.read(SocketInputStream.java:122) at >> > >> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535) >> > >> > >> at >> >> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504) >> > at >> > >> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566) >> > >> > >> at >> >> org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137) >> > at >> > >> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348) >> > >> > >> at org.apache.coyote.Request.doRead(Request.java:422) >> > at >> > >> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290) >> > >> > >> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390) >> > at >> > org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304) >> > >> > >> at >> >> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106) >> > at >> > >> com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123) >> > >> > >> at java.io.InputStream.read(InputStream.java:179) >> > at >> > >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977) >> > >> > >> at >> >> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) >> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) >> > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - >> > locked java.io.BufferedInputStream@6e81abfc at >> > java.io.FilterInputStream.read(FilterInputStream.java:107) >> >> Obviously, this thread is reading a file upload. Also these: >> >> "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native) >> java.lang.Thread.State: RUNNABLE >> "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native) >> java.lang.Thread.State: RUNNABLE >> >> This thread is making an outgoing HTTP request, which could be hanging: >> >> > "Thread-27550" Id=27623 RUNNABLE (in native) >> > java.lang.Thread.State: RUNNABLE at >> > java.net.SocketInputStream.socketRead0(Native Method) at >> > java.net.SocketInputStream.read(SocketInputStream.java:152) at >> > java.net.SocketInputStream.read(SocketInputStream.java:122) at >> > sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at >> > sun.security.ssl.InputRecord.read(InputRecord.java:480) at >> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) - >> > locked java.lang.Object@659eed19 at >> > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891) >> > >> > >> at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) >> > - locked sun.security.ssl.AppInputStream@7c98d0ab at >> > >> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) >> > >> > >> at >> >> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) >> > at >> > >> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) >> > >> > >> at >> >> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) >> > at >> > >> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) >> > >> > >> at >> >> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) >> > at >> > >> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) >> > >> > >> at >> >> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) >> > at >> > >> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) >> > >> > >> at >> >> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) >> > at >> > >> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) >> > >> > >> at >> >> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) >> > at >> > >> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) >> > >> > >> at >> >> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882) >> > at >> > >> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) >> > >> > >> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) >> > at >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > >> > >> at java.lang.reflect.Method.invoke(Method.java:606) >> > at >> > >> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416) >> > >> > >> at >> >> lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252) >> > at >> > >> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112) >> > >> > >> at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86) >> > at >> > lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108) >> > >> > >> at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58) >> >> >> Same with this one: >> >> "Thread-27551" Id=27624 RUNNABLE (in native) >> java.lang.Thread.State: RUNNABLE >> >> But neither of the above threads making HTTP requests are >> request-processing threads: they are independent and shouldn't block >> incoming requests. >> >> You have 12 total request-processing threads: the ones named >> "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just >> waiting on a new request to come in. >> >> You have an AJP connector defined, but it looks like it's not being used. >> >> Initially, you said that you used mod_proxy from httpd. Are you >> expecting to use mod_proxy_http or mod_proxy_ajp? > > > mod_proxy_http, as far as I know, but see below... > > >> It certainly looks >> like you are using the HTTP flavor of mod_proxy since your >> file-uploads are going to the HTTP connector on port 8888 instead of >> the AJP connector on port 8009. If you don't need the AJP connector, >> you might want to disable it and get a small amount of resources back. >> > > Okay, thanks. > > When you try to make a connection during these incidents, do you get >> any errors on the httpd side? > > > Unfortunately, at the time, I thought this would be easy to reproduce, so > I didn't take enough notes (including the time of the problem on my load > test box). Here are some errors that I think were just created by the app > timing out requests. I don't think these were associated with the failed > requests, so take these entries (from httpd's error_log) with a grain of > salt: > https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt > > I'll capture this information next time I trigger a crash. > > >> I know mod_jk will complain if it can't >> make a connection or if there is a timeout... I suspect mod_proxy_http >> will do the same. >> >> Can you post your <Connector> configuration from conf/server.xml? >> Remember to remove any sensitive information that you might have in there. >> > > I think this is what you're after: > > <Connector port="8888" protocol="HTTP/1.1" > connectionTimeout="93000" > redirectPort="8443" /> > > FWIW, I'm also noticing this: > > <Host name="127.0.0.1" appBase="webapps" > unpackWARs="false" autoDeploy="false"> > > <!-- Access log processes all example. > Documentation at: /docs/config/valve.html > Note: The pattern used is equivalent to using > pattern="common" --> > <Valve className="org.apache.catalina.valves.AccessLogValve" > directory="logs" > prefix="localhost_access_log." suffix=".txt" > pattern="%h %l %u %t "%r" %s %b" > resolveHosts="false"/> > > <!-- visit modcfml.org for details on mod_cfml configuration > options --> > <Valve className="mod_cfml.core" > loggingEnabled="false" > waitForContext="3" > maxContexts="200" > timeBetweenContexts="30000" > /> > </Host> > > > Also, what do your ProxyPass/ProxyPassReverse configuration directives >> look like in httpd.conf? > > > The apache config has: > > LoadModule proxy_module modules/mod_proxy.so > LoadModule proxy_balancer_module modules/mod_proxy_balancer.so > LoadModule proxy_ftp_module modules/mod_proxy_ftp.so > LoadModule proxy_http_module modules/mod_proxy_http.so > LoadModule proxy_ajp_module modules/mod_proxy_ajp.so > LoadModule proxy_connect_module modules/mod_proxy_connect.so > > And the virtual host has: > https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-virtual-host > > > >> - -chris >> -----BEGIN PGP SIGNATURE----- >> Comment: GPGTools - http://gpgtools.org >> >> iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf >> m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0 >> Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia >> PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv >> CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz >> Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg >> 3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j >> eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc >> o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf >> HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc >> AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls >> +0KkHZDkU8C3MP3MCySA >> =5Mun >> -----END PGP SIGNATURE----- >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >
