Jamie, On 10/16/15 3:12 PM, Jamie Jackson wrote: > On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Jamie, > > On 10/9/15 10:03 AM, Jamie Jackson wrote: >>>> Here's the stack trace dump: >>>> https://gist.github.com/jamiejackson/ca2a49d2c8afac496067 >>>> >>>> FWIW, I've been trying to come up with a reliable test case to >>>> trigger the problem, but I haven't nailed it yet. I've suspected >>>> that it's related to file (large or slow) HTTP file uploads, and >>>> that's what I was running at the time, which helps to explain the >>>> java.net.SocketInputStream.socketRead0 traces. > > Your server looks mostly idle. It also looks like it's running both > Tomcat and Jetty in the same instance. What's going on? > > >> I'm not sure yet. I'm asking around, and I'll let you know. > > About the appearance of Jetty in the stack trace dump: It's part of > FusionReactor (the JVM monitor)--it uses Jetty to serve its interface.
Okay. >>>> "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native) >>>> java.lang.Thread.State: RUNNABLE at >>>> java.net.SocketInputStream.socketRead0(Native Method) at >>>> java.net.SocketInputStream.read(SocketInputStream.java:152) at >>>> java.net.SocketInputStream.read(SocketInputStream.java:122) at >>>> [...] > org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887) >>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) >>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - >>>> locked java.io.BufferedInputStream@6e81abfc at >>>> java.io.FilterInputStream.read(FilterInputStream.java:107) > > Obviously, this thread is reading a file upload. Also these: > > "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE > "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE > > This thread is making an outgoing HTTP request, which could be hanging: > >>>> "Thread-27550" Id=27623 RUNNABLE (in native) >>>> java.lang.Thread.State: RUNNABLE at >>>> java.net.SocketInputStream.socketRead0(Native Method) at >>>> java.net.SocketInputStream.read(SocketInputStream.java:152) at >>>> java.net.SocketInputStream.read(SocketInputStream.java:122) at >>>> sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at >>>> sun.security.ssl.InputRecord.read(InputRecord.java:480) at >>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) - >>>> locked java.lang.Object@659eed19 at >>>> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891) > > Same with this one: > > "Thread-27551" Id=27624 RUNNABLE (in native) > java.lang.Thread.State: RUNNABLE > > But neither of the above threads making HTTP requests are > request-processing threads: they are independent and shouldn't block > incoming requests. > > You have 12 total request-processing threads: the ones named > "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just > waiting on a new request to come in. > > You have an AJP connector defined, but it looks like it's not being used. > > Initially, you said that you used mod_proxy from httpd. Are you > expecting to use mod_proxy_http or mod_proxy_ajp? > > >> mod_proxy_http, as far as I know, but see below... Yes, you are using mod_proxy_http. > When you try to make a connection during these incidents, do you get > any errors on the httpd side? > > >> Unfortunately, at the time, I thought this would be easy to reproduce, so I >> didn't take enough notes (including the time of the problem on my load test >> box). Here are some errors that I think were just created by the app timing >> out requests. I don't think these were associated with the failed requests, >> so take these entries (from httpd's error_log) with a grain of salt: >> https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt There's not a whole lot in there, other than the fact that something "bad" happened. > Can you post your <Connector> configuration from conf/server.xml? > Remember to remove any sensitive information that you might have in there. > > >> I think this is what you're after: > >> <Connector port="8888" protocol="HTTP/1.1" >> connectionTimeout="93000" >> redirectPort="8443" /> Your mostly-default <Connector> will default to a maximum of 200 incoming connections with 200 threads to handle them. You are only using 12, so something else must be going on. You have no obvious limits on httpd, so you are probably using the default there as well (coincidentally, also in the 200-connection range). That's a high connection timeout: 93 seconds (why 93?). Note that the connectionTimeout sets the amount of time Tomcat will wait for a client to send the request line (the "GET /foo HTTP/1.1"), not the amount of time the request is allowed to run -- like for an upload, etc. I usually lower this setting from the default of 60 seconds to more like 5 or 10 seconds. Clients shouldn't be waiting a long time between making a connection and sending a request. This timeout also applies to subsequent requests on a keep-alive connection. So if the browser opens a connection and sends 1, 2, 3 requests, Tomcat will hold that thread+connection open for 93 seconds after the last request (assuming the client doesn't terminate the connection, which it might NOT) before allowing other clients to be serviced by that thread. This is a BIO-Connector-only behavior. The NIO/NIO2 and APR connectors don't hold-up the request thread waiting for a follow-up keep-alive request from a client. -chris --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
