On Tue, Nov 18, 2014 at 2:26 PM, André Warnier <a...@ice-sa.com> wrote: > Lisa Woodring wrote: > ... >> In order to monitor >> the availability of the HTTPS/AJP port (Apache-->Tomcat), our >> monitoring software opens a port to verify that this works -- but then >> does not follow that up with an actual request. This happens every 2 >> minutes. > ... > > This sounds like the perfect recipe for simulating a DOS attack. Your > monitoring system is forcing Tomcat to allocate a thread to process the > request which should subsequently arrive on that connection, yet that > request never comes; so basically this thread is wasted, until the > ConnectionTimeout triggers (after 20 seconds, according to your HTTP > connector settings). > > ... >> >> The thread count grows over time (goes up to 130-150 threads after 2 >> hours). Setting 'connectionTimeout' (as opposed to the default of >> never timing out) does seems to help "some" > > > Have you tried setting it shorter ? 20000 = 20000 ms = 20 seconds. That is > still quite long if you think about a legitimate browser/application making > a connection, and then sending a request on that connection. Why would it > wait so long ? A browser would never do that : it would open a connection to > the server when it needs to send a request, and then send the request > immediately, as soon as the connection is established. > > In other words : anything which opens a HTTP connection to your server, and > then waits more than 1 or 2 seconds before sending a request on that > connection, is certainly not a browser. > And it probably is either a program designed to test or attack your server, > or else a badly-designed monitoring system.. ;-) >
The monitoring software is going thru Apache to AJP connector in Tomcat. As I described, with the default of no timeout, the # of threads were much higher. I currently have the AJP connectionTimeout set to 3 seconds. > > -- the # of threads isn't >> >> quite as bad (only 60-80 threads after 2 hours). However, the CPU >> Idle % is still not good -- was only 10% idle with default tomcat >> settings, is something like 40% idle with current settings. Also >> tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15) >> but this did not make any difference. > > > Note : this value is in milliseconds. setting it to 5 or 15 is almost > equivalent to disabling keep-alive altogether. 3000 may be a reasonable > value. No, Apache's configuration is in seconds. > > KeepAlive only happens after at least one request has been received and > processed, waiting for another (possible) request on the same connection. > If there is never any request sent on that connection, then it would not > apply here, and only the connectionTimeout would apply. > > Note that my comments above are relative to your HTTP Connector. > For the AJP Connector, other circumstances apply. > > If you are using AJP, it implies that there is a front-end server, using a > module such as mod_jk or mod_proxy_ajp to connect to Tomcat's AJP Connector. > In that case, you should probably leave Tomcat's connectionTimeout to its > default value, and let the front-end server handle such things as the > connection timeout and the keep-alive timeout. The connector module on the > front-end server will manage these connections to Tomcat, and it may > pre-allocate some connections, to constitute a pool of available connections > for when it actually does need to send a request to Tomcat over one such > connection. Timing out these connections at the Tomcat level may thus be > contra-productive, forcing the front-end to re-create them constantly. > >> Yes, as I stated, Apache is running in front of Tomcat using mod_jk. My big question is why is this now an issue? This monitoring software has been running for years now. It has only been an issue since we upgraded to Tomcat 8. I also forgot to mention that we are using APR. >> >> Is there some configuration we can set to make Tomcat tolerant of this >> monitoring? (We have tried setting connectionTimeout & >> keepAliveTimeout on the Connector. And we have tried putting the >> Connector behind an Executor with maxIdleTime.) >> OR, should we modify our monitoring somehow? And if so, suggestions? >> > > I would think so. Have your monitoring send an actual request to Tomcat > (and read the response); even a request that results in an error would > probably be better than no request at all. But better would be to request > something real but small, which at the Tomcat level would be efficient to > respond to (e.g. not a 5 MB image file). > Create a little webapp which just responds "I'm fine" (*), and check that > response in your monitor. It will tell you not only that Tomcat has opened > the port, but also that Tomcat webapps are actually working (and how quickly > it answers). > And do not try to monitor the AJP port directly. Monitor a request to the > front-end, which should arrive to Tomcat via the AJP port. The AJP > connector module on the front-end will respond with its own error, if it > cannot talk to Tomcat. > > (*) actually, there may even exist some built-in mechanism in Tomcat, > designed precisely for such kind of usage (or at least usable for it). > Any of the experts on the list ? does the standard vanilla Tomcat offer some > URL which can be called, and triggers some small efficient response readable > by a monitoring program ? > > > > > > ... >> >> >> * Running on Linux CentOS release 5.9 >> * running Apache in front of Tomcat for authentication, using mod_jk >> * Tomcat 8.0.14 >> >> relevant sections of tomcat/conf/server.xml: >> ------------------------------------------------------------------------ >> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" >> maxThreads="250" minSpareThreads="20" maxIdleTime="60000" >> /> >> >> <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1" >> connectionTimeout="20000" redirectPort="8443" /> >> >> <Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3" >> redirectPort="8443" maxThreads="256" >> connectionTimeout="3000" keepAliveTimeout="60000" /> >> >> ---------------------------------------------------------------------------- >> >> If interested, I can provide graphing of the machine's thread count, >> cpu idle%, and cpu load. >> Any suggestions would be most welcome. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
