On Wed, Nov 19, 2014 at 1:20 PM, Lisa Woodring <lisa.woodr...@iglass.net> wrote: > On Tue, Nov 18, 2014 at 2:26 PM, André Warnier <a...@ice-sa.com> wrote: >> Lisa Woodring wrote: >> ... >>> In order to monitor >>> the availability of the HTTPS/AJP port (Apache-->Tomcat), our >>> monitoring software opens a port to verify that this works -- but then >>> does not follow that up with an actual request. This happens every 2 >>> minutes. >> ... >> >> This sounds like the perfect recipe for simulating a DOS attack. Your >> monitoring system is forcing Tomcat to allocate a thread to process the >> request which should subsequently arrive on that connection, yet that >> request never comes; so basically this thread is wasted, until the >> ConnectionTimeout triggers (after 20 seconds, according to your HTTP >> connector settings). >> >> ... >>> >>> The thread count grows over time (goes up to 130-150 threads after 2 >>> hours). Setting 'connectionTimeout' (as opposed to the default of >>> never timing out) does seems to help "some" >> >> >> Have you tried setting it shorter ? 20000 = 20000 ms = 20 seconds. That is >> still quite long if you think about a legitimate browser/application making >> a connection, and then sending a request on that connection. Why would it >> wait so long ? A browser would never do that : it would open a connection to >> the server when it needs to send a request, and then send the request >> immediately, as soon as the connection is established. >> >> In other words : anything which opens a HTTP connection to your server, and >> then waits more than 1 or 2 seconds before sending a request on that >> connection, is certainly not a browser. >> And it probably is either a program designed to test or attack your server, >> or else a badly-designed monitoring system.. ;-) >> > > > The monitoring software is going thru Apache to AJP connector in > Tomcat. As I described, with the default of no timeout, the # of > threads were much higher. I currently have the AJP connectionTimeout > set to 3 seconds.
Actually, I received a little clarification on the monitoring software (I didn't write it). What it's trying to test is that the AJP port itself is actually accepting connections. With Apache in front in a production system, it could forward the actual request to one of several Tomcat boxes -- but we don't know which one from the outside. The monitoring software is trying to test -- for each Tomcat instance -- if it is accepting connections. It used to send an "nmap" request, but now sends essentially a "tcp ping" -- gets a response & moves on. My main questions are: 1) Why was this ok on Tomcat 6? but now an issue with Tomcat 8? 2) Suggestions on how to monitor this better? > > > >> >> -- the # of threads isn't >>> >>> quite as bad (only 60-80 threads after 2 hours). However, the CPU >>> Idle % is still not good -- was only 10% idle with default tomcat >>> settings, is something like 40% idle with current settings. Also >>> tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15) >>> but this did not make any difference. >> >> >> Note : this value is in milliseconds. setting it to 5 or 15 is almost >> equivalent to disabling keep-alive altogether. 3000 may be a reasonable >> value. > > > No, Apache's configuration is in seconds. > > >> >> KeepAlive only happens after at least one request has been received and >> processed, waiting for another (possible) request on the same connection. >> If there is never any request sent on that connection, then it would not >> apply here, and only the connectionTimeout would apply. >> >> Note that my comments above are relative to your HTTP Connector. >> For the AJP Connector, other circumstances apply. >> >> If you are using AJP, it implies that there is a front-end server, using a >> module such as mod_jk or mod_proxy_ajp to connect to Tomcat's AJP Connector. >> In that case, you should probably leave Tomcat's connectionTimeout to its >> default value, and let the front-end server handle such things as the >> connection timeout and the keep-alive timeout. The connector module on the >> front-end server will manage these connections to Tomcat, and it may >> pre-allocate some connections, to constitute a pool of available connections >> for when it actually does need to send a request to Tomcat over one such >> connection. Timing out these connections at the Tomcat level may thus be >> contra-productive, forcing the front-end to re-create them constantly. >> >>> > > > Yes, as I stated, Apache is running in front of Tomcat using mod_jk. > My big question is why is this now an issue? This monitoring software > has been running for years now. It has only been an issue since we > upgraded to Tomcat 8. > > I also forgot to mention that we are using APR. > > > >>> >>> Is there some configuration we can set to make Tomcat tolerant of this >>> monitoring? (We have tried setting connectionTimeout & >>> keepAliveTimeout on the Connector. And we have tried putting the >>> Connector behind an Executor with maxIdleTime.) >>> OR, should we modify our monitoring somehow? And if so, suggestions? >>> >> >> I would think so. Have your monitoring send an actual request to Tomcat >> (and read the response); even a request that results in an error would >> probably be better than no request at all. But better would be to request >> something real but small, which at the Tomcat level would be efficient to >> respond to (e.g. not a 5 MB image file). >> Create a little webapp which just responds "I'm fine" (*), and check that >> response in your monitor. It will tell you not only that Tomcat has opened >> the port, but also that Tomcat webapps are actually working (and how quickly >> it answers). >> And do not try to monitor the AJP port directly. Monitor a request to the >> front-end, which should arrive to Tomcat via the AJP port. The AJP >> connector module on the front-end will respond with its own error, if it >> cannot talk to Tomcat. >> >> (*) actually, there may even exist some built-in mechanism in Tomcat, >> designed precisely for such kind of usage (or at least usable for it). >> Any of the experts on the list ? does the standard vanilla Tomcat offer some >> URL which can be called, and triggers some small efficient response readable >> by a monitoring program ? >> >> >> >> >> >> ... >>> >>> >>> * Running on Linux CentOS release 5.9 >>> * running Apache in front of Tomcat for authentication, using mod_jk >>> * Tomcat 8.0.14 >>> >>> relevant sections of tomcat/conf/server.xml: >>> ------------------------------------------------------------------------ >>> <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" >>> maxThreads="250" minSpareThreads="20" maxIdleTime="60000" >>> /> >>> >>> <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1" >>> connectionTimeout="20000" redirectPort="8443" /> >>> >>> <Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3" >>> redirectPort="8443" maxThreads="256" >>> connectionTimeout="3000" keepAliveTimeout="60000" /> >>> >>> ---------------------------------------------------------------------------- >>> >>> If interested, I can provide graphing of the machine's thread count, >>> cpu idle%, and cpu load. >>> Any suggestions would be most welcome. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
