-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Lisa,
On 11/18/14 11:52 AM, Lisa Woodring wrote: > We recently upgraded from Tomcat 6.0.29 to Tomcat 8.0.14. > Everything appears to be working fine, except that Tomcat is > keeping a high # of threads (in TIMED_WAITING state) -- and the CPU > has a high load & low idle time. We are currently running Tomcat8 > on 2 internal test machines, where we also monitor their > statistics. In order to monitor the availability of the HTTPS/AJP > port (Apache-->Tomcat), our monitoring software opens a port to > verify that this works -- but then does not follow that up with an > actual request. This happens every 2 minutes. We have noticed > that the high thread/load activity on Tomcat coincides with this > monitoring. If we disable our monitoring, the issue does not > happen. We have enabled/disabled the monitoring on both machines > over several days (and there is only very minimal, sometimes > non-existent) internal traffic otherwise) -- in order to verify > that the monitoring is really the issue. Once these threads ramp > up, they stay there or keep increasing. We had no issues running > on Tomcat 6 (the thread count stayed low, low load, high idle > time). > > The thread backtraces for these threads look like this: > ----------------------------------------------------------------------------- > > Thread[catalina-exec-24,5,main] > at sun.misc.Unsafe.park(Native Method) at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85) > at > org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31) at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.lang.Thread.run(Thread.java:745) > ----------------------------------------------------------------------------- > > The thread count grows over time (goes up to 130-150 threads after 2 > hours). Setting 'connectionTimeout' (as opposed to the default of > never timing out) does seems to help "some" -- the # of threads > isn't quite as bad (only 60-80 threads after 2 hours). However, > the CPU Idle % is still not good -- was only 10% idle with default > tomcat settings, is something like 40% idle with current settings. > Also tried setting Apache's 'KeepAliveTimeout = 5' (currently set > to 15) but this did not make any difference. > > > Is there some configuration we can set to make Tomcat tolerant of > this monitoring? (We have tried setting connectionTimeout & > keepAliveTimeout on the Connector. And we have tried putting the > Connector behind an Executor with maxIdleTime.) OR, should we > modify our monitoring somehow? And if so, suggestions? > > > * Running on Linux CentOS release 5.9 * running Apache in front of > Tomcat for authentication, using mod_jk * Tomcat 8.0.14 > > relevant sections of tomcat/conf/server.xml: > ------------------------------------------------------------------------ > > <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" > maxThreads="250" minSpareThreads="20" maxIdleTime="60000" /> > > <Connector executor="tomcatThreadPool" port="8080" > protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" > /> > > <Connector executor="tomcatThreadPool" port="8009" > protocol="AJP/1.3" redirectPort="8443" maxThreads="256" > connectionTimeout="3000" keepAliveTimeout="60000" /> Both of these connectors should be NIO connectors, so they should not block while waiting for more input. That means that you should not run out of threads (which is good), but those connections will sit in the poller queue for a long time (20 seconds for HTTP, 3 seconds for AJP) and then sit in the acceptor queue for the same amount of time (to check for a "next" keepAlive request). Are you properly shutting-down the connection on the client end every 2 minutes? > ---------------------------------------------------------------------------- > > If interested, I can provide graphing of the machine's thread > count, cpu idle%, and cpu load. Any suggestions would be most > welcome. More data is always good. Remember that the list strips attachments, so maybe paste it somewhere (e.g. pastebin or whatever) and provide a link. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJUa6FiAAoJEBzwKT+lPKRY0U0P/0IlOG9MUEt23Nlmvg/EtsGf kSitwIp2wdZu7UpfZrtnyvdgXAMind1Hc+4CuOuDcnZCZxTyXmdHEQjaL50LuTZm ClwoqLIllxel6ZWABcNooCQUKmnaOjHW8n7aHfrQtzYBA0792W2sqUt3mug9ixvG 3SUgsvPec4EmyIoMoHRVsqvbj05IGg6CgD4KfzYb+szFvWxxlRwIse4ln2dsdf0j oo15uaOlnGLnn1f7FDI9llDHfUknL/Xd/KpwZJ8x6i2QJ3L14K/sSYV6FJYJlvgL 5xHKM+dzXZSmysdo+0z3qJtevDgmczv3JXIWYt6Q2FNfzix7UJtuNOdsvHMSWCvx f98SQiQNw/f8qhV1LVks7emdGsSXYZ7PE9rvMxkxQE9skQLTBH66QAInGFWWLQAJ NfSUwh8xj4KVMkrLhsosDXf8BQvFqEgkIVMSlR+bW0dM0G2Vm9Ry8rru2IcEHbwj qq3ls4e6vgudiGVjkD8eBN3naqM/C3Q4QBmGqC9gdm4JUVQuxP40gQ43qTFO0TAr WkgssYNbfQdGa5/LDZ2bLsaKryeMKtS+q4vKu/0zbAPA7fh7E79wcU5344GSMT7k UvGB1ey/3m+4QTQin+iMHOxCKnbYOxOhgFrb8ru+6W9DFhQ7P7dVWxE/wBwuItHg hx/veC0UfsHTDF8SlAM0 =zYRD -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
