We recently upgraded from Tomcat 6.0.29 to Tomcat 8.0.14. Everything appears to be working fine, except that Tomcat is keeping a high # of threads (in TIMED_WAITING state) -- and the CPU has a high load & low idle time. We are currently running Tomcat8 on 2 internal test machines, where we also monitor their statistics. In order to monitor the availability of the HTTPS/AJP port (Apache-->Tomcat), our monitoring software opens a port to verify that this works -- but then does not follow that up with an actual request. This happens every 2 minutes. We have noticed that the high thread/load activity on Tomcat coincides with this monitoring. If we disable our monitoring, the issue does not happen. We have enabled/disabled the monitoring on both machines over several days (and there is only very minimal, sometimes non-existent) internal traffic otherwise) -- in order to verify that the monitoring is really the issue. Once these threads ramp up, they stay there or keep increasing. We had no issues running on Tomcat 6 (the thread count stayed low, low load, high idle time).
The thread backtraces for these threads look like this: ----------------------------------------------------------------------------- Thread[catalina-exec-24,5,main] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:85) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:31) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) ----------------------------------------------------------------------------- The thread count grows over time (goes up to 130-150 threads after 2 hours). Setting 'connectionTimeout' (as opposed to the default of never timing out) does seems to help "some" -- the # of threads isn't quite as bad (only 60-80 threads after 2 hours). However, the CPU Idle % is still not good -- was only 10% idle with default tomcat settings, is something like 40% idle with current settings. Also tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15) but this did not make any difference. Is there some configuration we can set to make Tomcat tolerant of this monitoring? (We have tried setting connectionTimeout & keepAliveTimeout on the Connector. And we have tried putting the Connector behind an Executor with maxIdleTime.) OR, should we modify our monitoring somehow? And if so, suggestions? * Running on Linux CentOS release 5.9 * running Apache in front of Tomcat for authentication, using mod_jk * Tomcat 8.0.14 relevant sections of tomcat/conf/server.xml: ------------------------------------------------------------------------ <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" maxThreads="250" minSpareThreads="20" maxIdleTime="60000" /> <Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" /> <Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="256" connectionTimeout="3000" keepAliveTimeout="60000" /> ---------------------------------------------------------------------------- If interested, I can provide graphing of the machine's thread count, cpu idle%, and cpu load. Any suggestions would be most welcome. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org