Thanks, Dan. This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests.
It's awkward. Everything is going normal and suddenly, many threads start waiting on the same monitor. After restart Tomcat, everything works without even make any changes in Apache (service or config files). - When the problem occurs, check the access logs for both and see if the request is making it from HTTPD to Tomcat. OK. I'll check - Also when the problem occurs, try hitting Tomcat directly while the problem is occurring. See if you can access your app when you bypass HTTPD. I've already tried this. Tomcat becomes very very very slow when it happens. - Post your HTTPD config worker.properties: worker.list=balancer worker.tomsrv01.type=ajp13 worker.tomsrv01.host=x.y.z.1 worker.tomsrv01.port=8009 worker.tomsrv01.lbfactor=1 worker.tomsrv01.connect_timeout=10000 worker.tomsrv01.prepost_timeout=10000 worker.tomsrv01.socket_timeout=1800 worker.tomsrv01.connection_pool_timeout=60 worker.tomsrv02.type=ajp13 worker.tomsrv02.host=x.y.z.2 worker.tomsrv02.port=8009 worker.tomsrv02.lbfactor=1 worker.tomsrv02.connect_timeout=10000 worker.tomsrv02.prepost_timeout=10000 worker.tomsrv02.socket_timeout=1800 worker.tomsrv02.connection_pool_timeout=60 worker.tomsrv04.type=ajp13 worker.tomsrv04.host=x.y.z.3 worker.tomsrv04.port=8009 worker.tomsrv04.lbfactor=1 worker.tomsrv04.connect_timeout=10000 worker.tomsrv04.prepost_timeout=10000 worker.tomsrv04.socket_timeout=1800 worker.tomsrv04.connection_pool_timeout=60 worker.balancer.type=lb worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04 worker.balancer.sticky_session=1 - Make sure timeouts are configured appropriately - Ok Do you have a firewall in between HTTPD & Tomcat? Is there anything that could be breaking the connection? No. Best regards. 2015-07-16 12:44 GMT-03:00 Daniel Mikusa <dmik...@pivotal.io>: > On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson <ranom...@gmail.com> > wrote: > > > Hi, > > > > Sometimes, in moments of high traffic for our patterns (170 req/sec), we > > have a lot of threads like that: > > > > "ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon > > prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition > > [0x000000004ddcb000] > > java.lang.Thread.State: TIMED_WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x0000000609a53198> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > > at > > > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) > > at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at > > > > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:745) > > > > This is not a problem. The thread is in the pool waiting for work. It's > what you'd normally see on a server that's not handling any requests. > > > > > > Thus, Tomcat hangs (all available threads stay in that state) and we have > > to restart the service. > > > > Server version: Apache Tomcat/7.0.63 > > Server built: Jun 30 2015 08:08:33 UTC > > Server number: 7.0.63.0 > > OS Name: Linux > > OS Version: 2.6.18-194.17.1.el5 > > Architecture: amd64 > > JVM Version: 1.7.0_80-b15 > > JVM Vendor: Oracle Corporation > > > > We are using tomcat-native (APR connector) and Apache (2.2.3) as load > > balancer with mod_jk. > > > > > > server.xml > > > > <?xml version='1.0' encoding='utf-8'?> > > <Server port="8005" shutdown="SHUTDOWN"> > > <Listener className="org.apache.catalina.core.AprLifecycleListener" > > SSLEngine="off" /> > > <Listener className="org.apache.catalina.core.JasperListener" /> > > <Listener > > className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> > > <Listener > > className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" > /> > > <Listener > > className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> > > <GlobalNamingResources> > > <Resource name="UserDatabase" auth="Container" > > type="org.apache.catalina.UserDatabase" > > description="User database that can be updated and saved" > > > factory="org.apache.catalina.users.MemoryUserDatabaseFactory" > > pathname="conf/tomcat-users.xml" /> > > </GlobalNamingResources> > > > > <Service name="Catalina"> > > > > <Connector port="8080" protocol="HTTP/1.1" > > connectionTimeout="20000" > > redirectPort="8443" /> > > <Connector port="8009" enableLookups="false" > connectionTimeout="60000" > > protocol="AJP/1.3" redirectPort="8443" /> > > > > <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04"> > > > > <Realm className="org.apache.catalina.realm.LockOutRealm"> > > <Realm className="org.apache.catalina.realm.UserDatabaseRealm" > > resourceName="UserDatabase" digest="SHA"/> > > </Realm> > > > > <Host name="localhost" appBase="webapps" > > unpackWARs="true" autoDeploy="true"> > > > > <Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" > > threshold="60" interruptThreadThreshold="120" /> > > </Host> > > </Engine> > > </Service> > > </Server> > > > > > > Any suggestions? > > > > This could be an issue with HTTPD talking to Tomcat. > - When the problem occurs, check the access logs for both and see if the > request is making it from HTTPD to Tomcat. > - Also when the problem occurs, try hitting Tomcat directly while the > problem is occurring. See if you can access your app when you bypass > HTTPD. > - Post your HTTPD config > - Make sure timeouts are configured appropriately - > http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html > > Do you have a firewall in between HTTPD & Tomcat? Is there anything that > could be breaking the connection? > > Dan >
