On 16 July 2015 16:37:53 CEST, Robert Anderson <ranom...@gmail.com> wrote: >Hi, > >Sometimes, in moments of high traffic for our patterns (170 req/sec), >we >have a lot of threads like that: > >"ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute" daemon >prio=10 tid=0x00002aaab5c36800 nid=0x12f9 waiting on condition >[0x000000004ddcb000] > java.lang.Thread.State: TIMED_WAITING (parking) >at sun.misc.Unsafe.park(Native Method) >- parking to wait for <0x0000000609a53198> (a >java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) >at >java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) >at >java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) >at >java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) >at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) >at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) >at >java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) >at >java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) >at >java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >at >org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >at java.lang.Thread.run(Thread.java:745) > >Thus, Tomcat hangs (all available threads stay in that state) and we >have >to restart the service. > >Server version: Apache Tomcat/7.0.63 >Server built: Jun 30 2015 08:08:33 UTC >Server number: 7.0.63.0 >OS Name: Linux >OS Version: 2.6.18-194.17.1.el5 >Architecture: amd64 >JVM Version: 1.7.0_80-b15 >JVM Vendor: Oracle Corporation > >We are using tomcat-native (APR connector) and Apache (2.2.3) as load >balancer with mod_jk. > > >server.xml > ><?xml version='1.0' encoding='utf-8'?> ><Server port="8005" shutdown="SHUTDOWN"> > <Listener className="org.apache.catalina.core.AprLifecycleListener" >SSLEngine="off" /> > <Listener className="org.apache.catalina.core.JasperListener" /> > <Listener >className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> > <Listener >className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" >/> > <Listener >className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" >/> > <GlobalNamingResources> > <Resource name="UserDatabase" auth="Container" > type="org.apache.catalina.UserDatabase" > description="User database that can be updated and saved" > factory="org.apache.catalina.users.MemoryUserDatabaseFactory" > pathname="conf/tomcat-users.xml" /> > </GlobalNamingResources> > > <Service name="Catalina"> > > <Connector port="8080" protocol="HTTP/1.1" > connectionTimeout="20000" > redirectPort="8443" /> > <Connector port="8009" enableLookups="false" connectionTimeout="60000" >protocol="AJP/1.3" redirectPort="8443" /> > > <Engine name="Catalina" defaultHost="localhost" jvmRoute="tomsrv04"> > > <Realm className="org.apache.catalina.realm.LockOutRealm"> > <Realm className="org.apache.catalina.realm.UserDatabaseRealm" > resourceName="UserDatabase" digest="SHA"/> > </Realm> > > <Host name="localhost" appBase="webapps" > unpackWARs="true" autoDeploy="true"> > ><Valve className="org.apache.catalina.valves.StuckThreadDetectionValve" >threshold="60" interruptThreadThreshold="120" /> > </Host> > </Engine> > </Service> ></Server> > > >Any suggestions? > > >Thanks in advance. > > >Robert
Short since I'm on my phone. Connection exhaustion? Disable connection reuse or use NIO. Mark
