Hi Chris, I had an executor defined before and it had the exact same behavior/problem. Moved to the below model for the connector only after that.
Regards, John -----Original Message----- From: Christopher Schultz <ch...@christopherschultz.net> Sent: Monday, September 30, 2024 1:11 PM To: users@tomcat.apache.org Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads [You don't often get email from ch...@christopherschultz.net. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. John, On 9/30/24 10:53, John Williams wrote: > Sorry - didn't realize that images would be stripped off. > > The number of TIMED_WAITING threads starts at 150 or so and every hour > increases by 10. > > Below is an example of where the threads are stuck (by looking at the JVM > stack): > > https-jsse-nio2-443-exec-90 > TIMED_WAITING on > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ > 48986fd7 jdk.internal.misc.Unsafe.park(Native Method); > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252) > ; > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject. > awaitNanos(AbstractQueuedSynchronizer.java:1674); > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java > :460); > app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99); > app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33); > app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(ThreadP > oolExecutor.java:1113); > app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Threa > dPoolExecutor.java:1175); > app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thre > adPoolExecutor.java:659); > app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Ta > skThread.java:63); > java.base@17.0.12/java.lang.Thread.run(Thread.java:840); > > The connector configuration is: > > <Connector protocol="org.apache.coyote.http11.Http11Nio2Protocol" > port="443" maxThreads="1536" minSpareThreads="64" > > sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation" > scheme="https" secure="true" SSLEnabled="true" > enableLookups="false" acceptCount="12000" > connectionTimeout="60000" socket.rxBufSize="131072" socket.txBufSize="131072" > maxHttpHeaderSize="131072" maxPostSize="-1" maxKeepAliveRequests="1" > maxConnections="1536" URIEncoding="UTF-8" useSendfile="false" > tcpNoDelay="true" compression="force" > compressibleMimeType="font/woff,font/woff2,text/html,text/xml,text/plain,application/x-java-applet,application/octet-stream,application/xml,text/javascript,text/css,image/png,image/bmp,image/jpeg,image/gif,application/pdf,application/x-javascript,application/javascript,application/json,application/x-shockwave-flash,application/xhtml+xml,application/xml+xhtml" > noCompressionUserAgents="gozilla, traviata"> > <SSLHostConfig honorCipherOrder="true" > ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA" > certificateVerification="none" protocols="TLSv1.2" > sslProtocol="TLS"><Certificate certificateKeystoreFile="..." > certificateKeyAlias="..." certificateKeystorePassword="..." /></SSLHostConfig> > </Connector> You are not explicitly using an <Executor>. When you configure using <Connector> only, Tomcat will never retire the threads from the thread-pool. This is mostly historic and while it could be improved, it is better to switch to a mode modern configuration. If you use <Connector executor="..."> and: <Executor name="..." maxThreads="1500" minSpareThreads="64" /> Then you should see the number of threads fluxuate based upon the request volume. I'm not sure why you were seeing different behavior with Tomcat 10.0. IIRC this behavior has been consistent as long as <Connector> and <Executor> have been separate configuration elements. -chris > -----Original Message----- > From: Christopher Schultz <ch...@christopherschultz.net> > Sent: Monday, September 30, 2024 10:38 AM > To: users@tomcat.apache.org > Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads > > [You don't often get email from ch...@christopherschultz.net. Learn > why this is important at https://aka.ms/LearnAboutSenderIdentification > ] > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you recognize the sender and know the > content is safe. > > > John, > > On 9/30/24 10:30, John Williams wrote: >> The issue is that the number of threads in the TIMED_WAIT state keeps >> increasing over time. Starts at 150 and keeps growing by 10 every hour. >> Once it reaches close to the maxThreads setting we see connection drops. >> >> This application was working fine with Tomcat 10.0 and JDK 12. Unless >> the application was seeing load, threads would not be created in >> Tomcat's JVM. >> >> You can also see the thread stacks in this image. > > This list strips out image. Can you post text? > > Please post your <Connector> configuration taking care to remove any secrets. > > Thanks, > -chris > >> -----Original Message----- >> From: Christopher Schultz <ch...@christopherschultz.net> >> Sent: Monday, September 30, 2024 10:20 AM >> To: users@tomcat.apache.org >> Subject: Re: Tomcat 10.1.30 and Many Timed Waiting threads >> >> [You don't often get email from ch...@christopherschultz.net >> <mailto:ch...@christopherschultz.net>. Learn why this is important at >> https://aka.ms/LearnAboutSenderIdentification >> <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fak >> a%2F&data=05%7C02%7Csrinivas%40eginnovations.com%7C8aebfc9d22c74881c3 >> ea08dce17304c2%7C62c210cb17214d259b6b0c95ceb472ce%7C0%7C0%7C638633131 >> 394112582%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI >> iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=rOqji369GdpVTV%2Fm >> EVvVa%2FW5m3MFoSNLiVAPHSsjlrA%3D&reserved=0 >> .ms%2F&data=05%7C02%7Csrinivas%40eginnovations.com%7C5c93ec3de0884e78 >> 6 >> 7d708dce15d7e73%7C62c210cb17214d259b6b0c95ceb472ce%7C0%7C0%7C63863303 >> 8 >> 912339442%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI >> i >> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=6q8HQHLjceSHYLqEnLN >> 9 >> ue%2Fpa3zz5cougsajpZIsjDw%3D&reserved=0 >> LearnAboutSenderIdentification> ] >> >> CAUTION: This email originated from outside of the organization. Do >> not click links or open attachments unless you recognize the sender >> and know the content is safe. >> >> John, >> >> On 9/28/24 05:38, John Williams wrote: >> >> > >> >> > Hi Everyone, >> >> > >> >> > I am running Apache Tomcat 10.1.30 for a web application and >> notice >> ~950 timed_waiting threads. >> >> > >> >> > The stack trace for these threads is below: >> >> > >> >> > java.base@17.0.12/ >> jdk.internal.misc.Unsafe.park(Native<mailto:java.bas >> <mailto:java.base@17.0.12/ >> jdk.internal.misc.Unsafe.park(Native%3cmailto:java.bas> >> >> > e@17.0.12/jdk.internal.misc.Unsafe.park(Native <mailto:e@17.0.12/ >> jdk.internal.misc.Unsafe.park(Native>> Method); >> >> > java.base@17.0.12/ >> java.util.concurrent.locks.LockSupport.parkNanos(Loc >> <mailto:java.base@17.0.12/ >> java.util.concurrent.locks.LockSupport.parkNanos(Loc> >> >> > >> kSupport.java:252)<mailto:java.base@17.0.12/java.util.concurrent.lock >> s >> >> > .LockSupport.parkNanos(LockSupport.java:252)>; >> >> > java.base@17.0.12/ >> java.util.concurrent.locks.AbstractQueuedSynchronize >> <mailto:java.base@17.0.12/ >> java.util.concurrent.locks.AbstractQueuedSynchronize> >> >> > >> r$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674)<ma >> i >> >> > >> lto:java.base@17.0.12/java.util.concurrent.locks.AbstractQueuedSynchr >> o >> >> > >> nizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1674 >> ) >> >> > >; >> >> > java.base@17.0.12/ >> java.util.concurrent.LinkedBlockingQueue.poll(Linked >> <mailto:java.base@17.0.12/ >> java.util.concurrent.LinkedBlockingQueue.poll(Linked> >> >> > BlockingQueue.java:460)<mailto:java.base@17.0.12/java.util.concurrent. >> >> > LinkedBlockingQueue.poll(LinkedBlockingQueue.java:460)>; >> >> > >> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:99) >> ; >> >> > >> app//org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:33) >> ; >> >> > >> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.getTask(Thread >> P >> >> > oolExecutor.java:1113); >> >> > >> app//org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(Thre >> a >> >> > dPoolExecutor.java:1175); >> >> > >> app//org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(Thr >> e >> >> > adPoolExecutor.java:659); >> >> > >> app//org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(T >> a >> >> > skThread.java:63); >> >> > java.base@17.0.12/ >> java.lang.Thread.run(Thread.java:840)<mailto:java.ba >> <mailto:java.base@17.0.12/ >> java.lang.Thread.run(Thread.java:840)%3cmailto:java.ba> >> >> > se@17.0.12/java.lang.Thread.run(Thread.java:840) >> <mailto:se@17.0.12/ java.lang.Thread.run(Thread.java:840)>>; >> >> > >> >> > I am using an executor with maxThreads set to 1500 and >> minSpareThreads set to 64. >> >> > >> >> > I have not seen this issue with Tomcat 10.0. >> >> > >> >> > What could be the issue and how to resolve it? Any help is much >> appreciated. >> >> This looks like you have a large number of threads waiting for work. >> >> That seems normal to me if you have a max thread pool of 1500 threads. >> >> What's the problem? >> >> -chris >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> <mailto:users-unsubscr...@tomcat.apache.org> >> >> For additional commands, e-mail: users-h...@tomcat.apache.org >> <mailto:users-h...@tomcat.apache.org> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
