-----Original Message----- From: Mark Thomas <ma...@apache.org> Sent: Thursday, September 26, 2019 15:50 To: users@tomcat.apache.org Subject: Re: Tomcat 9.0.24/9.0.26 suspected memory leak
On 26/09/2019 18:22, Chen Levy wrote: > Hello Experts > > Several of my production servers were recently upgraded from Tomcat 9.0.14 to > 9.0.24; immediately after the upgrade the servers started accumulating memory > in a steady trend that was not observed before. In addition, CPU utilization > that used to hover around 2% not sits at 8%. > For now the servers are still serving but I suspect they'll become > unresponsive in a few hours. > I loaded a heap dump from one of the servers into MAT and received the > following Leak Suspect: > > One instance of "org.apache.coyote.http11.Http11NioProtocol" loaded by > "java.net.URLClassLoader @ 0x503f02c40" occupies 9,282,972,608 (96.88%) > bytes. The memory is accumulated in one instance of > "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class > loader>". > > The HashMap referenced in the report appears to be "waitingProcessors" inside > AbstractProtocol which contain 262K entries. OK. Those are asynchronous Servlets that are still in async mode. While it is possible for an application to deliberately get itself into a state like this (infinite async timeouts and don't complete/dispatch the async requests) given that it doesn't happen with 9.0.14 but does with 9.0.24 (and .26) that suggests a Tomcat bug. > The same issue was reproduced using v9.0.26 as well > > Please let me know whether I should provide additional information Can you do a binary search to determine which Tomcat 9.0.x release this problem was introduced in? How easily can you reproduce this? Do you have something approaching a test case we could use to repeat the issue? Meanwhile, I'll take a look at the changelog and see if anything jumps out as a possible cause. Thanks, Mark > > Current setup of the production servers: > AdoptOpenJDK (build 11.0.3+7) > Amazon Linux 2 > > <Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol" > maxHttpHeaderSize="16384" > maxThreads="500" minSpareThreads="25" > enableLookups="false" disableUploadTimeout="true" > connectionTimeout="10000" > compression="on" > SSLEnabled="true" scheme="https" secure="true"> > <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" > keepAliveTimeout="20000" > overheadDataThreadhold="0"/> > <SSLHostConfig protocols="+TLSv1.2+TLSv1.3"> > <Certificate certificateKeystoreFile="tomcat.keystore" > certificateKeyAlias="tomcat" > certificateKeystorePassword="" > certificateKeystoreType="PKCS12"/> > </SSLHostConfig> > </Connector> > > Thanks > Chen > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > Thanks for the attention Mark, here are some additional information and answers: * Once the memory was completely consumed, the servers stopped responding with CPU stuck at 100% * I do not employ async servlets in my application * I cannot do a binary search for a version because of this change: https://github.com/apache/tomcat/commit/c16d9d810a1f64cd768ff33058936cf8907e3117 which cause another memory leak and server failure between v9.0.16 and v9.0.21 and was fixed in v9.0.24 (as far as I know) * This is easily reproduced with the traffic in my farm and all the servers suffer the same. In a development environment it's more tricky; so currently I don't have a test case Thanks Chen
