> -----Original Message-----
> From: Mark Thomas <[email protected]>
> Sent: Monday, September 19, 2022 13:02
> To: [email protected]
> Subject: Re: Tomcat 9.0.65 suspected memory leak
>
> On 15/09/2022 14:11, Chen Levy wrote:
> > Hello Experts
> >
> > We’ve recently upgraded some of our production servers to Tomcat
> > 9.0.65; every upgraded server crashed with java.lang.OutOfMemoryError
> > within an hour or so under load.
> >
> > The exact same setup (same application, Linux kernel, Java version
> > etc.) with Tomcat 9.0.63 does not exhibit this issue.
> >
> > A heap-dump through MAT gave the following leak suspect (leak report
> > attached):
> >
> > “
> >
> > 14,364 instances of
> > "org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper", loaded by
> > "java.net.URLClassLoader @ 0x6be257090" occupy 4,489,221,944 (91.95%)
> bytes.
> >
> > These instances are referenced from one instance of
> > "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "<system
> > class loader>", which occupies 590,736 (0.01%) bytes.
> >
> > Keywords
> >
> > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper
> >
> > java.net.URLClassLoader @ 0x6be257090
> >
> > java.util.concurrent.ConcurrentHashMap$Node[]
> >
> > “
> >
> > Please let me know if I should provide additional information
>
> That looks like 14k current connections which isn't unreasonable for a
> Tomcat instance under load.
>
> There are connector related changes between 9.0.63 and 9.0.65 but nothing
> that is obviously related to the issue you are seeing.
>
> At this point there isn't enough information to differentiate between:
> - a regression introduced in Tomcat between 9.0.63 and 9.0.65
> - a change in Tomcat between 9.0.63 and 9.0.65 that exposed a bug in the
> deployed web application
> - a change in Tomcat between 9.0.63 and 9.0.65 that triggered an
> increase memory usage sufficient to trigger an OOME in your
> environment
>
> What we would need to investigate this further is a test case that
> demonstrates a leak. It doesn't have to trigger an OOME - it just has to
> demonstrate the JVM retaining references to objects you'd expect to have
> been eligible for GC. If you can reduce it to a single request even better.
>
> Mark
Mark, I believe a change in Tomcat 9.0.65 causes it to accumulate open
connections:
I took a fresh Tomcat, unzipped and modified server.xml with only the following:
1. Changed port 8080 to port 80
2. Changed port 8443 to port 443
3. Uncommented the nio connector and added the snippet
<UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
<SSLHostConfig>
<Certificate certificateKeystoreFile="conf/tomcat_noroot.p12"
certificateKeyAlias="..."
certificateKeystorePassword="..."
certificateKeystoreType="PKCS12"/>
</SSLHostConfig>
I used Chrome to call the default index.html with Wireshark in the middle:
With 9.0.63 - 20 seconds after the last data frame, came a GOAWAY from the
server.
With 9.0.65 - No GOAWAY was sent, and the server and client kept ACKing each
other.
Tomcat 9.0.71 and 10.1.5 behaved similarly - no GOAWAY was sent.
Test was conducted with:
Wireshark Version 4.0.3 (v4.0.3-0-gc552f74cdc23)
Chrome Version 109.0.5414.120
JDK 17.0.6+10
Windows 11
Chen
