On 15/09/2022 14:11, Chen Levy wrote:
Hello Experts
We’ve recently upgraded some of our production servers to Tomcat 9.0.65;
every upgraded server crashed with java.lang.OutOfMemoryError within an
hour or so under load.
The exact same setup (same application, Linux kernel, Java version etc.)
with Tomcat 9.0.63 does not exhibit this issue.
A heap-dump through MAT gave the following leak suspect (leak report
attached):
“
14,364 instances of
"org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper", loaded by
"java.net.URLClassLoader @ 0x6be257090" occupy 4,489,221,944 (91.95%) bytes.
These instances are referenced from one instance of
"java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "<system
class loader>", which occupies 590,736 (0.01%) bytes.
Keywords
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper
java.net.URLClassLoader @ 0x6be257090
java.util.concurrent.ConcurrentHashMap$Node[]
“
Please let me know if I should provide additional information
That looks like 14k current connections which isn't unreasonable for a
Tomcat instance under load.
There are connector related changes between 9.0.63 and 9.0.65 but
nothing that is obviously related to the issue you are seeing.
At this point there isn't enough information to differentiate between:
- a regression introduced in Tomcat between 9.0.63 and 9.0.65
- a change in Tomcat between 9.0.63 and 9.0.65 that exposed a bug in the
deployed web application
- a change in Tomcat between 9.0.63 and 9.0.65 that triggered an
increase memory usage sufficient to trigger an OOME in your
environment
What we would need to investigate this further is a test case that
demonstrates a leak. It doesn't have to trigger an OOME - it just has to
demonstrate the JVM retaining references to objects you'd expect to have
been eligible for GC. If you can reduce it to a single request even better.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
