I am following up on my own thread because I found the culprit!
I turns out that the JVM was leaking native memory (that's why profiling
the heap usage did not show anything, nor tracking file descriptors).
The process virtual size (VSZ in top) was growing slowly but surely,
until the OS starts denying malloc calls, triggering the error described
below. This is why the OOM only happened in native code and never in
straight java code.
It turns out that the culprit is struts 1.2.9. It ships with
commons-fileupload 1.0, which uses File.deleteOnExit() to clean up after
itself. File.deleteOnExit() leaks native memory (as documented here:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4513817): this means
that on a struts 1.2.9 webapp with fileupload functionality we leak
native memory on each upload...
Replacing the jar with the latest version (1.1.1) fixes the problem.
- Renaud
Renaud Bruyeron wrote:
We are running into OOM errors that we think are related to this:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4705373
Our setup:
JDK 1.5.0_08b3 on Debian Linux (2.4.21SMP)
tomcat 5.5.17
We have 3 webapps in the tomcat instance with fairly similar
WEB-INF/lib content
(struts + spring + hibernate, etc) - roughly 25-30MB worth of .jar files.
We have JAVA_OPTS="-server -XX:MaxPermSize=128M
-XX:+HeapDumpOnOutOfMemoryError -Xms$1024m -Xmx1024m" and we are
running on dual-xeon, 4GB main memory hardware.
RAM is never exhausted, we have on average 3GB of free swap, and we
never actually consume all of the non-swap RAM.
This is an example of the top of the stacktraces we experience *after
a while* (i.e. after a couple of days of operation, sometimes more):
java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:203)
at java.util.jar.JarFile.<init>(JarFile.java:132)
at java.util.jar.JarFile.<init>(JarFile.java:70)
at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:56)
at
sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:41)
at
sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:68)
at
sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:102)
at
org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:175)
at
org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:423)
at
org.apache.jasper.compiler.Parser.parseDirective(Parser.java:492)
at
org.apache.jasper.compiler.Parser.parseElements(Parser.java:1552)
at org.apache.jasper.compiler.Parser.parse(Parser.java:126)
at
org.apache.jasper.compiler.ParserController.doParse(ParserController.java:211)
at
org.apache.jasper.compiler.ParserController.parse(ParserController.java:116)
at
org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:335)
at
org.apache.jasper.compiler.Parser.parseIncludeDirective(Parser.java:372)
at
org.apache.jasper.compiler.Parser.parseDirective(Parser.java:484)
at
org.apache.jasper.compiler.Parser.parseElements(Parser.java:1552)
at org.apache.jasper.compiler.Parser.parse(Parser.java:126)
at
org.apache.jasper.compiler.ParserController.doParse(ParserController.java:211)
at
org.apache.jasper.compiler.ParserController.parse(ParserController.java:116)
at
org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:335)
at
org.apache.jasper.compiler.Parser.parseIncludeDirective(Parser.java:372)
at
org.apache.jasper.compiler.Parser.parseDirective(Parser.java:484)
at
org.apache.jasper.compiler.Parser.parseElements(Parser.java:1552)
at org.apache.jasper.compiler.Parser.parse(Parser.java:126)
at
org.apache.jasper.compiler.ParserController.doParse(ParserController.java:211)
at
org.apache.jasper.compiler.ParserController.parse(ParserController.java:100)
at
org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:155)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264)
at
org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:563)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:303)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
...
Has anyone else seen this?
The answer is yes, as per
http://mail-archives.apache.org/mod_mbox/tomcat-users/200508.mbox/[EMAIL PROTECTED]
Note that the user above had a different stacktrace involving the
webapp classloader accessing resources inside jars: we *DO* experience
this kind of stacktrace as well (for example when using javamail APIs,
which tries to open a META-INF/mailcap file), and I think it is the
same underlying problem.
What is the best approach to this problem?
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6280693 suggests
that the underlying JDK bug has been fixed, however this is a JDK6 fix
only at this point.
The original bug
(http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4705373) suggests
that using
ZipInputStream instead of ZipFile would also help.
Please note that our applications do not use the java.util.zip
package: all the exceptions are thrown while inside tomcat (either the
WebappClassLoader or inside jasper).
If anyone has some info on this problem, please let me know
- Renaud
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
