On 23/04/2019 16:12, Christopher Schultz wrote:
Olivier,
Hi Christopher,
Thanks for you answer.
On 4/23/19 05:58, Olivier Jaquemet wrote:
Hi all,
We were able to reproduce a OutOfMemory error when using AJP and
the Resources cachingAllowed=false directive. It looks like a bug
of AJP connector(s), as it does not occurs with other HTTP
connectors.
Could you confirm the behavior described below is indeed bug ? (if
you do, I'll create the issue on bugzilla)
To reproduce :
* Use latest tomcat 8.5 version (tested with Tomcat 8.5.40) * Add
an AJP connector to server.xml <Connector port="8009"
protocol="AJP/1.3" redirectPort="8443" />
nb: no compression
nb: NIO connector is in use ; no APR (see stack trace for thread name)
* Add the following directive to context.xml : <Resources
cachingAllowed="false" />
Okay. Why context.xml, by the way?
I don't even know (yet...) why this setting was added in the first place
in the environment where it was present... !
so why this file... I don't know either :)
* Create a large file in the samples webapp, for example : cd
webapps/examples dd if=/dev/zero of=large.txt bs=1k count=2000000
~2GiB static file
* Start Tomcat with a 1024 mb heap size (JAVA_OPTS="-Xms1024m
-Xmx1024m" * Configure Apache HTTPD to use mod_proxy_ajp, or mod_jk
(both will have the same issue) [1] * Start Apache HTTPD * Download
file through default HTTP connector
http://localhost:8080/examples/large.txt --> OK * Download file
through Apache/AJP http://localhost/examples/large.txt --> BUG :
OutOfMemory error occurs Exception in thread
"ajp-nio-8009-exec-10" java.lang.OutOfMemoryError: Java heap space
at
org.apache.catalina.webresources.FileResource.getContent(FileResource.
java:207)
at
org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServl
et.java:992)
at
org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:
438)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
...
That's ... interesting. What if you request the file more than once
via the default connector? I would have expected these code paths to
be relatively the same.
There are no problems at all if I request the file many times in //
using the default HTTP connector.
Can you generate a heap dump when the OOME occurs? Use the following
JVM startup options:
- -XX:+HeapDumpOnOutOfMemoryError
- -XX:HeapDumpPath=/path/to/heap-dump-file
Sure. Here are heap dumps and log file from two different run :
https://www.dropbox.com/sh/q6iqpe42fxvsvin/AAARR4uOnn-qJ4PCg8e_Ll8La?dl=0
If you generate a heap dump, are you able to navigate it? I've heard
that Eclipse MAT is good. I've mostly used YourKit and it's quite
good, able to pick-out the "big things" and help locate what kind of
object(s) is/are taking up all the space.
I've used MAT some times before, and indeed it's a really nice tool !
However, if you are familiar with YourKit, you might find the culprit
faster than me...
Can you have a look a the heap dump I shared above ?
I did have a look at it with MAT... however I could not find any
definitive conclusion ...
There is a BufferedWriter of an AccessLogValve which reference a large
char array with \0 (like the content of the large generated file...).
There are many (leaking?)
java.util.zip.ZipFile$ZipFileInflaterInputStream (however you said there
were no compression, though here they are)
...
Reading the code for FileResource.getContent, it's clear that the
entire file is being loaded into memory, which obviously isn't going
to work, here. I'm wondering why that's happening since streaming is
the correct behavior when caching=false. Also strange is that
DefaultServlet will attempt to call FileResource.getContent() -- which
returns a byte[] -- and, if that returns null, it will call
FileResource.getInputStream which ... calls this.getContent. So this
looks like a special-case for FileResource just trying to implement
that interface in the simplest way possible.
FileResource seems to implement in-memory caching whether it's enabled
or not.
I can't understand why this doesn't fail for the other kind of
connector. Everything else is the same? You have two separate
connectors in one instance, or are you changing the connector between
tests?
Everything is exactly the same as I have only one instance with two
separate connectors (AJP+HTTP).
One last (confusing) information in form of an exception, that I could
*not* reproduce our test environments, but that I saw on the server
where the symptom first occurred.
I don't think this is related to the OOM, but it might be another
symptom of the same resource configuration.
java.lang.ArrayIndexOutOfBoundsException: Unable to return
[/path/to/file/being/downloaded.ext] as a byte array since the resource
is [2,637,615,704] bytes in size which is larger than the maximum size
of a byte array
at
org.apache.catalina.webresources.FileResource.getContent(FileResource.java:196)
at
org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:1000)
at
org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:438)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
[...]
at
org.apache.catalina.servlets.DefaultServlet.service(DefaultServlet.java:418)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:486)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Olivier
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
