Prevent OOME and GC load during activation of large binaries
------------------------------------------------------------
Key: MAGNOLIA-3390
URL: http://jira.magnolia-cms.com/browse/MAGNOLIA-3390
Project: Magnolia
Issue Type: Improvement
Components: activation
Affects Versions: 4.3.8, 4.4.x, 5.0
Reporter: Jörg von Frantzius
Assignee: Philipp Bärfuss
During activation, the activated data is currently held completely in memory
for each activation request sent from author to public.
h4. The problem
When e.g. 250MB are activated on an author instance with 4 subscribers, these
250MB are allocated 4 times in a row in RAM and garbage-collected afterwards.
Even if no OutOfMemoryError occurs during this, a high load is put on the
Garbage Collector, likely forcing the VM to perform "stop the world" full
collections, leading to unresponsiveness of the author instance for editors.
Given large enough binary data or simultaneous attempts at activating it, any
maximum heap size can be exceeded.
h4. Current implementation
This seems to be due to the default behaviour of
{{java.net.URLConnection.getOutputStream()}} used by
{{info.magnolia.module.exchangesimple.Transporter}}, which returns a subclass
of {{ByteArrayOutputStream}} that caches the whole GET request in memory. This
probably happens in order to determine the content-length before actually
sending the request.
h4. Proposed solution
The solution is to use "chunked transfer coding", as defined in [RFC
2616|http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1]. This
needs to be explicitly enabled by calling
{{java.net.HttpURLConnection.setChunkedStreamingMode(int)}} prior to
{{getOutputStream()}}. I verified via debugger that doing so will result in a
{{sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream extends
FilterOutputStream}} instead of {{sun.net.www.http.PosterOutputStream extends
ByteArrayOutputStream}}.
Chunking requires the public server to be HTTP/1.1 compliant. In case HTTP/1.1
compliance poses a problem e.g. with proxied public servers or weird HTTP
servers, chunking of activation requests should be configurable. There could
e.g. be a configuration NodeData
"server/activation/subscribers/<subscribername>/useRequestChunking" with
default value "true".
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.magnolia-cms.com/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
----------------------------------------------------------------
For list details see
http://www.magnolia-cms.com/home/community/mailing-lists.html
To unsubscribe, E-mail to: <dev-list-unsubscr...@magnolia-cms.com>
----------------------------------------------------------------
