it was perhaps something to do with the cache, but i'm talking about some discussion I only vaguely remember, that involved setting the chunked flag Jörg mentioned, on an UrlConnection object, not something to do with our output.
On Nov 15, 2010, at 20:32, Jan Haderka wrote: > > afaik chunked encoding was a problem for cache not for activation. > > http://jira.magnolia-cms.com/browse/MAGNOLIA-1996 > > > On Nov 15, 2010, at 8:30 PM, Grégory Joseph wrote: > >> >> Jan, didn't we look into chunking once in the past, and had problems with it >> ? Or was it not in relation with activation ? >> Otherwise, I guess the improvement's fine, as long as it can be turned off, >> in case someone runs activation through a proxy or some odd http server ? >> >> -g >> >> On Nov 15, 2010, at 17:57, Jörg von Frantzius wrote: >> >>> Hi, >>> >>> we received a heap dump from our client, where there is a thread holding >>> 260GB of memory while trying to activating some seemingly large content: >>> at java.io.FileInputStream.readBytes([BII)I (Native Method) >>> at java.io.FileInputStream.read([B)I (FileInputStream.java:177) >>> at >>> org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J >>> (IOUtils.java:1025) >>> at >>> org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I >>> (IOUtils.java:999) >>> at >>> info.magnolia.module.exchangesimple.Transporter.transport(Ljava/net/HttpURLConnection;Linfo/magnolia/module/exchangesimple/ActivationContent;)V >>> (Transporter.java:134) >>> at >>> info.magnolia.module.exchangesimple.SimpleSyndicator.activate(Linfo/magnolia/cms/exchange/Subscriber;Linfo/magnolia/module/exchangesimple/ActivationContent;)Ljava/lang/String; >>> (SimpleSyndicator.java:173) >>> at info.magnolia.module.exchangesimple.SimpleSyndicator$2.run()V >>> (SimpleSyndicator.java:120) >>> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run()V (Unknown >>> Source) >>> at java.lang.Thread.run()V (Thread.java:662) >>> >>> <opened.gif> Accumulated Objects >>> >>> Class Name Shallow Heap Retained Heap Percentage >>> <i3.gif> >>> • java.lang.Thread @ 0x7fef5c1cb820 Thread-8694 >>> 176 268.476.352 33,87% >>> <corner.gif><i5.gif> >>> • sun.net.www.http.PosterOutputStream @ 0x7fef64353bc8 >>> 40 268.435.520 33,87% >>> <empty.gif><corner.gif><i15.gif> >>> • byte[268435456] @ 0x7fef78870000 >>> --mgnlExchange-cfc93688d385..content-disposition: form-data; >>> name="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"; >>> >>> filename="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"..content-type: >>> application/octet... >>> 268.435.480 268.435.480 33,87% >>> It seems that PosterOutputStream, being a ByteArrayOutputStream by >>> inheritance, will buffer the whole activation request in memory. In >>> addition, by looking at the Magnolia code, it seems that for a single >>> activation, this will happen once for every subscriber. So that's going to >>> put a lot of load on the GC when a single large binary is activated, or >>> even worse, when several users are simultaneously activating large >>> binaries. >>> >>> There has been a similar discussion here earlier: >>> http://www.mail-archive.com/user-l...@magnolia-cms.com/msg01809.html, where >>> Jan was wondering why a ByteArrayOutputStream was used by >>> java.net.Connection, resulting in the OOME for large activations. The >>> problem could be avoided if "chunked transfer coding" was used during the >>> activation requests from author to public servers. >>> >>> I think it would be really great if chunking was used reliably, since >>> currently system stability will become at danger with large binaries. So I >>> started digging a bit. >>> >>> There is a method java.net.HttpURLConnection.setChunkedStreamingMode(int) >>> to enable chunking, and to me it seems that this needs explicit invocation >>> in order to get chunking to happen, i.e. I don't think that chunking can be >>> enabled by means of some configuration. The method's javadoc says that the >>> request could fail if the server does not support chunking. RFC 2616, on >>> the other hand, says that 'All HTTP/1.1 applications MUST be able to >>> receive and decode the "chunked" transfer-coding'. >>> >>> So if the Magnolia code would always invoke >>> HttpURLConnection.setChunkedStreamingMode(int), this would only add the >>> requirement of a HTTP/1.1 capable server being used for the public >>> instances. I'd think that this shouldn't be a big problem? Alternatively, >>> there could be a fallback to non-chunked mode in case of failure. >>> >>> What do you think of this improvement? >>> >>> Regards, >>> Jörg >>> >>> >>> -- >>> Dipl. inf. Jörg von Frantzius, System Architect >>> Email mailto:joerg.frantz...@aperto.de >>> Phone +49 30 283921-318 >>> Fax +49 30 283921-29 >>> Aperto AG - In der Pianofabrik >>> Chausseestraße 5, D-10115 Berlin-Mitte >>> Web http://www.aperto.de >>> HRB 77049, AG Berlin Charlottenburg >>> Vorstand: Dirk Buddensiek >>> >>> >>> ---------------------------------------------------------------- >>> For list details see >>> http://www.magnolia-cms.com/home/community/mailing-lists.html >>> To unsubscribe, E-mail to: <dev-list-unsubscr...@magnolia-cms.com> >>> ---------------------------------------------------------------- >> >> >> >> ---------------------------------------------------------------- >> For list details see >> http://www.magnolia-cms.com/home/community/mailing-lists.html >> To unsubscribe, E-mail to: <dev-list-unsubscr...@magnolia-cms.com> >> ---------------------------------------------------------------- > > - > Best regards, > > Jan Haderka, PhD. > Magnolia International Ltd. > http://www.magnolia-cms.com > > http://twitter.com/magnolia_cms > http://facebook.com/Magnolia > -------------------------------------- > Magnolia® - Simple Open-Source Content Management > > > > ---------------------------------------------------------------- > For list details see > http://www.magnolia-cms.com/home/community/mailing-lists.html > To unsubscribe, E-mail to: <dev-list-unsubscr...@magnolia-cms.com> > ---------------------------------------------------------------- ---------------------------------------------------------------- For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to: <dev-list-unsubscr...@magnolia-cms.com> ----------------------------------------------------------------
