Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
In my opinion we should go for it as the current behavior is problematic enough. Regarding the testing I think it is probably hard to write meaningful JUnit tests as the potential problems will most likely depend on the environments: firewall, proxies, So what I think we should do is: - make the proposed code change - make it configurable on subscriber level - test it on tomcat, WS, WL If this manual tests succeed we can roll it out with having set that value to true. Cheers - Philipp On 16.11.2010, at 12:42, Jan Haderka wrote: > Actually since the Greg's concerns about previous attempt to enable the same > I think we should get a test for the feature. You might be able to build test > upon existing integration tests. > > > On Nov 16, 2010, at 12:38 PM, Jörg von Frantzius wrote: > >> OK I created http://jira.magnolia-cms.com/browse/MAGNOLIA-3390 and referred >> to it from the Wiki. >> >> I think the actual code change will be little, unless you'd like to >> accompany it with a JUnit-test that spawns a little local HTTP server in >> order to verify whether chunking is either turned on or off depending on the >> config value, or something like that... >> >> >> On 15.11.2010 18:13, Jan Haderka wrote: >>> What do you think of this improvement? >>> +1 from me >>> >>> There is already page collecting features and changes that need to be made >>> to the activation to make it more reliable and maintainable. Could you >>> please add your suggestions to the page? >>> >>> http://wiki.magnolia-cms.com/display/DEV/Concept+Activation >>> >>> Cheers, >>> Jan >>> >>> >>> >>> >>> >>> For list details see >>> http://www.magnolia-cms.com/home/community/mailing-lists.html >>> To unsubscribe, E-mail to: >>> >>> >> >> >> -- >> Dipl. inf. Jörg von Frantzius, System Architect >> Email mailto:joerg.frantz...@aperto.de >> Phone +49 30 283921-318 >> Fax +49 30 283921-29 >> Aperto AG - In der Pianofabrik >> Chausseestraße 5, D-10115 Berlin-Mitte >> Web http://www.aperto.de >> HRB 77049, AG Berlin Charlottenburg >> Vorstand: Dirk Buddensiek >> >> >> >> For list details see >> http://www.magnolia-cms.com/home/community/mailing-lists.html >> To unsubscribe, E-mail to: >> > > > > > > For list details see > http://www.magnolia-cms.com/home/community/mailing-lists.html > To unsubscribe, E-mail to: > For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to:
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
Actually since the Greg's concerns about previous attempt to enable the same I think we should get a test for the feature. You might be able to build test upon existing integration tests. On Nov 16, 2010, at 12:38 PM, Jörg von Frantzius wrote: > OK I created http://jira.magnolia-cms.com/browse/MAGNOLIA-3390 and referred > to it from the Wiki. > > I think the actual code change will be little, unless you'd like to accompany > it with a JUnit-test that spawns a little local HTTP server in order to > verify whether chunking is either turned on or off depending on the config > value, or something like that... > > > On 15.11.2010 18:13, Jan Haderka wrote: >> >>> What do you think of this improvement? >>> >> +1 from me >> >> There is already page collecting features and changes that need to be made >> to the activation to make it more reliable and maintainable. Could you >> please add your suggestions to the page? >> >> http://wiki.magnolia-cms.com/display/DEV/Concept+Activation >> >> Cheers, >> Jan >> >> >> >> >> >> For list details see >> http://www.magnolia-cms.com/home/community/mailing-lists.html >> To unsubscribe, E-mail to: >> >> > > > -- > Dipl. inf. Jörg von Frantzius, System Architect > Email mailto:joerg.frantz...@aperto.de > Phone +49 30 283921-318 > Fax +49 30 283921-29 > Aperto AG - In der Pianofabrik > Chausseestraße 5, D-10115 Berlin-Mitte > Web http://www.aperto.de > HRB 77049, AG Berlin Charlottenburg > Vorstand: Dirk Buddensiek > > > > For list details see > http://www.magnolia-cms.com/home/community/mailing-lists.html > To unsubscribe, E-mail to: > For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to:
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
OK I created http://jira.magnolia-cms.com/browse/MAGNOLIA-3390 and referred to it from the Wiki. I think the actual code change will be little, unless you'd like to accompany it with a JUnit-test that spawns a little local HTTP server in order to verify whether chunking is either turned on or off depending on the config value, or something like that... On 15.11.2010 18:13, Jan Haderka wrote: What do you think of this improvement? +1 from me There is already page collecting features and changes that need to be made to the activation to make it more reliable and maintainable. Could you please add your suggestions to the page? http://wiki.magnolia-cms.com/display/DEV/Concept+Activation Cheers, Jan For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to: -- Dipl. inf. Jörg von Frantzius, System Architect Email mailto:joerg.frantz...@aperto.de Phone +49 30 283921-318 Fax +49 30 283921-29 Aperto AG - In der Pianofabrik Chausseestraße 5, D-10115 Berlin-Mitte Web http://www.aperto.de HRB 77049, AG Berlin Charlottenburg Vorstand: Dirk Buddensiek For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to:
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
it was perhaps something to do with the cache, but i'm talking about some discussion I only vaguely remember, that involved setting the chunked flag Jörg mentioned, on an UrlConnection object, not something to do with our output. On Nov 15, 2010, at 20:32, Jan Haderka wrote: > > afaik chunked encoding was a problem for cache not for activation. > > http://jira.magnolia-cms.com/browse/MAGNOLIA-1996 > > > On Nov 15, 2010, at 8:30 PM, Grégory Joseph wrote: > >> >> Jan, didn't we look into chunking once in the past, and had problems with it >> ? Or was it not in relation with activation ? >> Otherwise, I guess the improvement's fine, as long as it can be turned off, >> in case someone runs activation through a proxy or some odd http server ? >> >> -g >> >> On Nov 15, 2010, at 17:57, Jörg von Frantzius wrote: >> >>> Hi, >>> >>> we received a heap dump from our client, where there is a thread holding >>> 260GB of memory while trying to activating some seemingly large content: >>> at java.io.FileInputStream.readBytes([BII)I (Native Method) >>> at java.io.FileInputStream.read([B)I (FileInputStream.java:177) >>> at >>> org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J >>> (IOUtils.java:1025) >>> at >>> org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I >>> (IOUtils.java:999) >>> at >>> info.magnolia.module.exchangesimple.Transporter.transport(Ljava/net/HttpURLConnection;Linfo/magnolia/module/exchangesimple/ActivationContent;)V >>> (Transporter.java:134) >>> at >>> info.magnolia.module.exchangesimple.SimpleSyndicator.activate(Linfo/magnolia/cms/exchange/Subscriber;Linfo/magnolia/module/exchangesimple/ActivationContent;)Ljava/lang/String; >>> (SimpleSyndicator.java:173) >>> at info.magnolia.module.exchangesimple.SimpleSyndicator$2.run()V >>> (SimpleSyndicator.java:120) >>> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run()V (Unknown >>> Source) >>> at java.lang.Thread.run()V (Thread.java:662) >>> >>> Accumulated Objects >>> >>> Class Name Shallow HeapRetained Heap Percentage >>> >>> • java.lang.Thread @ 0x7fef5c1cb820 Thread-8694 >>> 176 268.476.352 33,87% >>> >>> • sun.net.www.http.PosterOutputStream @ 0x7fef64353bc8 >>> 40 268.435.520 33,87% >>> >>> • byte[268435456] @ 0x7fef7887 >>> --mgnlExchange-cfc93688d385..content-disposition: form-data; >>> name="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"; >>> >>> filename="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"..content-type: >>> application/octet... >>> 268.435.480 268.435.480 33,87% >>> It seems that PosterOutputStream, being a ByteArrayOutputStream by >>> inheritance, will buffer the whole activation request in memory. In >>> addition, by looking at the Magnolia code, it seems that for a single >>> activation, this will happen once for every subscriber. So that's going to >>> put a lot of load on the GC when a single large binary is activated, or >>> even worse, when several users are simultaneously activating large >>> binaries. >>> >>> There has been a similar discussion here earlier: >>> http://www.mail-archive.com/user-l...@magnolia-cms.com/msg01809.html, where >>> Jan was wondering why a ByteArrayOutputStream was used by >>> java.net.Connection, resulting in the OOME for large activations. The >>> problem could be avoided if "chunked transfer coding" was used during the >>> activation requests from author to public servers. >>> >>> I think it would be really great if chunking was used reliably, since >>> currently system stability will become at danger with large binaries. So I >>> started digging a bit. >>> >>> There is a method java.net.HttpURLConnection.setChunkedStreamingMode(int) >>> to enable chunking, and to me it seems that this needs explicit invocation >>> in order to get chunking to happen, i.e. I don't think that chunking can be >>> enabled by means of some configuration. The method's javadoc says that the >>> request could fail if the server does not support chunking. RFC 2616, on >>> the other hand, says that 'All HTTP/1.1 applications MUST be able to >>> receive and decode the "chunked" transfer-coding'. >>> >>> So if the Magnolia code would always invoke >>> HttpURLConnection.setChunkedStreamingMode(int), this would only add the >>> requirement of a HTTP/1.1 capable server being used for the public >>> instances. I'd think that this shouldn't be a big problem? Alternatively, >>> there could be a fallback to non-chunked mode in case of failure. >>> >>> What do you think of this improvement? >>> >>> Regards, >>> Jörg >>> >>> >>> -- >>> Dipl. inf. Jörg von Frantzius, System Architect >>> Email mailto:joerg.frantz...@aperto.de >>> Phone +49 30 283921-318 >>> Fax +49 30 283921-29 >>> Aperto AG - In der Pianofabrik >>> Chausseestraße 5, D-10115 Berlin-Mitte >>> Web http://www.aperto.de >>> HRB 77049, AG Berli
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
afaik chunked encoding was a problem for cache not for activation. http://jira.magnolia-cms.com/browse/MAGNOLIA-1996 On Nov 15, 2010, at 8:30 PM, Grégory Joseph wrote: > > Jan, didn't we look into chunking once in the past, and had problems with it > ? Or was it not in relation with activation ? > Otherwise, I guess the improvement's fine, as long as it can be turned off, > in case someone runs activation through a proxy or some odd http server ? > > -g > > On Nov 15, 2010, at 17:57, Jörg von Frantzius wrote: > >> Hi, >> >> we received a heap dump from our client, where there is a thread holding >> 260GB of memory while trying to activating some seemingly large content: >> at java.io.FileInputStream.readBytes([BII)I (Native Method) >> at java.io.FileInputStream.read([B)I (FileInputStream.java:177) >> at >> org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J >> (IOUtils.java:1025) >> at >> org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I >> (IOUtils.java:999) >> at >> info.magnolia.module.exchangesimple.Transporter.transport(Ljava/net/HttpURLConnection;Linfo/magnolia/module/exchangesimple/ActivationContent;)V >> (Transporter.java:134) >> at >> info.magnolia.module.exchangesimple.SimpleSyndicator.activate(Linfo/magnolia/cms/exchange/Subscriber;Linfo/magnolia/module/exchangesimple/ActivationContent;)Ljava/lang/String; >> (SimpleSyndicator.java:173) >> at info.magnolia.module.exchangesimple.SimpleSyndicator$2.run()V >> (SimpleSyndicator.java:120) >> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run()V (Unknown >> Source) >> at java.lang.Thread.run()V (Thread.java:662) >> >> Accumulated Objects >> >> Class Name Shallow HeapRetained Heap Percentage >> >> • java.lang.Thread @ 0x7fef5c1cb820 Thread-8694 >> 176 268.476.352 33,87% >> >> • sun.net.www.http.PosterOutputStream @ 0x7fef64353bc8 >> 40 268.435.520 33,87% >> >> • byte[268435456] @ 0x7fef7887 >> --mgnlExchange-cfc93688d385..content-disposition: form-data; >> name="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"; >> >> filename="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"..content-type: >> application/octet... >> 268.435.480 268.435.480 33,87% >> It seems that PosterOutputStream, being a ByteArrayOutputStream by >> inheritance, will buffer the whole activation request in memory. In >> addition, by looking at the Magnolia code, it seems that for a single >> activation, this will happen once for every subscriber. So that's going to >> put a lot of load on the GC when a single large binary is activated, or even >> worse, when several users are simultaneously activating large binaries. >> >> There has been a similar discussion here earlier: >> http://www.mail-archive.com/user-l...@magnolia-cms.com/msg01809.html, where >> Jan was wondering why a ByteArrayOutputStream was used by >> java.net.Connection, resulting in the OOME for large activations. The >> problem could be avoided if "chunked transfer coding" was used during the >> activation requests from author to public servers. >> >> I think it would be really great if chunking was used reliably, since >> currently system stability will become at danger with large binaries. So I >> started digging a bit. >> >> There is a method java.net.HttpURLConnection.setChunkedStreamingMode(int) to >> enable chunking, and to me it seems that this needs explicit invocation in >> order to get chunking to happen, i.e. I don't think that chunking can be >> enabled by means of some configuration. The method's javadoc says that the >> request could fail if the server does not support chunking. RFC 2616, on the >> other hand, says that 'All HTTP/1.1 applications MUST be able to receive and >> decode the "chunked" transfer-coding'. >> >> So if the Magnolia code would always invoke >> HttpURLConnection.setChunkedStreamingMode(int), this would only add the >> requirement of a HTTP/1.1 capable server being used for the public >> instances. I'd think that this shouldn't be a big problem? Alternatively, >> there could be a fallback to non-chunked mode in case of failure. >> >> What do you think of this improvement? >> >> Regards, >> Jörg >> >> >> -- >> Dipl. inf. Jörg von Frantzius, System Architect >> Email mailto:joerg.frantz...@aperto.de >> Phone +49 30 283921-318 >> Fax +49 30 283921-29 >> Aperto AG - In der Pianofabrik >> Chausseestraße 5, D-10115 Berlin-Mitte >> Web http://www.aperto.de >> HRB 77049, AG Berlin Charlottenburg >> Vorstand: Dirk Buddensiek >> >> >> >> For list details see >> http://www.magnolia-cms.com/home/community/mailing-lists.html >> To unsubscribe, E-mail to: >> > > > >
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
Jan, didn't we look into chunking once in the past, and had problems with it ? Or was it not in relation with activation ? Otherwise, I guess the improvement's fine, as long as it can be turned off, in case someone runs activation through a proxy or some odd http server ? -g On Nov 15, 2010, at 17:57, Jörg von Frantzius wrote: > Hi, > > we received a heap dump from our client, where there is a thread holding > 260GB of memory while trying to activating some seemingly large content: > at java.io.FileInputStream.readBytes([BII)I (Native Method) > at java.io.FileInputStream.read([B)I (FileInputStream.java:177) > at > org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J > (IOUtils.java:1025) > at > org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I > (IOUtils.java:999) > at > info.magnolia.module.exchangesimple.Transporter.transport(Ljava/net/HttpURLConnection;Linfo/magnolia/module/exchangesimple/ActivationContent;)V > (Transporter.java:134) > at > info.magnolia.module.exchangesimple.SimpleSyndicator.activate(Linfo/magnolia/cms/exchange/Subscriber;Linfo/magnolia/module/exchangesimple/ActivationContent;)Ljava/lang/String; > (SimpleSyndicator.java:173) > at info.magnolia.module.exchangesimple.SimpleSyndicator$2.run()V > (SimpleSyndicator.java:120) > at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run()V (Unknown > Source) > at java.lang.Thread.run()V (Thread.java:662) > > Accumulated Objects > > Class NameShallow HeapRetained Heap Percentage > > • java.lang.Thread @ 0x7fef5c1cb820 Thread-8694 > 176 268.476.352 33,87% > > • sun.net.www.http.PosterOutputStream @ 0x7fef64353bc8 > 40268.435.520 33,87% > > • byte[268435456] @ 0x7fef7887 > --mgnlExchange-cfc93688d385..content-disposition: form-data; > name="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"; > > filename="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"..content-type: > application/octet... > 268.435.480 268.435.480 33,87% > It seems that PosterOutputStream, being a ByteArrayOutputStream by > inheritance, will buffer the whole activation request in memory. In addition, > by looking at the Magnolia code, it seems that for a single activation, this > will happen once for every subscriber. So that's going to put a lot of load > on the GC when a single large binary is activated, or even worse, when > several users are simultaneously activating large binaries. > > There has been a similar discussion here earlier: > http://www.mail-archive.com/user-l...@magnolia-cms.com/msg01809.html, where > Jan was wondering why a ByteArrayOutputStream was used by > java.net.Connection, resulting in the OOME for large activations. The problem > could be avoided if "chunked transfer coding" was used during the activation > requests from author to public servers. > > I think it would be really great if chunking was used reliably, since > currently system stability will become at danger with large binaries. So I > started digging a bit. > > There is a method java.net.HttpURLConnection.setChunkedStreamingMode(int) to > enable chunking, and to me it seems that this needs explicit invocation in > order to get chunking to happen, i.e. I don't think that chunking can be > enabled by means of some configuration. The method's javadoc says that the > request could fail if the server does not support chunking. RFC 2616, on the > other hand, says that 'All HTTP/1.1 applications MUST be able to receive and > decode the "chunked" transfer-coding'. > > So if the Magnolia code would always invoke > HttpURLConnection.setChunkedStreamingMode(int), this would only add the > requirement of a HTTP/1.1 capable server being used for the public instances. > I'd think that this shouldn't be a big problem? Alternatively, there could be > a fallback to non-chunked mode in case of failure. > > What do you think of this improvement? > > Regards, > Jörg > > > -- > Dipl. inf. Jörg von Frantzius, System Architect > Email mailto:joerg.frantz...@aperto.de > Phone +49 30 283921-318 > Fax +49 30 283921-29 > Aperto AG - In der Pianofabrik > Chausseestraße 5, D-10115 Berlin-Mitte > Web http://www.aperto.de > HRB 77049, AG Berlin Charlottenburg > Vorstand: Dirk Buddensiek > > > > For list details see > http://www.magnolia-cms.com/home/community/mailing-lists.html > To unsubscribe, E-mail to: > For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to:
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
And there I thought you got Magnolia running on some super sized memory cluster :D On Nov 15, 2010, at 6:22 PM, Jörg von Frantzius wrote: > Whoops, that's 260MB, not GB, of course! > > On 15.11.2010 17:57, Jörg von Frantzius wrote: >> >> Hi, >> >> we received a heap dump from our client, where there is a thread holding >> 260GB of memory while trying to activating some seemingly large content: >> at java.io.FileInputStream.readBytes([BII)I (Native Method) >> at java.io.FileInputStream.read([B)I (FileInputStream.java:177) >> at >> org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J >> (IOUtils.java:1025) >> at >> org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I >> (IOUtils.java:999) >> at >> info.magnolia.module.exchangesimple.Transporter.transport(Ljava/net/HttpURLConnection;Linfo/magnolia/module/exchangesimple/ActivationContent;)V >> (Transporter.java:134) >> at >> info.magnolia.module.exchangesimple.SimpleSyndicator.activate(Linfo/magnolia/cms/exchange/Subscriber;Linfo/magnolia/module/exchangesimple/ActivationContent;)Ljava/lang/String; >> (SimpleSyndicator.java:173) >> at info.magnolia.module.exchangesimple.SimpleSyndicator$2.run()V >> (SimpleSyndicator.java:120) >> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run()V (Unknown >> Source) >> at java.lang.Thread.run()V (Thread.java:662) >> Accumulated Objects >> >> Class Name Shallow HeapRetained Heap Percentage >> >> java.lang.Thread @ 0x7fef5c1cb820 Thread-8694 >> 176 268.476.352 33,87% >> >> sun.net.www.http.PosterOutputStream @ 0x7fef64353bc8 >> 40 268.435.520 33,87% >> >> byte[268435456] @ 0x7fef7887 >> --mgnlExchange-cfc93688d385..content-disposition: form-data; >> name="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"; >> >> filename="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"..content-type: >> application/octet... >> 268.435.480 268.435.480 33,87% >> It seems that PosterOutputStream, being a ByteArrayOutputStream by >> inheritance, will buffer the whole activation request in memory. In >> addition, by looking at the Magnolia code, it seems that for a single >> activation, this will happen once for every subscriber. So that's going to >> put a lot of load on the GC when a single large binary is activated, or even >> worse, when several users are simultaneously activating large binaries. >> >> There has been a similar discussion here earlier: >> http://www.mail-archive.com/user-l...@magnolia-cms.com/msg01809.html, where >> Jan was wondering why a ByteArrayOutputStream was used by >> java.net.Connection, resulting in the OOME for large activations. The >> problem could be avoided if "chunked transfer coding" was used during the >> activation requests from author to public servers. >> >> I think it would be really great if chunking was used reliably, since >> currently system stability will become at danger with large binaries. So I >> started digging a bit. >> >> There is a method java.net.HttpURLConnection.setChunkedStreamingMode(int) to >> enable chunking, and to me it seems that this needs explicit invocation in >> order to get chunking to happen, i.e. I don't think that chunking can be >> enabled by means of some configuration. The method's javadoc says that the >> request could fail if the server does not support chunking. RFC 2616, on the >> other hand, says that 'All HTTP/1.1 applications MUST be able to receive and >> decode the "chunked" transfer-coding'. >> >> So if the Magnolia code would always invoke >> HttpURLConnection.setChunkedStreamingMode(int), this would only add the >> requirement of a HTTP/1.1 capable server being used for the public >> instances. I'd think that this shouldn't be a big problem? Alternatively, >> there could be a fallback to non-chunked mode in case of failure. >> >> What do you think of this improvement? >> >> Regards, >> Jörg >> >> >> -- >> Dipl. inf. Jörg von Frantzius, System Architect >> Email mailto:joerg.frantz...@aperto.de >> Phone +49 30 283921-318 >> Fax +49 30 283921-29 >> Aperto AG - In der Pianofabrik >> Chausseestraße 5, D-10115 Berlin-Mitte >> Web http://www.aperto.de >> HRB 77049, AG Berlin Charlottenburg >> Vorstand: Dirk Buddensiek >> >> >> >> For list details see >> http://www.magnolia-cms.com/home/community/mailing-lists.html >> To unsubscribe, E-mail to: >> > > > -- > Dipl. inf. Jörg von Frantzius, System Architect > Email mailto:joerg.frantz...@aperto.de > Phone +49 30 283921-318 > Fax +49 30 283921-29 > Aperto AG - In der Pianofabrik > Chausseestraße 5, D-10115 Berlin-Mitte > Web http://www.aperto.de > HRB 77049, AG Berlin Charlottenburg > Vorstand: Dirk Buddensiek > > > ---
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
Whoops, that's 260MB, not GB, of course! On 15.11.2010 17:57, Jörg von Frantzius wrote: Hi, we received a heap dump from our client, where there is a thread holding 260GB of memory while trying to activating some seemingly large content: at java.io.FileInputStream.readBytes([BII)I (Native Method) at java.io.FileInputStream.read([B)I (FileInputStream.java:177) at org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J (IOUtils.java:1025) at org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I (IOUtils.java:999) at info.magnolia.module.exchangesimple.Transporter.transport(Ljava/net/HttpURLConnection;Linfo/magnolia/module/exchangesimple/ActivationContent;)V (Transporter.java:134) at info.magnolia.module.exchangesimple.SimpleSyndicator.activate(Linfo/magnolia/cms/exchange/Subscriber;Linfo/magnolia/module/exchangesimple/ActivationContent;)Ljava/lang/String; (SimpleSyndicator.java:173) at info.magnolia.module.exchangesimple.SimpleSyndicator$2.run()V (SimpleSyndicator.java:120) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run()V (Unknown Source) at java.lang.Thread.run()V (Thread.java:662) Accumulated Objects Class Name Shallow Heap Retained Heap Percentage java.lang.Thread @ 0x7fef5c1cb820 Thread-8694 176 268.476.352 33,87% sun.net.www.http.PosterOutputStream @ 0x7fef64353bc8 40 268.435.520 33,87% byte[268435456] @ 0x7fef7887 --mgnlExchange-cfc93688d385..content-disposition: form-data; name="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"; filename="exchange_3d9b7a32-16b6-493b-a80c-77374632719f1252654589289893304.xml.gz"..content-type: application/octet... 268.435.480 268.435.480 33,87% It seems that PosterOutputStream, being a ByteArrayOutputStream by inheritance, will buffer the whole activation request in memory. In addition, by looking at the Magnolia code, it seems that for a single activation, this will happen once for every subscriber. So that's going to put a lot of load on the GC when a single large binary is activated, or even worse, when several users are simultaneously activating large binaries. There has been a similar discussion here earlier: http://www.mail-archive.com/user-l...@magnolia-cms.com/msg01809.html, where Jan was wondering why a ByteArrayOutputStream was used by java.net.Connection, resulting in the OOME for large activations. The problem could be avoided if "chunked transfer coding" was used during the activation requests from author to public servers. I think it would be really great if chunking was used reliably, since currently system stability will become at danger with large binaries. So I started digging a bit. There is a method java.net.HttpURLConnection.setChunkedStreamingMode(int) to enable chunking, and to me it seems that this needs explicit invocation in order to get chunking to happen, i.e. I don't think that chunking can be enabled by means of some configuration. The method's javadoc says that the request could fail if the server does not support chunking. RFC 2616, on the other hand, says that 'All HTTP/1.1 applications MUST be able to receive and decode the "chunked" transfer-coding'. So if the Magnolia code would always invoke HttpURLConnection.setChunkedStreamingMode(int), this would only add the requirement of a HTTP/1.1 capable server being used for the public instances. I'd think that this shouldn't be a big problem? Alternatively, there could be a fallback to non-chunked mode in case of failure. What do you think of this improvement? Regards, Jörg -- Dipl. inf. Jörg von Frantzius, System Architect Email mailto:joerg.frantz...@aperto.de Phone +49 30 283921-318 Fax +49 30 283921-29 Aperto AG - In der Pianofabrik Chausseestraße 5, D-10115 Berlin-Mitte Web http://www.aperto.de
Re: [magnolia-dev] OOME during large activation : how to get chunked transfer coding?
> > What do you think of this improvement? > +1 from me There is already page collecting features and changes that need to be made to the activation to make it more reliable and maintainable. Could you please add your suggestions to the page? http://wiki.magnolia-cms.com/display/DEV/Concept+Activation Cheers, Jan For list details see http://www.magnolia-cms.com/home/community/mailing-lists.html To unsubscribe, E-mail to:
