JPEG2000 - RE: Fop Memory Use

2011-05-23 Thread Marquart, Joshua D
While somewhat unrelated, I figure this is the best thread in which to
note that there is a known memory leak within JAI when writing handling
JPEG2000 images that a third party has fixed in March and documented it
here:

 

http://www.jpedal.org/PDFblog/2011/03/java-jai-image-io-jpeg2000-memory-
leak-fix/

http://www.jpedal.org/PDFblog/2011/04/jai-memory-leak-source-changes/

 

They have been unable to commit their change to java-net since the move
to Oracle.

 

-Josh

 

 

From: Eric Douglas [mailto:edoug...@blockhouse.com] 
Sent: Wednesday, May 18, 2011 1:43 PM
To: fop-dev@xmlgraphics.apache.org
Subject: RE: Fop Memory Use

 

This test run isn't using SVG at all.  The PDFRenderer is working, the
PNGRenderer runs out of memory, so it is using images but as output.

I already broke up the input to multiple FOs with multiple calls to the
transform to generate a large document by combining small documents
using the initial-page-number.

As the program runs it just keeps increasing memory use.

I tried running a profile with Java's VisualVM though I'm not sure what
exactly I'm looking at or what to do with it.

The number one item showing memory hog in the profiler, as of my last
snapshot was: class int[], live bytes 23,273,952 B, live objects 382,
generations 10

After the program crashed with the profiler running I had an additional
file opened, Java2DRenderer.class, so I'm assuming it's doing something
that breaks PNGRenderer.

My class doesn't have any int[] references.

After that first reference the sizes drop off sharply in the profiler.
The next class reference is char[], then
org.apache.fop.area.inline.SpaceArea.

 

 



From: Peter Hancock [mailto:peter.hanc...@gmail.com] 
Sent: Wednesday, May 18, 2011 12:05 PM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use

Hi Eric,

Does your document contain many large SVG's?  If so take a look at
Bugzilla #46360.  This issue was resolved in rev 997602 of FOP trunk.

Pete





On Wed, May 18, 2011 at 5:10 PM, Adrian Cumiskey
adrian.cumis...@gmail.com wrote:

Hi Eric,

 

Fop calculates layout in page sequence chunks, so try breaking up your
pages into chunks of page sequences.  Pages should be available for
garbage collection once the page sequence has been rendered.


Cheers, Adrian.


On May 18, 2011, at 7:24 AM, Michael Rubin mru...@thunderhead.com
wrote:

Just a wild thought. But is there a way you could possibly get
the JVM to garbage collect between each run? Maybe that might free the
memory up?

Thanks.

-Mike

On 18/05/11 13:20, Eric Douglas wrote: 

I am using Fop 1.0. 
I tried using Fop to transform a single document.  When
I got a little over 100 pages my FO file was over 5 MB.  The transform
crashed with a Java heap out of memory error.

I managed to break the input down, as I'm using embedded
code generating the input programmatically, and the PDF output is a lot
smaller.

So I'm currently transforming 10 pages at a time,
setting the initial-page-number to the next sequence (1, 11, 21, etc).

Then I save all the generated PDFs in memory and merge
them using pdfbox.  So far this is working great. 

I tried to do the same thing with the PNGRenderer, just
calling a method to transform 10 pages at a time and save the output
images in an array.

The PNGRenderer is created locally in the method.  It
should be getting released when the method ends but the java process
never releases any memory.

I tested a 90 page report and the memory use was over 1
GB.  I tested on another machine where the memory limit is apparently
lower and it crashed on page 24.

Everything about the method to render to PNG is the same
as the method to render to PDF aside from the Renderer. 
Is there a problem with this renderer or something I
could need to do different? 

 

 


Michael Rubin

Developer

Thunderhead Logo

Error! Filename not specified.

Error! Filename not specified.

 T

 F

 M

 E

 W

+44 20 8238 7400 

+44 20 8238 7401 

 

mru...@thunderhead.com 

www.thunderhead.com http://www.thunderhead.com 

 

Thunderhead featured in The Sunday Times Profit Track 100 league table
of companies with fastest-growing profits. Click here
http://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf  to read more.


Error! Filename not specified.
http://www.linkedin.com/companies/25033/Thunderhead Error! Filename
not specified. http://twitter.com/Thunderheadon Error! Filename not
specified. http://www.thunderhead.com/rss/rss.php Error! Filename not
specified. http://www.youtube.com/user/ThunderheadOn Error! Filename
not specified. http://thunderheadinnovate.wordpress.com/ Error!
Filename not specified. http://thunderhead.com/about/careers.php 

The contents of this e-mail

RE: Fop Memory Use

2011-05-20 Thread Eric Douglas
Appears to be solved.
 
I believe I've found where the memory went.
The PNGRenderer needs a lot of memory to transform large documents.
If I'm reading this right, from the results of JVisualVM, the memory use is the 
pixel size.
The PNGRender stores every page as an image, which can be retrieved with 
getPageImage(pageNumber).
That image translates to 72 pixels per inch.
For an 11 x 8.5 document that's 11 (792) x 8.5 (612) = 484704.
It appears the images contain those int[] arrays and use up that amount 
(484704) x 4 + 16 bytes ( = 1938832)
I'm saving those images for print preview and loading them into a GUI window 
with a zoom.
For the zoom I'm just resizing the image with Java's Graphics2D.drawImage.
If I redraw that to a larger size it gets a bit blurry.  To help reduce that I 
was sizing the initial image.
Fop generates a larger image if I set the methods on the FOUserAgent 
(setSourceResolution(72), setTargetResolution(144)).
Plug that into the same calculation and it quadruples the memory use ((792 * 2 
= 1584) * (612 * 2 = 1224)) * 4 + 16 = 7755280 bytes per page.
For a 100 page report that's a ton.
I am generating my own input (it's a custom report writer) so I know exactly 
what fits on a page.  I'm writing everything with hard page breaks and absolute 
positioning.
I already had a memory problem just trying to create a PDF when I got around 
150 pages so I solved that by breaking it up.
I now transform all output 10 pages at a time, creating multiple PDFs (in 
memory) with xsl's initial-page-number then use pdfbox to put the pages 
together.
The FO actually wasn't using much memory, just something Fop was doing when I 
tried to use that FO to generate a parge PDF all at once.
So, I just save my FO files in an array and generate the PNG page images no 
more than 10 at a time and I've limited memory use!
I'm using the PNG to split the process so I can generate output on one machine 
(server) and display it on another (client).
If I only need to transform part of the document, and it's not taking much more 
than the Graphics2D redraw, I can just call the render again for new page 
requests so no more fuzzy images.



From: Georg Datterl [mailto:georg.datt...@geneon.de] 
Sent: Thursday, May 19, 2011 12:22 AM
To: fop-dev@xmlgraphics.apache.org
Subject: AW: Fop Memory Use



Hi Eric,

 

That sounds interesting. If you run the transformer for each page and set a 
breakpoint after the first run, there (IMHO) should not be a reference to any 
fop object. Ignore the int[]s first, they are used everywhere. Concentrate on 
the fop objects which should not be there. You could as well run your 
transformation X times and then investigate all objects which exists exactly X 
(or Y*X) times in memory. Those  are probably accumulated over many runs and 
crash your application sooner or later.

 

Regards,

 

Georg Datterl

 

-- Kontakt --

 

Georg Datterl

 

Geneon media solutions gmbh

Gutenstetter Straße 8a

90449 Nürnberg

 

HRB Nürnberg: 17193

Geschäftsführer: Yong-Harry Steiert 

 

Tel.: 0911/36 78 88 - 26

Fax: 0911/36 78 88 - 20

 

www.geneon.de

 

Weitere Mitglieder der Willmy MediaGroup:

 

IRS Integrated Realization Services GmbH:www.irs-nbg.de 
http://www.irs-nbg.de  

Willmy PrintMedia GmbH:  www.willmy.de 
http://www.willmy.de 

Willmy Consult  Content GmbH:   www.willmycc.de 
http://www.willmycc.de  

 

Von: Eric Douglas [mailto:edoug...@blockhouse.com] 
Gesendet: Mittwoch, 18. Mai 2011 19:43
An: fop-dev@xmlgraphics.apache.org
Betreff: RE: Fop Memory Use

 

This test run isn't using SVG at all.  The PDFRenderer is working, the 
PNGRenderer runs out of memory, so it is using images but as output.

I already broke up the input to multiple FOs with multiple calls to the 
transform to generate a large document by combining small documents using the 
initial-page-number.

As the program runs it just keeps increasing memory use.

I tried running a profile with Java's VisualVM though I'm not sure what exactly 
I'm looking at or what to do with it.

The number one item showing memory hog in the profiler, as of my last snapshot 
was: class int[], live bytes 23,273,952 B, live objects 382, generations 10

After the program crashed with the profiler running I had an additional file 
opened, Java2DRenderer.class, so I'm assuming it's doing something that breaks 
PNGRenderer.

My class doesn't have any int[] references.

After that first reference the sizes drop off sharply in the profiler.  The 
next class reference is char[], then org.apache.fop.area.inline.SpaceArea.

 

 



From: Peter Hancock [mailto:peter.hanc...@gmail.com] 
Sent: Wednesday, May 18, 2011 12:05 PM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use

Hi Eric,

Does your document contain many large SVG's?  If so take a look at  Bugzilla 
#46360.  This issue was resolved in rev 997602 of FOP trunk.

Pete

Re: Fop Memory Use

2011-05-20 Thread Andreas L. Delmelle
On 20 May 2011, at 20:44, Eric Douglas wrote:

Hi Eric

 snip /
 The FO actually wasn't using much memory, just something Fop was doing when I 
 tried to use that FO to generate a parge PDF all at once.
 So, I just save my FO files in an array and generate the PNG page images no 
 more than 10 at a time and I've limited memory use!

Good news!
Should you feel a need to share some of your experience on FOP's Wiki(*), by 
all means... ;-)

(*) http://wiki.apache.org/xmlgraphics-fop/SuccessStories

Thanks!

Andreas


Fop Memory Use

2011-05-18 Thread Eric Douglas
I am using Fop 1.0.
I tried using Fop to transform a single document.  When I got a little
over 100 pages my FO file was over 5 MB.  The transform crashed with a
Java heap out of memory error.
I managed to break the input down, as I'm using embedded code generating
the input programmatically, and the PDF output is a lot smaller.
So I'm currently transforming 10 pages at a time, setting the
initial-page-number to the next sequence (1, 11, 21, etc).
Then I save all the generated PDFs in memory and merge them using
pdfbox.  So far this is working great.

I tried to do the same thing with the PNGRenderer, just calling a method
to transform 10 pages at a time and save the output images in an array.
The PNGRenderer is created locally in the method.  It should be getting
released when the method ends but the java process never releases any
memory.
I tested a 90 page report and the memory use was over 1 GB.  I tested on
another machine where the memory limit is apparently lower and it
crashed on page 24.
Everything about the method to render to PNG is the same as the method
to render to PDF aside from the Renderer.
Is there a problem with this renderer or something I could need to do
different?


Re: Fop Memory Use

2011-05-18 Thread Michael Rubin

Just a wild thought. But is there a way you could possibly get the JVM to 
garbage collect between each run? Maybe that might free the memory up?

Thanks.

-Mike

On 18/05/11 13:20, Eric Douglas wrote:

I am using Fop 1.0.
I tried using Fop to transform a single document.  When I got a little over 100 
pages my FO file was over 5 MB.  The transform crashed with a Java heap out of 
memory error.

I managed to break the input down, as I'm using embedded code generating the 
input programmatically, and the PDF output is a lot smaller.

So I'm currently transforming 10 pages at a time, setting the 
initial-page-number to the next sequence (1, 11, 21, etc).

Then I save all the generated PDFs in memory and merge them using pdfbox.  So 
far this is working great.

I tried to do the same thing with the PNGRenderer, just calling a method to 
transform 10 pages at a time and save the output images in an array.

The PNGRenderer is created locally in the method.  It should be getting 
released when the method ends but the java process never releases any memory.

I tested a 90 page report and the memory use was over 1 GB.  I tested on 
another machine where the memory limit is apparently lower and it crashed on 
page 24.

Everything about the method to render to PNG is the same as the method to 
render to PDF aside from the Renderer.
Is there a problem with this renderer or something I could need to do different?




Michael Rubin


Developer


[http://thunderhead.com/email_signature/images/Thunderhead-logo.png]
[http://thunderhead.com/email_signature/images/make-every-communication-count.png]
  [http://thunderhead.com/email_signature/images/triangles.png]

T

F

M

E

W


+44 20 8238 7400

+44 20 8238 7401



mru...@thunderhead.commailto:mru...@thunderhead.com

www.thunderhead.comhttp://www.thunderhead.com



Thunderhead featured in The Sunday Times Profit Track 100 league table of companies 
with fastest-growing profits. Click 
herehttp://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf to read more.


[http://thunderhead.com/email_signature/images/linkedin.png]http://www.linkedin.com/companies/25033/Thunderhead
 [http://thunderhead.com/email_signature/images/twitter.png] http://twitter.com/Thunderheadon 
[http://thunderhead.com/email_signature/images/rss.png] http://www.thunderhead.com/rss/rss.php 
[http://thunderhead.com/email_signature/images/youtube.png] http://www.youtube.com/user/ThunderheadOn 
[http://thunderhead.com/email_signature/images/theblog.png] http://thunderheadinnovate.wordpress.com/  
[http://thunderhead.com/email_signature/images/werehiring.png] http://thunderhead.com/about/careers.php

The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it.






RE: Fop Memory Use

2011-05-18 Thread Eric Douglas
I don't know when it does it's automatic garbage collection if that's
the only issue.
I tried adding the statement to tell it to garbage collect right after
calling the method to render to PNG, though I believe I heard that
doesn't actually do garbage collection right when that executes.  It
didn't appear to do anything.
I'm running Java 6 Update 24.
 
I tested it straight in Eclipse and even tried debug mode so it takes
longer and watched it create a javaw.exe process in task manager which
just kept going up on memory use.
No matter how slowly I went through the program in debug mode the memory
never went down so I expect that means it didn't garbage collect or the
PNGRenderer is preventing some garbage from getting collected.
The Renderer itself should be created and destroyed in the method that
transforms.
The transform method looks a lot like the PDFRenderer method which works
fine.
Why could I not create and save more than a couple dozen pages in PNG
when I just tested a 1986 page report to PDF?
When I render it to preview I am also rendering it to PDF so I can
generate a print preview screen and be able to send to PDF/printer
without rerunning it, but when I run a test program to generate 91 page
output from Eclipse and watch the java process it creates in the task
manager, memory use just generating a PDF got up to 294 MB while the
preview got over 1 GB.
 



From: Michael Rubin [mailto:mru...@thunderhead.com] 
Sent: Wednesday, May 18, 2011 8:24 AM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use



Just a wild thought. But is there a way you could possibly get the JVM
to garbage collect between each run? Maybe that might free the memory
up?

Thanks.

-Mike

On 18/05/11 13:20, Eric Douglas wrote: 

I am using Fop 1.0. 
I tried using Fop to transform a single document.  When I got a
little over 100 pages my FO file was over 5 MB.  The transform crashed
with a Java heap out of memory error.

I managed to break the input down, as I'm using embedded code
generating the input programmatically, and the PDF output is a lot
smaller.

So I'm currently transforming 10 pages at a time, setting the
initial-page-number to the next sequence (1, 11, 21, etc).

Then I save all the generated PDFs in memory and merge them
using pdfbox.  So far this is working great. 

I tried to do the same thing with the PNGRenderer, just calling
a method to transform 10 pages at a time and save the output images in
an array.

The PNGRenderer is created locally in the method.  It should be
getting released when the method ends but the java process never
releases any memory.

I tested a 90 page report and the memory use was over 1 GB.  I
tested on another machine where the memory limit is apparently lower and
it crashed on page 24.

Everything about the method to render to PNG is the same as the
method to render to PDF aside from the Renderer. 
Is there a problem with this renderer or something I could need
to do different? 




Michael Rubin

Developer

 Thunderhead
Logohttp://thunderhead.com/email_signature/images/Thunderhead-logo.png
Taglinehttp://thunderhead.com/email_signature/images/make-every-communi
cation-count.png
Triangleshttp://thunderhead.com/email_signature/images/triangles.png  
 T

 F

 M

 E

 W

+44 20 8238 7400 

+44 20 8238 7401 

 

mru...@thunderhead.com 

www.thunderhead.com http://www.thunderhead.com   

Thunderhead featured in The Sunday Times Profit Track 100 league table
of companies with fastest-growing profits. Click here
http://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf  to read more.


LinkedIn http://www.linkedin.com/companies/25033/Thunderhead  twitter
http://twitter.com/Thunderheadon RSS
http://www.thunderhead.com/rss/rss.php YouTube
http://www.youtube.com/user/ThunderheadOn
http://thunderheadinnovate.wordpress.com/  were-hiring
http://thunderhead.com/about/careers.php  
The contents of this e-mail are intended for the named addressee only.
It contains information that may be confidential. Unless you are the
named addressee or an authorized designee, you may not copy or use it,
or disclose it to anyone else. If you received it in error please notify
us immediately and then destroy it.



RE: Fop Memory Use

2011-05-18 Thread Eric Douglas
If there's no obvious answer as to what Fop might be doing wrong or what I 
might be doing wrong that would be my next guess, except I've never done that 
so I'd have to figure out how.



From: Georg Datterl [mailto:georg.datt...@geneon.de] 
Sent: Wednesday, May 18, 2011 10:25 AM
To: fop-dev@xmlgraphics.apache.org
Subject: AW: Fop Memory Use



Hi Eric,

 

I'd run a debugger (available with modern JDKs). Set a breakpoint after the 
first run and memory profiling should tell you, which objects are still in 
memory and why.

 

Regards,

 

Georg Datterl

 

-- Kontakt --

 

Georg Datterl

 

Geneon media solutions gmbh

Gutenstetter Straße 8a

90449 Nürnberg

 

HRB Nürnberg: 17193

Geschäftsführer: Yong-Harry Steiert 

 

Tel.: 0911/36 78 88 - 26

Fax: 0911/36 78 88 - 20

 

www.geneon.de

 

Weitere Mitglieder der Willmy MediaGroup:

 

IRS Integrated Realization Services GmbH:www.irs-nbg.de 
http://www.irs-nbg.de  

Willmy PrintMedia GmbH:  www.willmy.de 
http://www.willmy.de 

Willmy Consult  Content GmbH:   www.willmycc.de 
http://www.willmycc.de  

 

Von: Eric Douglas [mailto:edoug...@blockhouse.com] 
Gesendet: Mittwoch, 18. Mai 2011 16:12
An: fop-dev@xmlgraphics.apache.org
Betreff: RE: Fop Memory Use

 

When I tested over 130 pages the PNG render crashed with this dump.

 

Exception in thread main java.lang.OutOfMemoryError: Java heap space
 at java.lang.AbstractStringBuilder.init(AbstractStringBuilder.java:45)
 at java.lang.StringBuilder.init(StringBuilder.java:92)
 at org.apache.fop.area.inline.SpaceArea.init(SpaceArea.java:43)
 at org.apache.fop.area.inline.TextArea.addSpace(TextArea.java:82)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.addSpaces(TextLayoutManager.java:578)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.setText(TextLayoutManager.java:497)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.build(TextLayoutManager.java:442)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.access$1(TextLayoutManager.java:435)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager.addAreaInfoAreas(TextLayoutManager.java:369)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager.addAreas(TextLayoutManager.java:297)
 at 
org.apache.fop.layoutmgr.inline.InlineLayoutManager.addAreas(InlineLayoutManager.java:479)
 at 
org.apache.fop.layoutmgr.inline.LineLayoutManager.addInlineArea(LineLayoutManager.java:1561)
 at 
org.apache.fop.layoutmgr.inline.LineLayoutManager.addAreas(LineLayoutManager.java:1430)
 at 
org.apache.fop.layoutmgr.BlockLayoutManager.addAreas(BlockLayoutManager.java:389)
 at 
org.apache.fop.layoutmgr.AreaAdditionUtil.addAreas(AreaAdditionUtil.java:121)
 at 
org.apache.fop.layoutmgr.BlockContainerLayoutManager$BlockContainerBreaker.addAreas(BlockContainerLayoutManager.java:939)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:626)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:497)
 at 
org.apache.fop.layoutmgr.BlockContainerLayoutManager$BlockContainerBreaker.addContainedAreas(BlockContainerLayoutManager.java:965)
 at 
org.apache.fop.layoutmgr.BlockContainerLayoutManager.addAreas(BlockContainerLayoutManager.java:1158)
 at 
org.apache.fop.layoutmgr.BlockLayoutManager.addAreas(BlockLayoutManager.java:389)
 at 
org.apache.fop.layoutmgr.AreaAdditionUtil.addAreas(AreaAdditionUtil.java:121)
 at 
org.apache.fop.layoutmgr.FlowLayoutManager.addAreas(FlowLayoutManager.java:342)
 at org.apache.fop.layoutmgr.PageBreaker.addAreas(PageBreaker.java:280)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:626)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:497)
 at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:308)
 at org.apache.fop.layoutmgr.AbstractBreaker.doLayout(AbstractBreaker.java:450)
 at org.apache.fop.layoutmgr.PageBreaker.doLayout(PageBreaker.java:85)
 at 
org.apache.fop.layoutmgr.PageSequenceLayoutManager.activateLayout(PageSequenceLayoutManager.java:107)
 at 
org.apache.fop.area.AreaTreeHandler.endPageSequence(AreaTreeHandler.java:238)
 at org.apache.fop.fo.pagination.PageSequence.endOfNode(PageSequence.java:120) 



 



AW: Fop Memory Use

2011-05-18 Thread Georg Datterl
Hi Eric,

On a windows system it's quite straight-forward. Run your application in your 
IDE. Stop at breakpoint. Run VisualVM. You will get a list of running java 
processes, including your application. A few clicks will get you a memory 
profile and there you can see a tree structure, where each object in memory 
references it's parent. There's probably one single root object you have to 
free to solve your memory issues.

Regards,

Georg Datterl

-- Kontakt --

Georg Datterl

Geneon media solutions gmbh
Gutenstetter Straße 8a
90449 Nürnberg

HRB Nürnberg: 17193
Geschäftsführer: Yong-Harry Steiert

Tel.: 0911/36 78 88 - 26
Fax: 0911/36 78 88 - 20

www.geneon.dehttp://www.geneon.de

Weitere Mitglieder der Willmy MediaGroup:

IRS Integrated Realization Services GmbH:
www.irs-nbg.dehttp://www.irs-nbg.de
Willmy PrintMedia GmbH:  www.willmy.dehttp://www.willmy.de
Willmy Consult  Content GmbH:   
www.willmycc.dehttp://www.willmycc.de

Von: Eric Douglas [mailto:edoug...@blockhouse.com]
Gesendet: Mittwoch, 18. Mai 2011 16:43
An: fop-dev@xmlgraphics.apache.org
Betreff: RE: Fop Memory Use

If there's no obvious answer as to what Fop might be doing wrong or what I 
might be doing wrong that would be my next guess, except I've never done that 
so I'd have to figure out how.


From: Georg Datterl [mailto:georg.datt...@geneon.de]
Sent: Wednesday, May 18, 2011 10:25 AM
To: fop-dev@xmlgraphics.apache.org
Subject: AW: Fop Memory Use
Hi Eric,

I'd run a debugger (available with modern JDKs). Set a breakpoint after the 
first run and memory profiling should tell you, which objects are still in 
memory and why.

Regards,

Georg Datterl

-- Kontakt --

Georg Datterl

Geneon media solutions gmbh
Gutenstetter Straße 8a
90449 Nürnberg

HRB Nürnberg: 17193
Geschäftsführer: Yong-Harry Steiert

Tel.: 0911/36 78 88 - 26
Fax: 0911/36 78 88 - 20

www.geneon.dehttp://www.geneon.de

Weitere Mitglieder der Willmy MediaGroup:

IRS Integrated Realization Services GmbH:
www.irs-nbg.dehttp://www.irs-nbg.de
Willmy PrintMedia GmbH:  www.willmy.dehttp://www.willmy.de
Willmy Consult  Content GmbH:   
www.willmycc.dehttp://www.willmycc.de

Von: Eric Douglas [mailto:edoug...@blockhouse.com]
Gesendet: Mittwoch, 18. Mai 2011 16:12
An: fop-dev@xmlgraphics.apache.org
Betreff: RE: Fop Memory Use

When I tested over 130 pages the PNG render crashed with this dump.

Exception in thread main java.lang.OutOfMemoryError: Java heap space
 at java.lang.AbstractStringBuilder.init(AbstractStringBuilder.java:45)
 at java.lang.StringBuilder.init(StringBuilder.java:92)
 at org.apache.fop.area.inline.SpaceArea.init(SpaceArea.java:43)
 at org.apache.fop.area.inline.TextArea.addSpace(TextArea.java:82)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.addSpaces(TextLayoutManager.java:578)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.setText(TextLayoutManager.java:497)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.build(TextLayoutManager.java:442)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager$TextAreaBuilder.access$1(TextLayoutManager.java:435)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager.addAreaInfoAreas(TextLayoutManager.java:369)
 at 
org.apache.fop.layoutmgr.inline.TextLayoutManager.addAreas(TextLayoutManager.java:297)
 at 
org.apache.fop.layoutmgr.inline.InlineLayoutManager.addAreas(InlineLayoutManager.java:479)
 at 
org.apache.fop.layoutmgr.inline.LineLayoutManager.addInlineArea(LineLayoutManager.java:1561)
 at 
org.apache.fop.layoutmgr.inline.LineLayoutManager.addAreas(LineLayoutManager.java:1430)
 at 
org.apache.fop.layoutmgr.BlockLayoutManager.addAreas(BlockLayoutManager.java:389)
 at 
org.apache.fop.layoutmgr.AreaAdditionUtil.addAreas(AreaAdditionUtil.java:121)
 at 
org.apache.fop.layoutmgr.BlockContainerLayoutManager$BlockContainerBreaker.addAreas(BlockContainerLayoutManager.java:939)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:626)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:497)
 at 
org.apache.fop.layoutmgr.BlockContainerLayoutManager$BlockContainerBreaker.addContainedAreas(BlockContainerLayoutManager.java:965)
 at 
org.apache.fop.layoutmgr.BlockContainerLayoutManager.addAreas(BlockContainerLayoutManager.java:1158)
 at 
org.apache.fop.layoutmgr.BlockLayoutManager.addAreas(BlockLayoutManager.java:389)
 at 
org.apache.fop.layoutmgr.AreaAdditionUtil.addAreas(AreaAdditionUtil.java:121)
 at 
org.apache.fop.layoutmgr.FlowLayoutManager.addAreas(FlowLayoutManager.java:342)
 at org.apache.fop.layoutmgr.PageBreaker.addAreas(PageBreaker.java:280)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:626)
 at org.apache.fop.layoutmgr.AbstractBreaker.addAreas(AbstractBreaker.java:497)
 at org.apache.fop.layoutmgr.PageBreaker.doPhase3(PageBreaker.java:308

Re: Fop Memory Use

2011-05-18 Thread Adrian Cumiskey
Hi Eric,

Fop calculates layout in page sequence chunks, so try breaking up your pages 
into chunks of page sequences.  Pages should be available for garbage 
collection once the page sequence has been rendered.

Cheers, Adrian.

On May 18, 2011, at 7:24 AM, Michael Rubin mru...@thunderhead.com wrote:

 Just a wild thought. But is there a way you could possibly get the JVM to 
 garbage collect between each run? Maybe that might free the memory up?
 
 Thanks.
 
 -Mike
 
 On 18/05/11 13:20, Eric Douglas wrote:
 
 I am using Fop 1.0. 
 I tried using Fop to transform a single document.  When I got a little over 
 100 pages my FO file was over 5 MB.  The transform crashed with a Java heap 
 out of memory error.
 
 I managed to break the input down, as I'm using embedded code generating the 
 input programmatically, and the PDF output is a lot smaller.
 
 So I'm currently transforming 10 pages at a time, setting the 
 initial-page-number to the next sequence (1, 11, 21, etc).
 
 Then I save all the generated PDFs in memory and merge them using pdfbox.  
 So far this is working great.
 
 I tried to do the same thing with the PNGRenderer, just calling a method to 
 transform 10 pages at a time and save the output images in an array.
 
 The PNGRenderer is created locally in the method.  It should be getting 
 released when the method ends but the java process never releases any memory.
 
 I tested a 90 page report and the memory use was over 1 GB.  I tested on 
 another machine where the memory limit is apparently lower and it crashed on 
 page 24.
 
 Everything about the method to render to PNG is the same as the method to 
 render to PDF aside from the Renderer. 
 Is there a problem with this renderer or something I could need to do 
 different?
 
 
 
 Michael Rubin
 Developer
   
  T
  F
  M
  E
  W
 +44 20 8238 7400 
 +44 20 8238 7401 
  
 mru...@thunderhead.com 
 www.thunderhead.com   
  
 Thunderhead featured in The Sunday Times Profit Track 100 league table of 
 companies with fastest-growing profits. Click here toread more. 
   
 The contents of this e-mail are intended for the named addressee only. It 
 contains information that may be confidential. Unless you are the named 
 addressee or an authorized designee, you may not copy or use it, or disclose 
 it to anyone else. If you received it in error please notify us immediately 
 and then destroy it.


Re: Fop Memory Use

2011-05-18 Thread Peter Hancock
Hi Eric,

Does your document contain many large SVG's?  If so take a look at  Bugzilla
#46360.  This issue was resolved in rev 997602 of FOP trunk.

Pete




On Wed, May 18, 2011 at 5:10 PM, Adrian Cumiskey
adrian.cumis...@gmail.comwrote:

 Hi Eric,

 Fop calculates layout in page sequence chunks, so try breaking up your
 pages into chunks of page sequences.  Pages should be available for garbage
 collection once the page sequence has been rendered.

 Cheers, Adrian.

 On May 18, 2011, at 7:24 AM, Michael Rubin mru...@thunderhead.com wrote:

  Just a wild thought. But is there a way you could possibly get the JVM to
 garbage collect between each run? Maybe that might free the memory up?

 Thanks.

 -Mike

 On 18/05/11 13:20, Eric Douglas wrote:

 I am using Fop 1.0.
 I tried using Fop to transform a single document.  When I got a little over
 100 pages my FO file was over 5 MB.  The transform crashed with a Java heap
 out of memory error.

 I managed to break the input down, as I'm using embedded code generating
 the input programmatically, and the PDF output is a lot smaller.

 So I'm currently transforming 10 pages at a time, setting the
 initial-page-number to the next sequence (1, 11, 21, etc).

 Then I save all the generated PDFs in memory and merge them using pdfbox.
 So far this is working great.

 I tried to do the same thing with the PNGRenderer, just calling a method to
 transform 10 pages at a time and save the output images in an array.

 The PNGRenderer is created locally in the method.  It should be getting
 released when the method ends but the java process never releases any
 memory.

 I tested a 90 page report and the memory use was over 1 GB.  I tested on
 another machine where the memory limit is apparently lower and it crashed on
 page 24.

 Everything about the method to render to PNG is the same as the method to
 render to PDF aside from the Renderer.
 Is there a problem with this renderer or something I could need to do
 different?



  *Michael Rubin*

 Developer
  [image: Thunderhead Logo] [image: Tagline] [image: Triangles]

  *T*

  *F*

  *M*

  *E*

  *W*

 +44 20 8238 7400

 +44 20 8238 7401



 mru...@thunderhead.commru...@thunderhead.com
 www.thunderhead.com



 Thunderhead featured in The Sunday Times Profit Track 100 league table of
 companies with fastest-growing profits. Click 
 herehttp://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdfto read more.
  [image: LinkedIn] http://www.linkedin.com/companies/25033/Thunderhead 
 [image:
 twitter] http://twitter.com/Thunderheadon[image: 
 RSS]http://www.thunderhead.com/rss/rss.php[image:
 YouTube] 
 http://www.youtube.com/user/ThunderheadOnhttp://thunderheadinnovate.wordpress.com/
  [image:
 were-hiring] http://thunderhead.com/about/careers.php

 The contents of this e-mail are intended for the named addressee only. It
 contains information that may be confidential. Unless you are the named
 addressee or an authorized designee, you may not copy or use it, or disclose
 it to anyone else. If you received it in error please notify us immediately
 and then destroy it.




RE: Fop Memory Use

2011-05-18 Thread Eric Douglas
This test run isn't using SVG at all.  The PDFRenderer is working, the
PNGRenderer runs out of memory, so it is using images but as output.
I already broke up the input to multiple FOs with multiple calls to the
transform to generate a large document by combining small documents
using the initial-page-number.
As the program runs it just keeps increasing memory use.
I tried running a profile with Java's VisualVM though I'm not sure what
exactly I'm looking at or what to do with it.
The number one item showing memory hog in the profiler, as of my last
snapshot was: class int[], live bytes 23,273,952 B, live objects 382,
generations 10
After the program crashed with the profiler running I had an additional
file opened, Java2DRenderer.class, so I'm assuming it's doing something
that breaks PNGRenderer.
My class doesn't have any int[] references.
After that first reference the sizes drop off sharply in the profiler.
The next class reference is char[], then
org.apache.fop.area.inline.SpaceArea.
 



From: Peter Hancock [mailto:peter.hanc...@gmail.com] 
Sent: Wednesday, May 18, 2011 12:05 PM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use


Hi Eric,

Does your document contain many large SVG's?  If so take a look at
Bugzilla #46360.  This issue was resolved in rev 997602 of FOP trunk.

Pete





On Wed, May 18, 2011 at 5:10 PM, Adrian Cumiskey
adrian.cumis...@gmail.com wrote:


Hi Eric,

Fop calculates layout in page sequence chunks, so try breaking
up your pages into chunks of page sequences.  Pages should be available
for garbage collection once the page sequence has been rendered.

Cheers, Adrian.

On May 18, 2011, at 7:24 AM, Michael Rubin
mru...@thunderhead.com wrote:



Just a wild thought. But is there a way you could
possibly get the JVM to garbage collect between each run? Maybe that
might free the memory up?

Thanks.

-Mike

On 18/05/11 13:20, Eric Douglas wrote: 

I am using Fop 1.0. 
I tried using Fop to transform a single
document.  When I got a little over 100 pages my FO file was over 5 MB.
The transform crashed with a Java heap out of memory error.

I managed to break the input down, as I'm using
embedded code generating the input programmatically, and the PDF output
is a lot smaller.

So I'm currently transforming 10 pages at a
time, setting the initial-page-number to the next sequence (1, 11, 21,
etc).

Then I save all the generated PDFs in memory and
merge them using pdfbox.  So far this is working great. 

I tried to do the same thing with the
PNGRenderer, just calling a method to transform 10 pages at a time and
save the output images in an array.

The PNGRenderer is created locally in the
method.  It should be getting released when the method ends but the java
process never releases any memory.

I tested a 90 page report and the memory use was
over 1 GB.  I tested on another machine where the memory limit is
apparently lower and it crashed on page 24.

Everything about the method to render to PNG is
the same as the method to render to PDF aside from the Renderer. 
Is there a problem with this renderer or
something I could need to do different? 






Michael Rubin

Developer

Thunderhead Logo Tagline Triangles  
 T

 F

 M

 E

 W

+44 20 8238 7400 

+44 20 8238 7401 

 

mailto:mru...@thunderhead.com mru...@thunderhead.com 

www.thunderhead.com http://www.thunderhead.com   

Thunderhead featured in The Sunday Times Profit Track 100 league table
of companies with fastest-growing profits. Click here
http://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf  to read more.


LinkedIn http://www.linkedin.com/companies/25033/Thunderhead  twitter
http://twitter.com/Thunderheadon RSS
http://www.thunderhead.com/rss/rss.php YouTube
http://www.youtube.com/user/ThunderheadOn
http://thunderheadinnovate.wordpress.com/  were-hiring
http://thunderhead.com/about/careers.php  
The contents of this e-mail are intended for the named addressee only.
It contains information that may be confidential. Unless you are the
named addressee or an authorized designee, you may not copy or use it,
or disclose it to anyone else. If you received it in error please notify
us immediately and then destroy it.










RE: Fop Memory Use

2011-05-18 Thread Eric Douglas
What I could find on int[] references seems to be...
org.apache.fop.render.pdf.PDFRenderer extends
AbstractPathOrientedRenderer implements PDFConfigurationConstants

 

org.apache.fop.render.bitmap.PNGRenderer extends Java2DRenderer

 

org.apache.fop.render.java2d.Java2DRenderer extends
AbstractPathOrientedRenderer implements Printable

int[] references:

public static void renderText(TextArea text, Graphics2D g2d, Font font)

private static int[] getGlyphOffsets(String s, Font font, TextArea text,
int[] letterAdjust)

 

Could this be because I'm loading in custom fonts?

Is this a bug in the Java2DRenderer?  A simple workaround?




From: Peter Hancock [mailto:peter.hanc...@gmail.com] 
Sent: Wednesday, May 18, 2011 12:05 PM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use


Hi Eric,

Does your document contain many large SVG's?  If so take a look at
Bugzilla #46360.  This issue was resolved in rev 997602 of FOP trunk.

Pete





On Wed, May 18, 2011 at 5:10 PM, Adrian Cumiskey
adrian.cumis...@gmail.com wrote:


Hi Eric,

Fop calculates layout in page sequence chunks, so try breaking
up your pages into chunks of page sequences.  Pages should be available
for garbage collection once the page sequence has been rendered.

Cheers, Adrian.

On May 18, 2011, at 7:24 AM, Michael Rubin
mru...@thunderhead.com wrote:



Just a wild thought. But is there a way you could
possibly get the JVM to garbage collect between each run? Maybe that
might free the memory up?

Thanks.

-Mike

On 18/05/11 13:20, Eric Douglas wrote: 

I am using Fop 1.0. 
I tried using Fop to transform a single
document.  When I got a little over 100 pages my FO file was over 5 MB.
The transform crashed with a Java heap out of memory error.

I managed to break the input down, as I'm using
embedded code generating the input programmatically, and the PDF output
is a lot smaller.

So I'm currently transforming 10 pages at a
time, setting the initial-page-number to the next sequence (1, 11, 21,
etc).

Then I save all the generated PDFs in memory and
merge them using pdfbox.  So far this is working great. 

I tried to do the same thing with the
PNGRenderer, just calling a method to transform 10 pages at a time and
save the output images in an array.

The PNGRenderer is created locally in the
method.  It should be getting released when the method ends but the java
process never releases any memory.

I tested a 90 page report and the memory use was
over 1 GB.  I tested on another machine where the memory limit is
apparently lower and it crashed on page 24.

Everything about the method to render to PNG is
the same as the method to render to PDF aside from the Renderer. 
Is there a problem with this renderer or
something I could need to do different? 






Michael Rubin

Developer

Thunderhead Logo Tagline Triangles  
 T

 F

 M

 E

 W

+44 20 8238 7400 

+44 20 8238 7401 

 

mailto:mru...@thunderhead.com mru...@thunderhead.com 

www.thunderhead.com http://www.thunderhead.com   

Thunderhead featured in The Sunday Times Profit Track 100 league table
of companies with fastest-growing profits. Click here
http://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf  to read more.


LinkedIn http://www.linkedin.com/companies/25033/Thunderhead  twitter
http://twitter.com/Thunderheadon RSS
http://www.thunderhead.com/rss/rss.php YouTube
http://www.youtube.com/user/ThunderheadOn
http://thunderheadinnovate.wordpress.com/  were-hiring
http://thunderhead.com/about/careers.php  
The contents of this e-mail are intended for the named addressee only.
It contains information that may be confidential. Unless you are the
named addressee or an authorized designee, you may not copy or use it,
or disclose it to anyone else. If you received it in error please notify
us immediately and then destroy it.










AW: Fop Memory Use

2011-05-18 Thread Georg Datterl
Hi Eric,

That sounds interesting. If you run the transformer for each page and set a 
breakpoint after the first run, there (IMHO) should not be a reference to any 
fop object. Ignore the int[]s first, they are used everywhere. Concentrate on 
the fop objects which should not be there. You could as well run your 
transformation X times and then investigate all objects which exists exactly X 
(or Y*X) times in memory. Those  are probably accumulated over many runs and 
crash your application sooner or later.

Regards,

Georg Datterl

-- Kontakt --

Georg Datterl

Geneon media solutions gmbh
Gutenstetter Straße 8a
90449 Nürnberg

HRB Nürnberg: 17193
Geschäftsführer: Yong-Harry Steiert

Tel.: 0911/36 78 88 - 26
Fax: 0911/36 78 88 - 20

www.geneon.dehttp://www.geneon.de

Weitere Mitglieder der Willmy MediaGroup:

IRS Integrated Realization Services GmbH:
www.irs-nbg.dehttp://www.irs-nbg.de
Willmy PrintMedia GmbH:  www.willmy.dehttp://www.willmy.de
Willmy Consult  Content GmbH:   
www.willmycc.dehttp://www.willmycc.de

Von: Eric Douglas [mailto:edoug...@blockhouse.com]
Gesendet: Mittwoch, 18. Mai 2011 19:43
An: fop-dev@xmlgraphics.apache.org
Betreff: RE: Fop Memory Use

This test run isn't using SVG at all.  The PDFRenderer is working, the 
PNGRenderer runs out of memory, so it is using images but as output.
I already broke up the input to multiple FOs with multiple calls to the 
transform to generate a large document by combining small documents using the 
initial-page-number.
As the program runs it just keeps increasing memory use.
I tried running a profile with Java's VisualVM though I'm not sure what exactly 
I'm looking at or what to do with it.
The number one item showing memory hog in the profiler, as of my last snapshot 
was: class int[], live bytes 23,273,952 B, live objects 382, generations 10
After the program crashed with the profiler running I had an additional file 
opened, Java2DRenderer.class, so I'm assuming it's doing something that breaks 
PNGRenderer.
My class doesn't have any int[] references.
After that first reference the sizes drop off sharply in the profiler.  The 
next class reference is char[], then org.apache.fop.area.inline.SpaceArea.



From: Peter Hancock [mailto:peter.hanc...@gmail.com]
Sent: Wednesday, May 18, 2011 12:05 PM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use
Hi Eric,

Does your document contain many large SVG's?  If so take a look at  Bugzilla 
#46360.  This issue was resolved in rev 997602 of FOP trunk.

Pete



On Wed, May 18, 2011 at 5:10 PM, Adrian Cumiskey 
adrian.cumis...@gmail.commailto:adrian.cumis...@gmail.com wrote:
Hi Eric,

Fop calculates layout in page sequence chunks, so try breaking up your pages 
into chunks of page sequences.  Pages should be available for garbage 
collection once the page sequence has been rendered.

Cheers, Adrian.

On May 18, 2011, at 7:24 AM, Michael Rubin 
mru...@thunderhead.commailto:mru...@thunderhead.com wrote:

Just a wild thought. But is there a way you could possibly get the JVM to 
garbage collect between each run? Maybe that might free the memory up?

Thanks.

-Mike

On 18/05/11 13:20, Eric Douglas wrote:

I am using Fop 1.0.
I tried using Fop to transform a single document.  When I got a little over 100 
pages my FO file was over 5 MB.  The transform crashed with a Java heap out of 
memory error.

I managed to break the input down, as I'm using embedded code generating the 
input programmatically, and the PDF output is a lot smaller.

So I'm currently transforming 10 pages at a time, setting the 
initial-page-number to the next sequence (1, 11, 21, etc).

Then I save all the generated PDFs in memory and merge them using pdfbox.  So 
far this is working great.

I tried to do the same thing with the PNGRenderer, just calling a method to 
transform 10 pages at a time and save the output images in an array.

The PNGRenderer is created locally in the method.  It should be getting 
released when the method ends but the java process never releases any memory.

I tested a 90 page report and the memory use was over 1 GB.  I tested on 
another machine where the memory limit is apparently lower and it crashed on 
page 24.

Everything about the method to render to PNG is the same as the method to 
render to PDF aside from the Renderer.
Is there a problem with this renderer or something I could need to do different?




Michael Rubin


Developer


Fehler! Es wurde kein Dateiname angegeben.

Fehler! Es wurde kein Dateiname angegeben.


 T

 F

 M

 E

 W


+44 20 8238 7400

+44 20 8238 7401



mru...@thunderhead.commailto:mru...@thunderhead.com
www.thunderhead.comhttp://www.thunderhead.com




Thunderhead featured in The Sunday Times Profit Track 100 league table of 
companies with fastest-growing profits. Click 
herehttp://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf to read more.

Fehler! Es wurde kein Dateiname 
angegeben.http