Re: Memory issue
Jean-François El Fouly a écrit : So I hired the team member who's competent in profiler usage next week but I must say at the moment I'm still stuck :-( The sysadmins made a tarball from the staging server and copied everything to a similar server that has full profiling instrumentation (with JProfiler). And obviously there the application works and memory usage is quite normal. Completely different behaviour. Very, very strange. I'll try to understand what's happening (well, I badly need to) but it's probably not going to be easy :-(
Re: Memory issue
On May 8, 2008, at 12:03, Andreas Delmelle wrote: Hi Jean-François, On May 8, 2008, at 12:57, Jean-François El Fouly wrote: Andreas Delmelle a écrit : OK. Just curious: Any chance you could test it on another build or maybe even Java 6? Probably, if required or useful. Our sys admins are very cooperative ;-) For the moment, that would be more a nice-to-know. Chances are that, if it's not JVM-related, this won't help a thing, so no need to go out of your way to do that Yes. That is exactly what happened to the stylesheet we use. I've reduced it drastically. One issue with stylesheets generated by StyleVision is that you must be careful when you tweak them to avoid certain [fo-block inside fo:inline] combinations that make FOP crash with a stack trace and no really useful information about what's happening or where. This bug is mentioned in the FOP bug tracker, though in a rather raw, loose manner. I removed all such constructs and that made the XSLT much simpler and cleaner. OK, so we can exclude that as well. AFAIU, this gives little opportunity for the XSLT processor to clean up anything. Java 1.5 uses Xalan XSLTC by default, which converts templates into Java objects. One giant template would then mean one very long-living object that may reference numerous others for the whole duration of the processing run. If you look at the chain, when using XML+XSLT input, FOP is always the first one to finish, then the XSLT processor, then the XML parser. If the XSLT processor cannot reclaim anything, this will give FOP less room to work with, so it ultimately runs slower. As the heap increases to reach the maximum, the points where the JVM will launch the GC by itself, will also increase. Since it cannot expand the heap anymore, it will try to clean up more frequently. Yep, that is why I've tried to be cautious not to accuse FOP publicly ;-) ... which is also why /we/ are so cooperative/responsive. ;-) BTW: If all users would have the time and motivation to be as thorough as yourself, the traffic on this list would probably drop significantly. The problem is in the (Xalan + FOP) subsystem and the profiling could well show that the issue is Xalan-related. Or maybe even Xerces...? Xerces is a very feature-complete parser, but reports in the past have shown that all those nice features come with a price-tag. For FOP this holds as well, of course, and to be honest, FOP can be a pretty memory-hungry beast if you're not careful (but you definitely seem to be). A relatively easy way to find out whether it's XSLT-related, would be to try out Saxon instead. I don't know if you have any experience with plugging in a different XSLT processor, but this is pretty straightforward (but might require re-starting the JBoss service, depending on how you go about it; for testing purposes, you could ultimately also change the app-code to reference Saxon directly instead of letting the JVM choose the javax.xml.transform.TransformerFactory implementation, and then redeploy). Cheers Andreas - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory issue
Andreas Delmelle a écrit : OK. Just curious: Any chance you could test it on another build or maybe even Java 6? Probably, if required or useful. Our sys admins are very cooperative ;-) In my personal experience, optimizing the stylesheet code usually does not offer much improvement in terms of global memory usage, but it could have a noticeable impact on the processing time. One of the things I've learned about generated XSL-FO stylesheets by Altova is that they add a lot of fo:inlines to specify, for example, font-properties on the lowest levels in the generated FO while, when comparing to the font-properties of the fo:inlines' parents nothing really changes, except for the size, style or weight. From FOP's point of view, that's somewhat of a waste. Much better to specify a global font-size on the page-sequence, and override on the lower levels only what is really necessary. After adapting the stylesheet manually, and removing the redundant fo:inlines, the stylesheet and the generated FO were reduced to not even half the original size. Yes. That is exactly what happened to the stylesheet we use. I've reduced it drastically. One issue with stylesheets generated by StyleVision is that you must be careful when you tweak them to avoid certain [fo-block inside fo:inline] combinations that make FOP crash with a stack trace and no really useful information about what's happening or where. This bug is mentioned in the FOP bug tracker, though in a rather raw, loose manner. I removed all such constructs and that made the XSLT much simpler and cleaner. Something else that bothered me, but I don't know if that was also generated by Altova, is that in one of the stylesheets I saw, the entire transformation was contained in one giant template... With the last version, or our XSLT ? this was no longer the case. AFAIU, this gives little opportunity for the XSLT processor to clean up anything. Java 1.5 uses Xalan XSLTC by default, which converts templates into Java objects. One giant template would then mean one very long-living object that may reference numerous others for the whole duration of the processing run. If you look at the chain, when using XML+XSLT input, FOP is always the first one to finish, then the XSLT processor, then the XML parser. If the XSLT processor cannot reclaim anything, this will give FOP less room to work with, so it ultimately runs slower. As the heap increases to reach the maximum, the points where the JVM will launch the GC by itself, will also increase. Since it cannot expand the heap anymore, it will try to clean up more frequently. Yep, that is why I've tried to be cautious not to accuse FOP publicly ;-) The problem is in the (Xalan + FOP) subsystem and the profiling could well show that the issue is Xalan-related. BTW, we've made the Xalan-FOP coupling a parameter so that we can use tight coupling (with Sax events) or loose coupling (writing the intermediate FO files on disk). We usually use the second option, since the possibility to read the FO intermediate code is helpful when you debug. And I guess without being really sure that not to have Xalan and FOP working at the same time should use less memory. This separation probably accounts for the long execution time, but that is not an issue since document generation does not occur often in the target system (you can generate chapters for proofreading but you generate the whole document once-twice a day). - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory issue
On May 8, 2008, at 11:38, Jean-François El Fouly wrote: Andreas Delmelle a écrit : Which Java VM are you using? Practically every time someone tells us about memory/GC issues, it appears they are using an implementation other than Sun (IBM, GNU...) Up to now, we still have to find out why precisely non-Sun VMs have difficulties with FOP... Nope. I'll double check but I'm pretty sure it's a genuine Sun JVM 1.5.0_11, or maybe the very minor build after. OK. Just curious: Any chance you could test it on another build or maybe even Java 6? How large would the resulting FO-files be if you dump them to the filesystem? The XML by itself says very little. From a 1.5MB XML, you could get a FO of a few KB or one of 26MB, depending on the stylesheet. 5.08 Mb. That's not what I would call a large FO, so this should be no problem. Does the stylesheet adhere to XSLT best practices? Does it generate a lot of redundant fo:blocks, fo:inlines? I hope not. It has been a complicated thing generated by StyleVision in the very beginning but it has been simplified and tweaked a lot. In my personal experience, optimizing the stylesheet code usually does not offer much improvement in terms of global memory usage, but it could have a noticeable impact on the processing time. One of the things I've learned about generated XSL-FO stylesheets by Altova is that they add a lot of fo:inlines to specify, for example, font- properties on the lowest levels in the generated FO while, when comparing to the font-properties of the fo:inlines' parents nothing really changes, except for the size, style or weight. From FOP's point of view, that's somewhat of a waste. Much better to specify a global font-size on the page-sequence, and override on the lower levels only what is really necessary. After adapting the stylesheet manually, and removing the redundant fo:inlines, the stylesheet and the generated FO were reduced to not even half the original size. Something else that bothered me, but I don't know if that was also generated by Altova, is that in one of the stylesheets I saw, the entire transformation was contained in one giant template... AFAIU, this gives little opportunity for the XSLT processor to clean up anything. Java 1.5 uses Xalan XSLTC by default, which converts templates into Java objects. One giant template would then mean one very long-living object that may reference numerous others for the whole duration of the processing run. If you look at the chain, when using XML+XSLT input, FOP is always the first one to finish, then the XSLT processor, then the XML parser. If the XSLT processor cannot reclaim anything, this will give FOP less room to work with, so it ultimately runs slower. As the heap increases to reach the maximum, the points where the JVM will launch the GC by itself, will also increase. Since it cannot expand the heap anymore, it will try to clean up more frequently. Cheers Andreas - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory issue
Andreas Delmelle a écrit : Which Java VM are you using? Practically every time someone tells us about memory/GC issues, it appears they are using an implementation other than Sun (IBM, GNU...) Up to now, we still have to find out why precisely non-Sun VMs have difficulties with FOP... Nope. I'll double check but I'm pretty sure it's a genuine Sun JVM 1.5.0_11, or maybe the very minor build after. How large would the resulting FO-files be if you dump them to the filesystem? The XML by itself says very little. From a 1.5MB XML, you could get a FO of a few KB or one of 26MB, depending on the stylesheet. 5.08 Mb. Does the stylesheet adhere to XSLT best practices? Does it generate a lot of redundant fo:blocks, fo:inlines? I hope not. It has been a complicated thing generated by StyleVision in the very beginning but it has been simplified and tweaked a lot. A nit, for the record: There is no such thing as 'forcing garbage collection'. The most you can do with System.gc() is indicate to the VM that it should run the GC as soon as possible. Admitted, most implementations do run the algorithm virtually immediately upon execution of the statement, but the Java spec does not mandate such behavior. In theory, if the VM is too busy, it could still postpone the actual GC-run, until it acquires the necessary resources... Indeed, but the log4j log has timestamps and they show that 20 seconds are spent around System.gc() so my guess is that something really happens at that time. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory issue (was: Re: Major issue with image loading in FOP 0.95beta)
On May 8, 2008, at 08:40, Jean-François El Fouly wrote: Hi Jeremias Maerki a écrit : And my next problem is to find a way to force memory recycling after this long and hefty FOP processing, but until further investigated this is OT ;-) You probably didn't get my hint earlier but with the new image loading framework you should actually get away with lower memory settings. In my tests I have been able to produce PDF with little text and many images with 40MB of VM memory which wasn't possible with 0.94 and earlier. Well, I got the hint, but it seems in contradiction with what I read. There are, of course, other factors to take into account than simply document/image sizes. Which Java VM are you using? Practically every time someone tells us about memory/GC issues, it appears they are using an implementation other than Sun (IBM, GNU...) Up to now, we still have to find out why precisely non-Sun VMs have difficulties with FOP... What options does the Java VM offer to tweak the GC? What options does it use by default? So to take the picture from a bit higher: - all XSL-FO transformation + FOP generation now work OK. - this generates 20-30 documents (chapters) for a grand total of about 150 Mb, to be bound together by iText. - source XML is 1.5 Mb - 1011 PNG images for a total of 151 Mb, the largest image is 715 kb. Now the figures: - XML -> XSL-FO transformation + FOP generation take 15 minutes on a pretty decent DELL Server (running Debian 4.0) having all the physical RAM possible (staging server for several customers) How large would the resulting FO-files be if you dump them to the filesystem? The XML by itself says very little. From a 1.5MB XML, you could get a FO of a few KB or one of 26MB, depending on the stylesheet. Does the stylesheet adhere to XSLT best practices? Does it generate a lot of redundant fo:blocks, fo:inlines? - JVM has 2000 Mb (which is BTW the grand max on this processor/ server/OS/JVM architecture) On my end, that has proven to be more than enough to generate one page-sequence with a table of 15 columns, spanning 500+ pages. (Note: only text-content, no images; more a test to check the memory usage without doing anything special, just a whole lot of FOs) If I try to investigate memory usage using Runtime.getRuntime ().getFreeMemory() and logging the figures with log4j, these are the figures I get: - before XSLT + FOP: 1900 Mb free/2000 Mb - end of XSLT + FOP: 241 Mb free Yikes! That looks troublesome indeed... :/ - set FopFactory instance to null as a desperate hint to the GC that FOP objects could be/should be recycled - I force garbage collection using System.gc()[OK, in an application server this is a brute force approach, but could not see a more clever maneuver ATM] A nit, for the record: There is no such thing as 'forcing garbage collection'. The most you can do with System.gc() is indicate to the VM that it should run the GC as soon as possible. Admitted, most implementations do run the algorithm virtually immediately upon execution of the statement, but the Java spec does not mandate such behavior. In theory, if the VM is too busy, it could still postpone the actual GC-run, until it acquires the necessary resources... Now I don't take runtime.getXXXMemory() for bible word but at least it "looks like" the Xalan + FOP subsystem hogs 1500 Mb of RAM which I cannot recover. So I hired the team member who's competent in profiler usage next week but I must say at the moment I'm still stuck :-( If you're not on a Sun VM, then I have a very vague feeling that he's going to discover the issue to be related to arrays, a special type of object, but I could be wrong about this. Someone once reported that the VM seemed to hold on to a lot of arrays. When profiling, he discovered that the arrays were referenced nowhere, but still the GC did not clean them up. Of course I've made my homework and read the f...riendly manual before daring to ask. Did I miss any important indication ? I don't think so, but it seems we might do well by putting some of the info concerning JVM/GC implementation we have gathered so far, on the website or a Wiki. Cheers Andreas - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory issue (was: Re: Major issue with image loading in FOP 0.95beta)
I've done extensive tests about memory allocation with FOP when implementing the new image loading framework and in my case the memory was always released. So, some results from a profiler would be helpful. Anyway, what I meant with my hint was that the iText step might not be necessary anymore and that you should be able to safely reduce -Xmx on the JVM (provided your document doesn't contain to much non-image content). But if you release FopFactory and the memory is still not reclaimed something's wrong somewhere and I have a somewhat hard time believing that FOP itself somehow still holds on to it. Please make sure you don't hold on to the FOUserAgent and other FOP-related objects because they might have a reference to the FopFactory. On 08.05.2008 08:40:55 Jean-François El Fouly wrote: > Jeremias Maerki a écrit : > >> And my next problem is to find a way to force memory recycling after > >> this long and hefty FOP processing, but until further investigated this > >> is OT ;-) > >> > > > > You probably didn't get my hint earlier but with the new image loading > > framework you should actually get away with lower memory settings. In my > > tests I have been able to produce PDF with little text and many images > > with 40MB of VM memory which wasn't possible with 0.94 and earlier. > > > > Well, I got the hint, but it seems in contradiction with what I read. > So to take the picture from a bit higher: > - all XSL-FO transformation + FOP generation now work OK. > - this generates 20-30 documents (chapters) for a grand total of about > 150 Mb, to be bound together by iText. > - source XML is 1.5 Mb > - 1011 PNG images for a total of 151 Mb, the largest image is 715 kb. > > Now the figures: > - XML -> XSL-FO transformation + FOP generation take 15 minutes on a > pretty decent DELL Server (running Debian 4.0) having all the physical > RAM possible (staging server for several customers) > - JVM has 2000 Mb (which is BTW the grand max on this > processor/server/OS/JVM architecture) > - only one instance of FOP launched (one document generation) > - the second next step in the publication process (opening the 150 Mb > with iText to add the bookmarks) fails immediately (at file open) saying > it cannot allocate memory > > If I try to investigate memory usage using > Runtime.getRuntime().getFreeMemory() and logging the figures with log4j, > these are the figures I get: > - before XSLT + FOP: 1900 Mb free/2000 Mb > - end of XSLT + FOP: 241 Mb free > - set FopFactory instance to null as a desperate hint to the GC that FOP > objects could be/should be recycled > - I force garbage collection using System.gc()[OK, in an application > server this is a brute force approach, but could not see a more clever > maneuver ATM] > - 350 Mb free/2000 Mb total > - Bind all chapters with iText > - 250 Mb free > - Force another GC > - 350 Mb free again (so the binding operation has no effect on the > available memory). > - the next iText step still fails. > > Now I don't take runtime.getXXXMemory() for bible word but at least it > "looks like" the Xalan + FOP subsystem hogs 1500 Mb of RAM which I > cannot recover. > So I hired the team member who's competent in profiler usage next week > but I must say at the moment I'm still stuck :-( > > Of course I've made my homework and read the f...riendly manual before > daring to ask. > Did I miss any important indication ? > Jeremias Maerki - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
