memory behavior to expect with gjdoc-0.7.6
Greetings, I'm using gjdoc-0.7.6 to test a new port of JamVM-1.4.1 + classpath-0.19 on OpenBSD. I'm trying to get gjdoc to do all the classpath documentation. gjdoc makes a lot of progress (it's outputting stuff), but blows off by hitting an upper memory limit. My question is this: Should I expect that gjdoc will need to use > 1GB per process virtual memory? How big? (This may be transparently satisfied on Linux; on OpenBSD I have to recompile the kernel. Bummer.) >From previous e-mails on this list it seems that gjdoc needs a larger machine, but in what way? (physical memory, virtual memory, ...) The notebook I'm running this on has 1.2GB physical memory; and 1.5GB of swap, but OpenBSD generic per process limit of 1GB. TIA, Fred ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
Hi Frederick, On Sun, 2005-12-04 at 21:26 -0500, Frederick C Druseikis wrote: > I'm using gjdoc-0.7.6 to test a new port of JamVM-1.4.1 + > classpath-0.19 on OpenBSD. I'm trying to get gjdoc to do all the > classpath documentation. gjdoc makes a lot of progress (it's > outputting stuff), but blows off by hitting an upper memory limit. > > My question is this: Should I expect that gjdoc will need to use > 1GB > per process virtual memory? How big? (This may be transparently > satisfied on Linux; on OpenBSD I have to recompile the kernel. Bummer.) Running gjdoc over all of GNU Classpath at once is pretty memory intensive. I don't have a setup at this moment using jamvm. But a natively compiled (with gcj) gjdoc seems to have a top virtual memory usage of just above 300MB. So 1GB seems a bit much. But try generating documentation first of just one or two subpackages to see if that works. Cheers, Mark signature.asc Description: This is a digitally signed message part ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
Hi, On 12/5/05, Mark Wielaard <[EMAIL PROTECTED]> wrote: > Hi Frederick, > > On Sun, 2005-12-04 at 21:26 -0500, Frederick C Druseikis wrote: > > I'm using gjdoc-0.7.6 to test a new port of JamVM-1.4.1 + > > classpath-0.19 on OpenBSD. I'm trying to get gjdoc to do all the > > classpath documentation. gjdoc makes a lot of progress (it's > > outputting stuff), but blows off by hitting an upper memory limit. > > > > My question is this: Should I expect that gjdoc will need to use > 1GB > > per process virtual memory? How big? (This may be transparently > > satisfied on Linux; on OpenBSD I have to recompile the kernel. Bummer.) > > Running gjdoc over all of GNU Classpath at once is pretty memory > intensive. I don't have a setup at this moment using jamvm. But a > natively compiled (with gcj) gjdoc seems to have a top virtual memory > usage of just above 300MB. So 1GB seems a bit much. But try generating > documentation first of just one or two subpackages to see if that works. > Could you also run JamVM with -verbose:gc and send me the output? I've never ran gjdoc myself, but I'll have a look at running it to see what happens on my setup. Thanks, Rob. > Cheers, > > Mark > > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.2 (GNU/Linux) > > iD8DBQBDlD2+xVhZCJWr9QwRAmEzAKCnq+9RsN5f/gjR2W5g4hBCNn3stgCgmiz4 > zHjDNYYTionk4l3SDIIAGuY= > =r/+u > -END PGP SIGNATURE- > > > ___ > Classpath mailing list > Classpath@gnu.org > http://lists.gnu.org/mailman/listinfo/classpath > > > ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
Hi Fred, On Mon, 2005-12-05 at 12:32 -0500, [EMAIL PROTECTED] wrote: > > Could you also run JamVM with -verbose:gc and send me the output? > > > Attached. Thanks. This seems to point out two things: 1) There is a huge allocation (2MB+): at this point in the code: // REVIEW: Using max instead of average may allocate a very large // buffer. Maybe we should do something more efficient? int remaining = in.remaining (); int n = (int) (remaining * maxBytesPerChar ()); ByteBuffer out = ByteBuffer.allocate (n); I believe that REVIEW note gives us a hint :) 2) JamVM has fragmented its heap so much that it cannot allocate such a block of data even though there is enough space in total: Or am I reading the output incorrectly? Cheers, Mark signature.asc Description: This is a digitally signed message part ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
Hi Rob, > > Could you also run JamVM with -verbose:gc and send me the output? > Attached. I ran this from the Makefile in .../classpath-0.19/doc/api/Makefile defining the symbol GJDOC in the Makefile as GJDOC= gjdoc -J-Xms200m -J-Xmx900m -verbose:gc then do gmake html I also had to fiddle with my ulimit or it would blow off much earlier. It seems like gjdoc has to get to about 700MB before it starts to ouptut HTML pages. It took almost 7 min. to produce this output on my 1.3GHz Celeron M. With "top" running on another screen SIZE eventually showed 935MB, RES showed 936MB. I had about 300MB of physical memory left. Fred gjdoc-20051205.out Description: Binary data ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
Hi, On 12/5/05, Mark Wielaard <[EMAIL PROTECTED]> wrote: > Hi Fred, > > On Mon, 2005-12-05 at 12:32 -0500, [EMAIL PROTECTED] wrote: > > > Could you also run JamVM with -verbose:gc and send me the output? > > > > > Attached. > > Thanks. This seems to point out two things: > > 1) There is a huge allocation (2MB+): > > at this point in the code: > >// REVIEW: Using max instead of average may allocate a very large >// buffer. Maybe we should do something more efficient? >int remaining = in.remaining (); >int n = (int) (remaining * maxBytesPerChar ()); >ByteBuffer out = ByteBuffer.allocate (n); > > I believe that REVIEW note gives us a hint :) > > 2) JamVM has fragmented its heap so much that it cannot allocate such a > block of data even though there is enough space in total: > 943718392 (82%)> > > Or am I reading the output incorrectly? > This is what I was afraid of. JamVM doesn't use handles so compaction is a non-trivial exercise. However, I'd like to analyse the gc output myself, but I don't seem to have been sent it... Rob. > Cheers, > > Mark ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
On 12/5/05, Robert Lougher <[EMAIL PROTECTED]> wrote: > Hi, > > On 12/5/05, Mark Wielaard <[EMAIL PROTECTED]> wrote: > > Hi Fred, > > > > On Mon, 2005-12-05 at 12:32 -0500, [EMAIL PROTECTED] wrote: > > > > Could you also run JamVM with -verbose:gc and send me the output? > > > > > > > Attached. > > > > Thanks. This seems to point out two things: > > > > 1) There is a huge allocation (2MB+): > > > > at this point in the code: > > > >// REVIEW: Using max instead of average may allocate a very large > >// buffer. Maybe we should do something more efficient? > >int remaining = in.remaining (); > >int n = (int) (remaining * maxBytesPerChar ()); > >ByteBuffer out = ByteBuffer.allocate (n); > > > > I believe that REVIEW note gives us a hint :) > > > > 2) JamVM has fragmented its heap so much that it cannot allocate such a > > block of data even though there is enough space in total: > >>943718392 (82%)> > > > > Or am I reading the output incorrectly? > > > > This is what I was afraid of. JamVM doesn't use handles so compaction > is a non-trivial exercise. However, I'd like to analyse the gc output > myself, but I don't seem to have been sent it... > On second thoughts, it may have been rejected by gmail if the attachment was too big (how big was it?). Could you try compressing it (if it wasn't)? Thanks, Rob. > Rob. > > > Cheers, > > > > Mark > ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
Hi Robert, On Mon, 2005-12-05 at 18:10 +, Robert Lougher wrote: > On second thoughts, it may have been rejected by gmail if the > attachment was too big (how big was it?). Could you try compressing > it (if it wasn't)? It is in the gmane archives here: http://article.gmane.org/gmane.comp.java.classpath.devel/6717 Cheers, Mark signature.asc Description: This is a digitally signed message part ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
Hi, On 12/5/05, Robert Lougher <[EMAIL PROTECTED]> wrote: > Hi, > > On 12/5/05, Mark Wielaard <[EMAIL PROTECTED]> wrote: > > 2) JamVM has fragmented its heap so much that it cannot allocate such a > > block of data even though there is enough space in total: > >>943718392 (82%)> > > > > Or am I reading the output incorrectly? > > > > This is what I was afraid of. JamVM doesn't use handles so compaction > is a non-trivial exercise. However, I'd like to analyse the gc output > myself, but I don't seem to have been sent it... > If I remember correctly, gcc/gij uses the Boehm GC? This doesn't do compaction either, but it does maintain different size free-lists, enabling it to use an approximate best fit algorithm. JamVMs allocator uses a next-fit allocation strategy -- the rationale here is to try and keep objects created together in time close together in the heap, as objects tend to die together as well (and locality of reference means it may exhibit better cache behaviour). However, it's obviously performing appallingly here. I'll have a look at changing the allocation strategy and see if it improves things. Thanks, Rob. > > Cheers, > > > > Mark > ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
Re: memory behavior to expect with gjdoc-0.7.6
[Sorry for the duplicate message, trouble with my primary email account] Hi, On Mon, 2005-12-05 at 18:52 +0100, Mark Wielaard wrote: > Thanks. This seems to point out two things: > > 1) There is a huge allocation (2MB+): > >at this point in the code: > > // REVIEW: Using max instead of average may allocate a very large > // buffer. Maybe we should do something more efficient? > int remaining = in.remaining (); > int n = (int) (remaining * maxBytesPerChar ()); > ByteBuffer out = ByteBuffer.allocate (n); > > I believe that REVIEW note gives us a hint :) It gives us a hint (thanks for review help from Sven on irc) that we are pushing huge Strings through the encoders. In particular gjdoc creates a full String for each XHTMLified/pretty-printed/color-coded/etc source file in HtmlDoclet.java. Although all the rest of the generated pages are "streamed" to disk the actual source code formating is done in one go: String result = java2xhtml.makeHTML(sourceBuffer.getBuffer(), sourceFile.getName()); Which is then written to disk in one go. For the larger source files this can be pretty big. So a quick workaround for now would be to not include the -linksource flag to gjdoc. If someone wants a nice hacking task then they could look into making java2xhtml take a Reader to read the source from and a HtmlPage to output to instead of creating one huge String. Cheers, Mark ___ Classpath mailing list Classpath@gnu.org http://lists.gnu.org/mailman/listinfo/classpath
