I have some baseline memory snapshots using yourkit that I can compare against. All someone has to do is tell me what changes to make or send me a patch (were using 1.5 if it matters).
Could you send me a patch file? I am not familiar with that code, but is the potential savings 24 bytes per node? In the snapshots I took for our application, we have almost 1 million nodes. Ryan -----Original Message----- From: Will Glass-Husain [mailto:[EMAIL PROTECTED] Sent: Tuesday, March 13, 2007 11:18 PM To: Velocity Developers List Subject: Re: JavaCC memory problems [was: Re: Problems with build Velocity] Did "ant test" pass? It'd be interesting to see an example high-memory use case and get a feel for what percentage this lowers the memory use. WILL On 3/13/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote: > Will, > > i try to test more JavaCC compiler and do some optimalizations with > code generation. > > In Parser.jjt we can remove USER_CHAR_STREAM=true becouse auto > generated SimpleCharStream is almost identical to our > VelocityCharStream. If we decide to do this it's possible in Parser > class to: > - remove VelocityCharStream velcharstream variable - this will reduce > memory use, every VCS has 24kb arrays in memory > - change Parser CTR to simple > > public Parser( RuntimeServices rs) > { > /* > * need to call the CTOR first thing. > */ > this( new ByteArrayInputStream("\n".getBytes())); > > /* > * and save the RuntimeServices > */ > rsvc = rs; > } > -- and parse method to simple one: > > > public SimpleNode parse( Reader reader, String templateName ) > throws ParseException > { > SimpleNode sn = null; > > currentTemplateName = templateName; > > try > { > token_source.clearStateVars(); > /* > * now reinit the Parser with this CharStream > */ > ReInit(reader); > > [..] > > catch (Exception e) > { > rsvc.getLog().error("Parser Error: " + templateName, e); > } > > currentTemplateName = ""; > > // finish with input strem > token_source.clearStateVars(); > jj_input_stream.inputStream = null; // gc - to unrefernce > passed Reader from Parser class > > return sn; > } > - (optional) TokenMgrError can be regenerated to fix some typos in comment > > - regenerate javacc code and recompile. > > I run test with latest Jira (3.7.4-#189 Standalone/Java6) Velocity > 1.6-dev and JavaCC 4.0 and this still works ;) Of course, this need > more test - someone can test this with larger app with more templates? > > > Sylwester > > 2007/3/13, Will Glass-Husain <[EMAIL PROTECTED]>: > > Sylvester, > > > > Perhaps RuntimeInstance.parser could call a cleanup method? You can > > insert new methods into Parser.java by putting them in Parser.jjt. > > > > Re: VelocityCharStream. I'm a little leery of lazy inits unless they > > demonstrate significant performance improvements. I've found them to > > be an easy way to convolute the code without always providing > > performance benefits. > > > > WILL > > > > On 3/12/07, Will Glass-Husain <[EMAIL PROTECTED]> wrote: > > > We need to figure out a way to put this in Parser.jj. Parser.java is > > > automatically generated-- it's too complex to generate Parser.java and > > > then modify it. > > > > > > WILL > > > > > > On 3/12/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote: > > > > I do something like this. This works for me, but need some more tests. > > > > Also in method parse() - resource cleanup should be in finally block > > > > (currentTemplateName, token_source, velcharstream). > > > > > > > > In VelocityCharStream class we can also do some rework to lazy init int,char > > > > buffers > > > > > > > > I'll try to contact JavaCC developers to find where Done() method should be > > > > called during parsing. > > > > > > > > Sylwester > > > > > > > > Index: Parser.java > > > > =================================================================== > > > > --- Parser.java (revision 517275) > > > > +++ Parser.java (working copy) > > > > @@ -53,8 +53,7 @@ > > > > * need to call the CTOR first thing. > > > > */ > > > > > > > > - this( new VelocityCharStream( > > > > - new ByteArrayInputStream("\n".getBytes()), 1, 1 )); > > > > + this( (CharStream)null ); > > > > > > > > /* > > > > * now setup a VCS for later use > > > > @@ -129,7 +128,8 @@ > > > > } > > > > > > > > currentTemplateName = ""; > > > > - > > > > + token_source.clearStateVars(); > > > > + if (velcharstream!=null) velcharstream.Done(); > > > > return sn; > > > > } > > > > > > > > > > > > > > > > 2007/3/12, Ryan Smith <[EMAIL PROTECTED]>: > > > > > > > > > > > > > > > Will, > > > > > > > > > > Where would you put the code to call the Done() method and free up the > > > > > memory? > > > > > > > > > > We currently doing profiling of Velocity as well and are trying to find > > > > > places exactly like this. > > > > > > > > > > Ryan > > > > > > > > > > -----Original Message----- > > > > > From: Will Glass-Husain [mailto:[EMAIL PROTECTED] > > > > > Sent: Monday, March 12, 2007 2:47 PM > > > > > To: Velocity Developers List > > > > > Subject: Re: JavaCC memory problems [was: Re: Problems with build > > > > > Velocity] > > > > > > > > > > Thanks for the tips! Definitely something to investigate. > > > > > > > > > > WILL > > > > > > > > > > On 3/12/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote: > > > > > > Hi, > > > > > > > > > > > > today JavaCC works fine - maybe there was bad day ;) > > > > > > > > > > > > I run profiler with simple velocity app (2 .vm file, simple merge) and > > > > > found > > > > > > that internal by design CharStream has method : > > > > > > --- > > > > > > /** > > > > > > * The lexer calls this function to indicate that it is done with > > > > > the > > > > > > stream > > > > > > * and hence implementations can free any resources held by this > > > > > class. > > > > > > * Again, the body of this function can be just empty and it will > > > > > not > > > > > > * affect the lexer's operation. > > > > > > */ > > > > > > void Done(); > > > > > > -- > > > > > > but this method is never called by any JavaCC code (lexer/parser). > > > > > > In class VelocityCharStream.Done() implementation buffers are > > > > > cleaned. > > > > > > > > > > > > More, in VelocityCharStream constructor we have 3 times allocation of > > > > > 4kb > > > > > > buffer > > > > > > buffer = new char[buffersize]; > > > > > > bufline = new int[buffersize]; > > > > > > bufcolumn = new int[buffersize]; > > > > > > > > > > > > And in class Parser constructor we create 2 times VelocityCharStream. > > > > > > So every Parser allocates about 24kb memory, even when is not needed. > > > > > > > > > > > > When I run trace I can find that memory is deallocated from old > > > > > tempate only > > > > > > on new parse request, but only to this 24kb. > > > > > > In Done method we can also null readerStreem. > > > > > > > > > > > > Mayby this will help improve memory management. > > > > > > > > > > > > Sylwester > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2007/3/12, Will Glass-Husain <[EMAIL PROTECTED]>: > > > > > > > > > > > > > > Hi Sylvester, > > > > > > > > > > > > > > Thanks for the note on the build script. It really needs to check > > > > > for > > > > > > > a min of 1.6. > > > > > > > > > > > > > > I'm not sure why you have problems compiling. I use javacc 3.2 as > > > > > > > well. Did you type "ant clean" first? > > > > > > > > > > > > > > JavaCC generates node files and Parser.java. The node files are not > > > > > > > needed and can be deleted. (I think this happens in the build > > > > > > > script). If you are just changing existing syntax, no manual > > > > > copying > > > > > > > of files is required. If you are adding a new Node you'll have to > > > > > > > move that into the Node directory. > > > > > > > > > > > > > > WILL > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -- Forio Business Simulations Will Glass-Husain [EMAIL PROTECTED] www.forio.com --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
