Looked at the code. Looks promising - nice work.
The comments indicate the more detailed constructor and velcharstream
were created to allow parsers to be reused. Does this still work?
(I'm not familiar with that particular part of code). Need to be
careful that we don't introduce scalability issues with this
optimization.
WILL
On 3/13/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote:
Will,
i try to test more JavaCC compiler and do some optimalizations with
code generation.
In Parser.jjt we can remove USER_CHAR_STREAM=true becouse auto
generated SimpleCharStream is almost identical to our
VelocityCharStream. If we decide to do this it's possible in Parser
class to:
- remove VelocityCharStream velcharstream variable - this will reduce
memory use, every VCS has 24kb arrays in memory
- change Parser CTR to simple
public Parser( RuntimeServices rs)
{
/*
* need to call the CTOR first thing.
*/
this( new ByteArrayInputStream("\n".getBytes()));
/*
* and save the RuntimeServices
*/
rsvc = rs;
}
-- and parse method to simple one:
public SimpleNode parse( Reader reader, String templateName )
throws ParseException
{
SimpleNode sn = null;
currentTemplateName = templateName;
try
{
token_source.clearStateVars();
/*
* now reinit the Parser with this CharStream
*/
ReInit(reader);
[..]
catch (Exception e)
{
rsvc.getLog().error("Parser Error: " + templateName, e);
}
currentTemplateName = "";
// finish with input strem
token_source.clearStateVars();
jj_input_stream.inputStream = null; // gc - to unrefernce
passed Reader from Parser class
return sn;
}
- (optional) TokenMgrError can be regenerated to fix some typos in comment
- regenerate javacc code and recompile.
I run test with latest Jira (3.7.4-#189 Standalone/Java6) Velocity
1.6-dev and JavaCC 4.0 and this still works ;) Of course, this need
more test - someone can test this with larger app with more templates?
Sylwester
2007/3/13, Will Glass-Husain <[EMAIL PROTECTED]>:
> Sylvester,
>
> Perhaps RuntimeInstance.parser could call a cleanup method? You can
> insert new methods into Parser.java by putting them in Parser.jjt.
>
> Re: VelocityCharStream. I'm a little leery of lazy inits unless they
> demonstrate significant performance improvements. I've found them to
> be an easy way to convolute the code without always providing
> performance benefits.
>
> WILL
>
> On 3/12/07, Will Glass-Husain <[EMAIL PROTECTED]> wrote:
> > We need to figure out a way to put this in Parser.jj. Parser.java is
> > automatically generated-- it's too complex to generate Parser.java and
> > then modify it.
> >
> > WILL
> >
> > On 3/12/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote:
> > > I do something like this. This works for me, but need some more tests.
> > > Also in method parse() - resource cleanup should be in finally block
> > > (currentTemplateName, token_source, velcharstream).
> > >
> > > In VelocityCharStream class we can also do some rework to lazy init
int,char
> > > buffers
> > >
> > > I'll try to contact JavaCC developers to find where Done() method should
be
> > > called during parsing.
> > >
> > > Sylwester
> > >
> > > Index: Parser.java
> > > ===================================================================
> > > --- Parser.java (revision 517275)
> > > +++ Parser.java (working copy)
> > > @@ -53,8 +53,7 @@
> > > * need to call the CTOR first thing.
> > > */
> > >
> > > - this( new VelocityCharStream(
> > > - new ByteArrayInputStream("\n".getBytes()), 1, 1 ));
> > > + this( (CharStream)null );
> > >
> > > /*
> > > * now setup a VCS for later use
> > > @@ -129,7 +128,8 @@
> > > }
> > >
> > > currentTemplateName = "";
> > > -
> > > + token_source.clearStateVars();
> > > + if (velcharstream!=null) velcharstream.Done();
> > > return sn;
> > > }
> > >
> > >
> > >
> > > 2007/3/12, Ryan Smith <[EMAIL PROTECTED]>:
> > > >
> > > >
> > > > Will,
> > > >
> > > > Where would you put the code to call the Done() method and free up the
> > > > memory?
> > > >
> > > > We currently doing profiling of Velocity as well and are trying to find
> > > > places exactly like this.
> > > >
> > > > Ryan
> > > >
> > > > -----Original Message-----
> > > > From: Will Glass-Husain [mailto:[EMAIL PROTECTED]
> > > > Sent: Monday, March 12, 2007 2:47 PM
> > > > To: Velocity Developers List
> > > > Subject: Re: JavaCC memory problems [was: Re: Problems with build
> > > > Velocity]
> > > >
> > > > Thanks for the tips! Definitely something to investigate.
> > > >
> > > > WILL
> > > >
> > > > On 3/12/07, Sylwester Lachiewicz <[EMAIL PROTECTED]> wrote:
> > > > > Hi,
> > > > >
> > > > > today JavaCC works fine - maybe there was bad day ;)
> > > > >
> > > > > I run profiler with simple velocity app (2 .vm file, simple merge) and
> > > > found
> > > > > that internal by design CharStream has method :
> > > > > ---
> > > > > /**
> > > > > * The lexer calls this function to indicate that it is done with
> > > > the
> > > > > stream
> > > > > * and hence implementations can free any resources held by this
> > > > class.
> > > > > * Again, the body of this function can be just empty and it will
> > > > not
> > > > > * affect the lexer's operation.
> > > > > */
> > > > > void Done();
> > > > > --
> > > > > but this method is never called by any JavaCC code (lexer/parser).
> > > > > In class VelocityCharStream.Done() implementation buffers are
> > > > cleaned.
> > > > >
> > > > > More, in VelocityCharStream constructor we have 3 times allocation of
> > > > 4kb
> > > > > buffer
> > > > > buffer = new char[buffersize];
> > > > > bufline = new int[buffersize];
> > > > > bufcolumn = new int[buffersize];
> > > > >
> > > > > And in class Parser constructor we create 2 times VelocityCharStream.
> > > > > So every Parser allocates about 24kb memory, even when is not needed.
> > > > >
> > > > > When I run trace I can find that memory is deallocated from old
> > > > tempate only
> > > > > on new parse request, but only to this 24kb.
> > > > > In Done method we can also null readerStreem.
> > > > >
> > > > > Mayby this will help improve memory management.
> > > > >
> > > > > Sylwester
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2007/3/12, Will Glass-Husain <[EMAIL PROTECTED]>:
> > > > > >
> > > > > > Hi Sylvester,
> > > > > >
> > > > > > Thanks for the note on the build script. It really needs to check
> > > > for
> > > > > > a min of 1.6.
> > > > > >
> > > > > > I'm not sure why you have problems compiling. I use javacc 3.2 as
> > > > > > well. Did you type "ant clean" first?
> > > > > >
> > > > > > JavaCC generates node files and Parser.java. The node files are not
> > > > > > needed and can be deleted. (I think this happens in the build
> > > > > > script). If you are just changing existing syntax, no manual
> > > > copying
> > > > > > of files is required. If you are adding a new Node you'll have to
> > > > > > move that into the Node directory.
> > > > > >
> > > > > > WILL
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Forio Business Simulations
Will Glass-Husain
[EMAIL PROTECTED]
www.forio.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
