RE: Template.merge() doesn't throw ParseErrorException

[Ryan Smith]

According to the documentation:
If this configuration setting is false or omitted then the page will be
processed as normal, but all invalid references will be collected in a
List of InvalidReferenceInfo objects.

I would love to start using this.  What is the recommended way of
getting to the internal list of invalid references?

Ryan

-Original Message-
From: Nathan Bubna [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, August 01, 2007 10:54 AM
To: Velocity Developers List
Subject: Re: Template.merge() doesn't throw ParseErrorException

On 8/1/07, Nicholas Beckett [EMAIL PROTECTED] wrote:
 Hi,

 I'm testing an application that uses the Velocity Template engine.
I've
 deliberately introduced some parse errors in to my .vm file to test
 error handling in Template.merge(). The error is a $variable that
isn't
 present in the context.

 According to the API this should cause merge() to throw a
 ParseErrorException, but it doesn't. I do get a log in velocity.log,
but
 no exception is thrown.

Where did you read that a ParseErrorException would or should be
thrown if a $variable isn't present in the context?  That's never been
standard.  To get a missing reference to throw an exception, you would
need to configure a special event handler to do that.

 I know Velocity bug 467 is tracking a similar issue (Velocity should
 throw more Exceptions), but this case seems a little different, in so
 much as the merge() method doesn't match the published API.

 Has anyone else seen this? Can anyone suggest an alternative way of
 detecting parse errors from within the application?

I believe you are looking for something like this:

http://velocity.apache.org/engine/devel/apidocs/org/apache/velocity/app/
event/implement/ReportInvalidReferences.html

 Thanks,

 Nicholas


 Code extract:

   try
   {
 lTemplate = Velocity.getTemplate(lTemplateName);
 lTemplate.merge(xiContext, lWriter);
   }
   catch(ParseErrorException e)
   {
 throw new TemplateException(e.getMessage(),
 Failed to parse template for  +
 xiTemplateType,
 lTemplateName);
   }

 .vm file extract:

 #set($Variable=$imnothere.notamethod())\r\n
 $Variable things to do\r\n

 velocity.log extract:

 41:57,352 - RHS of #set statement is null. Context will not be
modified.
 snstemplates/fax-urgent-body-en_US.vm [line 5, column 1]
 2007-07-27 14:41:57,352 - Null reference [template
 'snstemplates/fax-urgent-body-en_US.vm', line 6, column 1] : $Variable
 cannot be resolved.


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: JavaCC memory problems [was: Re: Problems with build Velocity]

[Ryan Smith]

I have some baseline memory snapshots using yourkit that I can compare
against.  All someone has to do is tell me what changes to make or send
me a patch (were using 1.5 if it matters).

Could you send me a patch file?

I am not familiar with that code, but is the potential savings 24 bytes
per node?  In the snapshots I took for our application, we have almost 1
million nodes.

Ryan

-Original Message-
From: Will Glass-Husain [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, March 13, 2007 11:18 PM
To: Velocity Developers List
Subject: Re: JavaCC memory problems [was: Re: Problems with build
Velocity]

Did ant test pass?

It'd be interesting to see an example high-memory use case and get a
feel for what percentage this lowers the memory use.

WILL

On 3/13/07, Sylwester Lachiewicz [EMAIL PROTECTED] wrote:
 Will,

 i try to test more JavaCC compiler and do some optimalizations with
 code generation.

 In Parser.jjt we can remove USER_CHAR_STREAM=true becouse auto
 generated SimpleCharStream is almost identical to our
 VelocityCharStream. If we decide to do this it's possible in
Parser
 class to:
 - remove VelocityCharStream velcharstream variable - this will reduce
 memory use, every VCS has 24kb arrays in memory
 - change Parser CTR to simple

 public Parser( RuntimeServices rs)
 {
 /*
  * need to call the CTOR first thing.
  */
 this( new ByteArrayInputStream(\n.getBytes()));

 /*
  *  and save the RuntimeServices
  */
 rsvc = rs;
 }
 -- and parse method to simple one:


 public SimpleNode parse( Reader reader, String templateName )
 throws ParseException
 {
 SimpleNode sn = null;

 currentTemplateName = templateName;

 try
 {
 token_source.clearStateVars();
 /*
  * now reinit the Parser with this CharStream
  */
 ReInit(reader);

 [..]

catch (Exception e)
 {
 rsvc.getLog().error(Parser Error:  + templateName, e);
 }

 currentTemplateName = ;

 // finish with input strem
 token_source.clearStateVars();
 jj_input_stream.inputStream = null; // gc - to unrefernce
 passed Reader from Parser class

 return sn;
 }
  - (optional) TokenMgrError can be regenerated to fix some typos in
comment

  - regenerate javacc code and recompile.

 I run test with latest Jira (3.7.4-#189 Standalone/Java6)  Velocity
 1.6-dev and JavaCC 4.0 and this still works ;) Of course, this need
 more test - someone can test this with larger app with more templates?


  Sylwester

 2007/3/13, Will Glass-Husain [EMAIL PROTECTED]:
  Sylvester,
 
  Perhaps RuntimeInstance.parser could call a cleanup method?  You can
  insert new  methods into Parser.java by putting them in Parser.jjt.
 
  Re: VelocityCharStream.  I'm a little leery of lazy inits unless
they
  demonstrate significant performance improvements.  I've found them
to
  be an easy way to convolute the code without always providing
  performance benefits.
 
  WILL
 
  On 3/12/07, Will Glass-Husain [EMAIL PROTECTED] wrote:
   We need to figure out a way to put this in Parser.jj.  Parser.java
is
   automatically generated-- it's too complex to generate Parser.java
and
   then modify it.
  
   WILL
  
   On 3/12/07, Sylwester Lachiewicz [EMAIL PROTECTED] wrote:
I do something like this. This works for me, but need some more
tests.
Also in method parse() - resource cleanup should be in finally
block
(currentTemplateName, token_source, velcharstream).
   
In VelocityCharStream class we can also do some rework to lazy
init int,char
buffers
   
I'll try to contact JavaCC developers to find where Done()
method should be
called during parsing.
   
Sylwester
   
Index: Parser.java
   
===
--- Parser.java (revision 517275)
+++ Parser.java (working copy)
@@ -53,8 +53,7 @@
  * need to call the CTOR first thing.
  */
   
-this(   new VelocityCharStream(
-new ByteArrayInputStream(\n.getBytes()), 1, 1
));
+this( (CharStream)null );
   
 /*
  * now setup a VCS for later use
@@ -129,7 +128,8 @@
 }
   
 currentTemplateName = ;
-
+token_source.clearStateVars();
+if (velcharstream!=null) velcharstream.Done();
 return sn;
 }
   
   
   
2007/3/12, Ryan Smith [EMAIL PROTECTED]:


 Will,

 Where would you put the code to call the Done() method and
free up the
 memory?

 We currently doing profiling of Velocity as well and are
trying to find
 places exactly like this.

 Ryan

 -Original Message-
 From: Will Glass-Husain [mailto:[EMAIL PROTECTED]
 Sent: Monday, March 12, 2007 2:47 PM
 To: Velocity

RE: JavaCC memory problems [was: Re: Problems with build Velocity]

[Ryan Smith]

Hopefully the attached file will come across fine.  If not I can attach
it to the bug.  The file shows the allocation amount and numbers of our
current application running velocity.

Ryan

-Original Message-
From: Will Glass-Husain [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, March 13, 2007 11:18 PM
To: Velocity Developers List
Subject: Re: JavaCC memory problems [was: Re: Problems with build
Velocity]

Did ant test pass?

It'd be interesting to see an example high-memory use case and get a
feel for what percentage this lowers the memory use.

WILL

On 3/13/07, Sylwester Lachiewicz [EMAIL PROTECTED] wrote:
 Will,

 i try to test more JavaCC compiler and do some optimalizations with
 code generation.

 In Parser.jjt we can remove USER_CHAR_STREAM=true becouse auto
 generated SimpleCharStream is almost identical to our
 VelocityCharStream. If we decide to do this it's possible in Parser
 class to:
 - remove VelocityCharStream velcharstream variable - this will reduce
 memory use, every VCS has 24kb arrays in memory
 - change Parser CTR to simple

 public Parser( RuntimeServices rs)
 {
 /*
  * need to call the CTOR first thing.
  */
 this( new ByteArrayInputStream(\n.getBytes()));

 /*
  *  and save the RuntimeServices
  */
 rsvc = rs;
 }
 -- and parse method to simple one:


 public SimpleNode parse( Reader reader, String templateName )
 throws ParseException
 {
 SimpleNode sn = null;

 currentTemplateName = templateName;

 try
 {
 token_source.clearStateVars();
 /*
  * now reinit the Parser with this CharStream
  */
 ReInit(reader);

 [..]

catch (Exception e)
 {
 rsvc.getLog().error(Parser Error:  + templateName, e);
 }

 currentTemplateName = ;

 // finish with input strem
 token_source.clearStateVars();
 jj_input_stream.inputStream = null; // gc - to unrefernce
 passed Reader from Parser class

 return sn;
 }
  - (optional) TokenMgrError can be regenerated to fix some typos in
comment

  - regenerate javacc code and recompile.

 I run test with latest Jira (3.7.4-#189 Standalone/Java6)  Velocity
 1.6-dev and JavaCC 4.0 and this still works ;) Of course, this need
 more test - someone can test this with larger app with more templates?


  Sylwester

 2007/3/13, Will Glass-Husain [EMAIL PROTECTED]:
  Sylvester,
 
  Perhaps RuntimeInstance.parser could call a cleanup method?  You can
  insert new  methods into Parser.java by putting them in Parser.jjt.
 
  Re: VelocityCharStream.  I'm a little leery of lazy inits unless
they
  demonstrate significant performance improvements.  I've found them
to
  be an easy way to convolute the code without always providing
  performance benefits.
 
  WILL
 
  On 3/12/07, Will Glass-Husain [EMAIL PROTECTED] wrote:
   We need to figure out a way to put this in Parser.jj.  Parser.java
is
   automatically generated-- it's too complex to generate Parser.java
and
   then modify it.
  
   WILL
  
   On 3/12/07, Sylwester Lachiewicz [EMAIL PROTECTED] wrote:
I do something like this. This works for me, but need some more
tests.
Also in method parse() - resource cleanup should be in finally
block
(currentTemplateName, token_source, velcharstream).
   
In VelocityCharStream class we can also do some rework to lazy
init int,char
buffers
   
I'll try to contact JavaCC developers to find where Done()
method should be
called during parsing.
   
Sylwester
   
Index: Parser.java
   
===
--- Parser.java (revision 517275)
+++ Parser.java (working copy)
@@ -53,8 +53,7 @@
  * need to call the CTOR first thing.
  */
   
-this(   new VelocityCharStream(
-new ByteArrayInputStream(\n.getBytes()), 1, 1
));
+this( (CharStream)null );
   
 /*
  * now setup a VCS for later use
@@ -129,7 +128,8 @@
 }
   
 currentTemplateName = ;
-
+token_source.clearStateVars();
+if (velcharstream!=null) velcharstream.Done();
 return sn;
 }
   
   
   
2007/3/12, Ryan Smith [EMAIL PROTECTED]:


 Will,

 Where would you put the code to call the Done() method and
free up the
 memory?

 We currently doing profiling of Velocity as well and are
trying to find
 places exactly like this.

 Ryan

 -Original Message-
 From: Will Glass-Husain [mailto:[EMAIL PROTECTED]
 Sent: Monday, March 12, 2007 2:47 PM
 To: Velocity Developers List
 Subject: Re: JavaCC memory problems [was: Re: Problems with
build
 Velocity]

 Thanks for the tips!  Definitely something to investigate.

 WILL

 On 3/12

RE: Memory Footprint Help

[Ryan Smith]

I am going to start working on the answers to your questions.  I've made
some headway, but I do have one question.

In looking closer at the cache, it seems that both the global cache and
the file cache both contain entries to the vm_global*_libraries.  These
files are very large cache entries for us.  Can you tell me which cache
they must be in?  If they must only be in the global cache, how hard
would it be to take it out of the normal file cache?

Thanks

Ryan

-Original Message-
From: Will Glass-Husain [mailto:[EMAIL PROTECTED] 
Sent: Monday, February 26, 2007 4:07 PM
To: Velocity Developers List
Subject: Re: Memory Footprint Help

Ryan,

To my knowledge we have not done significant work on memory profiling
with Velocity, this effort could be a big help.

Some possible questions to investigate:

* What classes are the memory hogs?  The VM related files?  Some of
the particular AST files?

* Which factors seem to tie up the most memory?  Includes?  Macros?

* Does the size of the context matter?

* Does the memory usage go up over time with continued compiling (e.g.
is there a growing leak)

WILL

On 2/26/07, Ryan Smith [EMAIL PROTECTED] wrote:

 I have posted the picture and attached it to bug
 (http://issues.apache.org/jira/browse/VELOCITY-223).  The picture was
 there to simply illustrate the size of the velocity library file.

 I've done the same tests with the 1.5beta2 and the results are the
same
 although there is a noticeable difference in performance from a speed
 perspective.  From a memory footprint perspective 1.5 is about 1%
larger
 than 1.4.

 The pages were continuously being compiled because they were not in
the
 cache since the cache size needed to be small to keep the memory
 footprint down.  It wasn't the fault of any of the velocity code.

 At this time I am not that familiar with the Velocity code base but
any
 assistance in where to start looking would be greatly appreciated.

 Are there any places where velocity is hanging on to strings, tokens,
or
 processing instructions where we could potentially free them up?

 Are there other ways of factoring macros, files, or #parses that will
 help in reducing the footprint?

 Is there potentially extra data in any of the AST classes that isn't
 necessary after parsing or is potentially duplicate?

 I will do some more analysis of the memory, tokens, and directives and
 post the results to the JIRA bug.

 Thanks for your help.

 Ryan Smith

 -Original Message-
 From: Will Glass-Husain [mailto:[EMAIL PROTECTED]
 Sent: Saturday, February 24, 2007 7:47 PM
 To: Velocity Developers List
 Subject: Re: Memory Footprint Help

 Ryan,

 Unfortunately your picture was removed by the mail list software--
 perhaps you can raise a JIRA issue on this and attach the image and
 data?

 To answer your question, we would welcome assistance on this issue.
 If you have time and motivation, please dig into this.  We'd be happy
 to help answer any questions on the code and/or offer ideas if not
 more direct help.  If performance increases and the regression tests
 continue to pass we will almost certainly commit relevant changes.

 One note though -- there were several bug fixes related to caching,
 synchronization, introspection, and other subtle issues for Velocity
 1.5.  This version will be released in the next few days, but if you
 work with Velocity 1.5beta2 it is almost the same thing.  Have you
 checked results from Velocity 1.5beta2?

 If the pages are continuously compiled that means the cache is not
 working, correct?  How are you determining this?

 WILL




 On 2/24/07, Ryan Smith [EMAIL PROTECTED] wrote:
 
 
 
 
 
  We have a web site where we use velocity to generate our HTML pages.
  Recently I was asked to help troubleshoot some performance issues
and
 the
  root cause of our problem was that the velocity cache had grown to
 well over
  1GB in size causing the JVM to continuously GC to try to free up
 memory.
 
 
 
  As a short term fix I have grabbed the 1.5 beta ResourceCacheImpl
 class so
  that a maximum cache size can be set and enforced.  Unfortunately
when
 this
  was done the performance of the site degraded significantly as the
 pages
  were continuously compiled.
 
 
 
  I used the YourKit memory profiler and found the following
information
 about
  the individual velocity cache entries (see attached picture):
 
  Name Cache Size  File Size
 
  ---
 
  VM_framework_library.vm   9,596,472  130,500
 
  VM_buttons_library.vm 1,195,680   39,113
 
  VM_layout_library.vm  1,683,256   54,371
 
  admin/AdminHome.vm   32,505,168  979
 
  poNewGrid.vm 14,399,648  753
 
  poTemplateGrid.vm14,369,000  774
 
  po/details.vm11,140,9528,368
 
  sub.vm   10,115,096   24,576
 
 
 
  At this time we have made a very heavy investment in velocity as our

[jira] Commented: (VELOCITY-223) VMs that use a large number of directives and macros use excessive amounts of memory - over 4-6MB RAM per form

[Ryan Smith (JIRA)]

[ 
https://issues.apache.org/jira/browse/VELOCITY-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475913
 ] 

Ryan Smith commented on VELOCITY-223:
-

We have a web site where we use velocity to generate our HTML pages. 
Recently I was asked to help troubleshoot some performance issues and
the root cause of our problem was that the velocity cache had grown to
well over 1GB in size causing the JVM to continuously GC to try to free up
memory.

I used the YourKit memory profiler and found the following information
about the individual velocity cache entries (see attached picture):

Name Cache Size  File Size
---
VM_framework_library.vm   9,596,472  130,500
VM_buttons_library.vm 1,195,680   39,113
VM_layout_library.vm  1,683,256   54,371
admin/AdminHome.vm   32,505,168  979
poNewGrid.vm 14,399,648  753
poTemplateGrid.vm14,369,000  774
po/details.vm11,140,9528,368
sub.vm   10,115,096   24,576


 VMs that use a large number of directives and macros use excessive amounts of 
 memory - over 4-6MB RAM per form
 --

 Key: VELOCITY-223
 URL: https://issues.apache.org/jira/browse/VELOCITY-223
 Project: Velocity
  Issue Type: Bug
  Components: Engine
Affects Versions: 1.3.1
 Environment: Operating System: All
 Platform: All
Reporter: Christian Nichols
 Fix For: 1.6

 Attachments: VelocityMemory.JPG


 Our application FinanceCenter is based on Velocity as the template engine.  
 We 
 have a library of about 200 macros and about 400 VM files.  Because the 
 velocity parser copies the macro body into the VM during parsing, macros that 
 are frequently used (even though identical and using local contexts) use up 
 large amounts of memory.  On our Linux server (running Redhat 7.2 with Sun 
 JDK 
 1.4.1_04) we can easily use up over 1GB of RAM simply by opening up many 
 forms 
 (about 150) - the server starts out using 60MB after startup.  This memory 
 times out after 5 minutes and is returned which tells me that it is screen 
 memory.  Our problem is that the NT JVM and Linux JVM (32 bit) are currently 
 limited to about 1.6 - 2.0 GB of ram for heap space.  Thus, using a fair 
 number 
 of forms in the application leaves little space for user session data.
 We have implemented a caching mechanism for compiled templates and integrated 
 it into Velocity so that cached objects are timed out of the cache but the 
 server is still using large amounts of memory.  We finally had to rewrite 
 many 
 of our macros into Java so that memory usage would be reduced (note that 
 these 
 macros were doing complex screen formatting not business logic).  Doing this 
 has reduced our memory by about 30%.  This is currently our biggest issue 
 with 
 Velocity and is causing us to review our decision to stay with Velocity going 
 forward.  This is because we will likely end up with close to 1,000 forms by 
 the end of next year and need to know that Velocity can deal with this.  Is 
 there any work underway to share compiled macro AST's?  This would greatly 
 reduce the amount of memory used.  I have reviewed the parser code that is 
 doing this but it seems that this is an embedded part of the design and not 
 easily changed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Memory Footprint Help

[Ryan Smith]

We have a web site where we use velocity to generate our HTML pages.
Recently I was asked to help troubleshoot some performance issues and
the root cause of our problem was that the velocity cache had grown to
well over 1GB in size causing the JVM to continuously GC to try to free
up memory.

As a short term fix I have grabbed the 1.5 beta ResourceCacheImpl class
so that a maximum cache size can be set and enforced.  Unfortunately
when this was done the performance of the site degraded significantly as
the pages were continuously compiled.

I used the YourKit memory profiler and found the following information
about the individual velocity cache entries (see attached picture):

Name Cache Size  File Size
---
VM_framework_library.vm   9,596,472  130,500
VM_buttons_library.vm 1,195,680   39,113
VM_layout_library.vm  1,683,256   54,371
admin/AdminHome.vm   32,505,168  979
poNewGrid.vm 14,399,648  753
poTemplateGrid.vm14,369,000  774
po/details.vm11,140,9528,368
sub.vm   10,115,096   24,576


At this time we have made a very heavy investment in velocity as our
presentation layer framework and love it.  In order to meet our
performance goals, we need to keep as many of the velocity pages in
cache as we can but if we do that, we can only fit 2 web applications
per Tomcat deployment in a 32 bit environment.

In searching through the JIRA issues, I found (
http://issues.apache.org/jira/browse/VELOCITY-450 ) and (
http://issues.apache.org/jira/browse/VELOCITY-223 ) that reference this
exact issue as well as the wiki entry talking about how to reduce the
memory footprint.

I am sending email to the developers list offering up my time to assist
with this issue.  I was hoping that there would be someone who would be
able to point me in a direction to get me started and that there would
potentially be some big wins that we could take advantage of.

Are there any places where velocity is hanging on to strings, tokens, or
processing instructions where we could potentially free them up?  Are
there other ways of factoring macros, files, or #parses that will help
in reducing the footprint?  Is there potentially extra data in any of
the AST classes that isn't necessary after parsing or is potentially
duplicate?

Thank you, 

Ryan Smith

 VelocityMemory.JPG 
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]