Fast template system

2001-12-30 Thread Ryan Thompson


Hey everybody,

Some time ago, I coded an HTML template engine that runs under
mod_perl that allows a mod_perl application to parse variables, simple
conditionals, and nested include files in an HTML file read from disk
(or from my module's very quick memory cache).

Trouble is, all the regexps I'm using to substitute variables, handle
conditionals, etc, are extremely slow. (In fact, they are the
bottleneck in several applications that do other expensive things like
SQL queries on large tables). I have done some work to optimize the
regexps, but my stuff is still only able to handle about 50-60
requests per second. (Without my template module, the same scripts can
achieve 120-130 r/s on the same machine, making the same SQL queries).

There must be a faster way. I have thought about pre-compiling each
HTML file into a Perl module, but there would have to be an automated
(and secure) way to suck these in if the original file changes.

Either that, or maybe someone has written a better parser. My code
looks something like this, to give you an idea of what I need:

pMy name is: !-- name --/p
!--#ifdef age --
  pMy age is: !-- age --/p
!--#endif --

To keep things simple and relatively efficient, I don't allow nested
IFs.

I've used HTML comments so that WYSIWYG editors or HTML validators
don't complain.

I've looked at some other template systems, and haven't really found
much that interests me. Maybe someone can spark my interest again.

Any thoughts?

- Ryan

-- 
  Ryan Thompson [EMAIL PROTECTED]
  Network Administrator, Accounts

  SaskNow Technologies - http://www.sasknow.com
  #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2

Tel: 306-664-3600   Fax: 306-664-1161   Saskatoon
  Toll-Free: 877-727-5669 (877-SASKNOW) North America




Re: Fast template system

2001-12-30 Thread Dave Hodgkinson

Ryan Thompson [EMAIL PROTECTED] writes:

 Any thoughts?

You really have to ask?!!!

* _Dave thinks: Template Toolit.


-- 
David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com
Editor-in-chief, The Highway Star   http://www.deep-purple.com
Deep Purple Family Tree news  http://www.slashrock.com
   Interim Technical Director, Web Architecture Consultant for hire



Re: Fast template system

2001-12-30 Thread Dave Rolsky

On Sun, 30 Dec 2001, Ryan Thompson wrote:

 Any thoughts?

There's quite a number of HTML template systems on CPAN which have been
under development for years and are well supported.  Use one of those and
save yourself the hassle.  I like Mason (but then again, I'm one of the
developers ;) but also check out Embperl, HTML::Template, Apache::ASP, and
Template Toolkit.  Those are, IMHO, the major players in the text
templating space, and are the most widely used.


-dave




Re: Fast template system

2001-12-30 Thread Kenny Gatdula

At 02:18 PM 12/30/2001 -0600, Ryan Thompson wrote:
Any thoughts?

Have a look at this document that compares most of the template engines out 
there. http://perl.apache.org/features/tmpl-cmp.html

I think you'll have an easy time converting your homegrown templates over 
to Template Toolkit, and, since it's fast, it should meet your needs.

here's your example using tt's syntax.

pMy name is: [% name %]/p
[% IF age %]
   pMy age is: [% age %]/p
[% END %]

Good luck,
Kenny




Re: Fast template system

2001-12-30 Thread Ryan Thompson

Dave Hodgkinson wrote to Ryan Thompson:

 Ryan Thompson [EMAIL PROTECTED] writes:

  Any thoughts?

 You really have to ask?!!!

Yes!! :-)

I've tried or at least taken a critical look at most of the template
systems out there, and they are either far too simple (i.e., variable
expansion only), far too complex, or use constructs/syntax that break
HTML validation.


 * _Dave thinks: Template Toolit.

I've looked at TT (and have heard it's praises sung), but it requires
Perl 5.6.0, which is, unfortunately, not yet stable on all of the
production systems my projects are deployed on. The syntax and
features look about right, though... So it is something I might try
when 5.6.0 is more widely deployed.

So far, the only complaint about my system is that it is slower than
it needs to be. It works under 5.005, can be installed ANYWHERE,
caches content, doesn't break HTML validation, is proven to be easy
for non-programmers to use, and has been through about three years of
careful development and production testing. So, I'm willing to try
something different.. but this is not a decision I take lightly. :-)

Thanks,
- Ryan

-- 
  Ryan Thompson [EMAIL PROTECTED]
  Network Administrator, Accounts

  SaskNow Technologies - http://www.sasknow.com
  #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2

Tel: 306-664-3600   Fax: 306-664-1161   Saskatoon
  Toll-Free: 877-727-5669 (877-SASKNOW) North America




Re: Fast template system

2001-12-30 Thread Mark Fowler

On Sun, 30 Dec 2001, Kenny Gatdula wrote:

 I think you'll have an easy time converting your homegrown templates over 
 to Template Toolkit, and, since it's fast, it should meet your needs.
 
 here's your example using tt's syntax.

Using Apache::Template (the Apache/mod_perl interface to the Template 
Toolkit) you can just set the tag delimiters to use html comments by doing

 TT2Tagshtml

In your httpd.conf.  Of course, TMTOWTDI, and you can also switch the 
comments in the template itself with a [% %] directive at the start, pass 
different arguments to Template.pm or it's submodules in your own custom 
handlers etc.

Later.

Mark.

-- 
s''  Mark Fowler London.pm   Bath.pm
 http://www.twoshortplanks.com/  [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t-Tputs(cl);for$w(split/  +/
){for(0..30){$|=print$t-Tgoto(cm,$_,$y). $w;select$k,$k,$k,.03}$y+=2}





Re: Fast template system

2001-12-30 Thread Ryan Thompson

Kenny Gatdula wrote to Ryan Thompson and [EMAIL PROTECTED]:

 At 02:18 PM 12/30/2001 -0600, Ryan Thompson wrote:
 Any thoughts?

 Have a look at this document that compares most of the template
 engines out there. http://perl.apache.org/features/tmpl-cmp.html

Thanks, I wasn't aware of that page.


 I think you'll have an easy time converting your homegrown
 templates over to Template Toolkit, and, since it's fast, it
 should meet your needs.

 here's your example using tt's syntax.

 pMy name is: [% name %]/p
 [% IF age %]
 [...]

Yeah, TT is good, all right, especially with the ability to change the
start and end tags to something more compliant, like !-- and --.

I'd gladly switch, if someone knows how to get it working on FreeBSD
-STABLE without breaking everything else :-)

- Ryan

-- 
  Ryan Thompson [EMAIL PROTECTED]
  Network Administrator, Accounts

  SaskNow Technologies - http://www.sasknow.com
  #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2

Tel: 306-664-3600   Fax: 306-664-1161   Saskatoon
  Toll-Free: 877-727-5669 (877-SASKNOW) North America




Re: Fast template system

2001-12-30 Thread Gunther Birznieks

At 05:05 AM 12/31/2001, Ryan Thompson wrote:

I've looked at TT (and have heard it's praises sung), but it requires
Perl 5.6.0, which is, unfortunately, not yet stable on all of the
production systems my projects are deployed on. The syntax and
features look about right, though... So it is something I might try
when 5.6.0 is more widely deployed.

When did it switch it's position? I am using a version of TT from a few 
months ago but on 5.00503 quite nicely.





Re: Fast template system

2001-12-30 Thread Ryan Thompson

[EMAIL PROTECTED] wrote to Ryan Thompson:

 Ryan Thompson wrote:

  Yeah, TT is good, all right, especially with the ability to change the
  start and end tags to something more compliant, like !-- and --.
 
  I'd gladly switch, if someone knows how to get it working on FreeBSD
  -STABLE without breaking everything else :-)

 Update your ports tree. Version 2.06 (the latest) is there and it
 works with the Perl in FreeBSD 4.4

OK... Thanks. Looks like that was committed on Dec 3. The last time I
tried to build TT was a few days prior to that, at the time when it
was still apparently forbidden.

Once I saw it was dependent on 5.6.0, I was not optimistic that a
future update to the port would bring in 5.005_03 compatibility.

- Ryan

-- 
  Ryan Thompson [EMAIL PROTECTED]
  Network Administrator, Accounts

  SaskNow Technologies - http://www.sasknow.com
  #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2

Tel: 306-664-3600   Fax: 306-664-1161   Saskatoon
  Toll-Free: 877-727-5669 (877-SASKNOW) North America




Template-Toolkit performance tuning

2001-12-30 Thread Ryan Thompson


Hello again,

Thanks for all the good replies on the template issue. I had forgotten
how much of a powder-keg that subject was... lucky nobody lit a match
:-)

Anyways, I've re-written one of my modules using TT to try it out, but
I am less than satisfied with the performance... To the point where I
think I may have missed an important step. Here's the details:

The template I'm testing uses about 10
[% IF variable %]Do something with $variable[% END %]
type of blocks. Nothing horribly complex.

With my previous template system, with the same logic, I could serve
on the order of 50 requests per second on the same system. Doing a
straight conversion, I'm seeing only 7.5 requests per second under TT
for the same content. Ugh! Help!

Here's a snipped of what I've done:

use Template;

my %vars;

$var{foo} = bar;   # About 30 scalars like this
.
.

my $tt = new Template({INTERPOLATE = 1});
$tt-process($file, \%vars);

Everything displays fine, and no errors/warnings show up... It's just
horribly slow. At ~150ms per request, the delay is noticeable on our
local LAN.

Without delving deeper than I already have into the documentation, I
suspect that the template files are not being cached, resulting in a
disk hit on every request. Not good! There must be a way around this.

Any thoughts?

Thanks,
- Ryan

-- 
  Ryan Thompson [EMAIL PROTECTED]
  Network Administrator, Accounts

  SaskNow Technologies - http://www.sasknow.com
  #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2

Tel: 306-664-3600   Fax: 306-664-1161   Saskatoon
  Toll-Free: 877-727-5669 (877-SASKNOW) North America




Re: Fast template system

2001-12-30 Thread Mark Maunder

Ryan Thompson wrote:

 There must be a faster way. I have thought about pre-compiling each
 HTML file into a Perl module, but there would have to be an automated
 (and secure) way to suck these in if the original file changes.

 Either that, or maybe someone has written a better parser. My code
 looks something like this, to give you an idea of what I need:


Sure there are tons of good template systems out there. I think someone
made a comment about writing a template system being a right of passage as
a perl developer. But it's also more fun to do it yourself.

I guess you've tried compiling your regex with the o modifier? Also, have
you tried caching your HTML in global package variables instead of shared
memory? I think it may be a bit faster than shared memory segments like
Apache::Cache uses. (The first request for each child will be slower, but
after they've each served once, they'll all be fast). Does  your engine
stat (access) the html file on disk for each request? You mentioned you're
caching, but perhaps you're checking for changes to the file. Try to stat
as infrequently as possible, or if you have to then use the _ special
filehandle for multiple stats.

Just my 2c.

~mark.




Re: Fast template system

2001-12-30 Thread Matt Sergeant

On Sun, 30 Dec 2001, Ryan Thompson wrote:

 Dave Hodgkinson wrote to Ryan Thompson:

  Ryan Thompson [EMAIL PROTECTED] writes:
 
   Any thoughts?
 
  You really have to ask?!!!

 Yes!! :-)

 I've tried or at least taken a critical look at most of the template
 systems out there, and they are either far too simple (i.e., variable
 expansion only), far too complex, or use constructs/syntax that break
 HTML validation.

You want validation? Use XSLT.

-- 
!-- Matt --
:-Get a smart net/:-




Re: Fast template system

2001-12-30 Thread iain truskett

* Ryan Thompson ([EMAIL PROTECTED]) [31 Dec 2001 08:19]:

[...]
 I've tried or at least taken a critical look at most of the template
 systems out there, and they [...] use constructs/syntax that break
 HTML validation.

Surely you only validate your HTML that's served and not your template
system source?

I, personally, would be quite annoyed if I had to do things like:

if expr=$a  $b
blah blah
else /
flarp flarp
/if


cheers,
-- 
iain.  http://eh.org/~koschei/



Re: Template-Toolkit performance tuning

2001-12-30 Thread Bill Moseley

At 05:17 PM 12/30/01 -0600, Ryan Thompson wrote:
   use Template;

   my %vars;

   $var{foo} = bar;   # About 30 scalars like this
   .
   .

   my $tt = new Template({INTERPOLATE = 1});

Cache your template object between requests.


-- 
Bill Moseley
mailto:[EMAIL PROTECTED]



Re: Fast template system

2001-12-30 Thread Ryan Thompson

Mark Maunder wrote to Ryan Thompson:

 Ryan Thompson wrote:

  There must be a faster way. I have thought about pre-compiling each
  HTML file into a Perl module, but there would have to be an automated
  (and secure) way to suck these in if the original file changes.
 
  Either that, or maybe someone has written a better parser. My code
  looks something like this, to give you an idea of what I need:

 Sure there are tons of good template systems out there. I think
 someone made a comment about writing a template system being a
 right of passage as a perl developer. But it's also more fun to do
 it yourself.

:-)


 I guess you've tried compiling your regex with the o modifier?

Yep, problem is there are several of them. I've done some work
recently to simplify things, which might have a positive effect.


 Also, have you tried caching your HTML in global package variables
 instead of shared memory?  I think it may be a bit faster than
 shared memory segments like Apache::Cache uses. (The first request
 for each child will be slower, but after they've each served once,
 they'll all be fast). Does your engine stat (access) the html file
 on disk for each request? You mentioned you're caching, but
 perhaps you're checking for changes to the file. Try to stat as

My caching algorithm uses 2 levels:

When an HTML file is requested, the instance of my template class
checks in its memory cache. If it finds it there, great... everything
is done within that server process.

If it's not in the memory cache, it checks in a central MySQL cache
database on the local machine. These requests are on the order of a
few ms, thanks to an optimized query and Apache::DBI. NOT a big deal.

If it's not in either cache, it takes it's lumps and goes to disk.

In each cache, I use a TTL. (time() + $TTL), which is configurable,
and usually set to something like 5 minutes in production, or 60
seconds during development/bug fixes. (And, for this kind of data, 5
minutes is pretty granular, as templates don't change very often.. but
setting it any higher would, on average, have only a negligible
improvement in performance at the risk of annoying developers :-).

And, with debugging in my template module turned on, it has been
observed that cache misses are VERY infrequent ( 0.1% of all
requests).

In fact, if I use this cache system and disable all parsing (i.e.,
just use it to include straight HTML into mod_perl apps), I can serve
150-200 requests/second on the same system.

With my parsing regexps enabled, it drops to 50-60 requests/second.

So, to me, it is clear where performance needs to be improved. :-)

- Ryan

-- 
  Ryan Thompson [EMAIL PROTECTED]
  Network Administrator, Accounts

  SaskNow Technologies - http://www.sasknow.com
  #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2

Tel: 306-664-3600   Fax: 306-664-1161   Saskatoon
  Toll-Free: 877-727-5669 (877-SASKNOW) North America




Re: Fast template system

2001-12-30 Thread Randal L. Schwartz

 Ryan == Ryan Thompson [EMAIL PROTECTED] writes:

Ryan I've looked at TT (and have heard it's praises sung), but it requires
Ryan Perl 5.6.0,

Wrong.  I'm running it on 5.5.3 just fine.  Where did you see it
requires 5.6.0?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
[EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!



Re: Template-Toolkit performance tuning

2001-12-30 Thread perrin

At Sun, 30 Dec 2001 16:39:47 -0800 , Bill Moseley [EMAIL PROTECTED] wrote: 

At 05:17 PM 12/30/01 -0600, Ryan Thompson wrote:
  use Template;

  my %vars;

  $var{foo} = bar;   # About 30 scalars like this
  .
  .

  my $tt = new Template({INTERPOLATE = 1});

Cache your template object between requests.

Yes, you are not using caching at all if you create a new Template object on each 
request.  And turn on disk-caching as well.  And use the new XS code option if you can.

You might want to move your questions over to the Template Toolkit list for more 
specific and detailed help.

- Perrin



Happy New Year

2001-12-30 Thread Anand Ratnasabapathy

Hi Fokes of mod_perl,

Wishing everyone a very Happy New Year


Regards,
Anand Ratnasabapathy
DSM Soft,Chennai

[Filename: new_year1.exe, Content-Type: application/x-msdownload]
The attachment file in the message has been removed by eManager.



Perl _ WAP

2001-12-30 Thread Anand Ratnasabapathy

some body asked about Perl -WAP

have a look at this Journal

http://www.sysadminmag.com/tpj/code/


weather the link will help

regards
Anand 



win32 docs on perl.apache.org

2001-12-30 Thread Randy Kobes

Hi,
   Some Win32 specific docs related to compiling modperl,
using binary packages, and multithreaded issues are now
at http://perl.apache.org/#docs. Please let me know if
there's any errors/omissions in them, or if you have ideas
for additions.

best regards,
randy kobes