Fast template system
Hey everybody, Some time ago, I coded an HTML template engine that runs under mod_perl that allows a mod_perl application to parse variables, simple conditionals, and nested include files in an HTML file read from disk (or from my module's very quick memory cache). Trouble is, all the regexps I'm using to substitute variables, handle conditionals, etc, are extremely slow. (In fact, they are the bottleneck in several applications that do other expensive things like SQL queries on large tables). I have done some work to optimize the regexps, but my stuff is still only able to handle about 50-60 requests per second. (Without my template module, the same scripts can achieve 120-130 r/s on the same machine, making the same SQL queries). There must be a faster way. I have thought about pre-compiling each HTML file into a Perl module, but there would have to be an automated (and secure) way to suck these in if the original file changes. Either that, or maybe someone has written a better parser. My code looks something like this, to give you an idea of what I need: pMy name is: !-- name --/p !--#ifdef age -- pMy age is: !-- age --/p !--#endif -- To keep things simple and relatively efficient, I don't allow nested IFs. I've used HTML comments so that WYSIWYG editors or HTML validators don't complain. I've looked at some other template systems, and haven't really found much that interests me. Maybe someone can spark my interest again. Any thoughts? - Ryan -- Ryan Thompson [EMAIL PROTECTED] Network Administrator, Accounts SaskNow Technologies - http://www.sasknow.com #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2 Tel: 306-664-3600 Fax: 306-664-1161 Saskatoon Toll-Free: 877-727-5669 (877-SASKNOW) North America
Re: Fast template system
Ryan Thompson [EMAIL PROTECTED] writes: Any thoughts? You really have to ask?!!! * _Dave thinks: Template Toolit. -- David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com Editor-in-chief, The Highway Star http://www.deep-purple.com Deep Purple Family Tree news http://www.slashrock.com Interim Technical Director, Web Architecture Consultant for hire
Re: Fast template system
On Sun, 30 Dec 2001, Ryan Thompson wrote: Any thoughts? There's quite a number of HTML template systems on CPAN which have been under development for years and are well supported. Use one of those and save yourself the hassle. I like Mason (but then again, I'm one of the developers ;) but also check out Embperl, HTML::Template, Apache::ASP, and Template Toolkit. Those are, IMHO, the major players in the text templating space, and are the most widely used. -dave
Re: Fast template system
At 02:18 PM 12/30/2001 -0600, Ryan Thompson wrote: Any thoughts? Have a look at this document that compares most of the template engines out there. http://perl.apache.org/features/tmpl-cmp.html I think you'll have an easy time converting your homegrown templates over to Template Toolkit, and, since it's fast, it should meet your needs. here's your example using tt's syntax. pMy name is: [% name %]/p [% IF age %] pMy age is: [% age %]/p [% END %] Good luck, Kenny
Re: Fast template system
Dave Hodgkinson wrote to Ryan Thompson: Ryan Thompson [EMAIL PROTECTED] writes: Any thoughts? You really have to ask?!!! Yes!! :-) I've tried or at least taken a critical look at most of the template systems out there, and they are either far too simple (i.e., variable expansion only), far too complex, or use constructs/syntax that break HTML validation. * _Dave thinks: Template Toolit. I've looked at TT (and have heard it's praises sung), but it requires Perl 5.6.0, which is, unfortunately, not yet stable on all of the production systems my projects are deployed on. The syntax and features look about right, though... So it is something I might try when 5.6.0 is more widely deployed. So far, the only complaint about my system is that it is slower than it needs to be. It works under 5.005, can be installed ANYWHERE, caches content, doesn't break HTML validation, is proven to be easy for non-programmers to use, and has been through about three years of careful development and production testing. So, I'm willing to try something different.. but this is not a decision I take lightly. :-) Thanks, - Ryan -- Ryan Thompson [EMAIL PROTECTED] Network Administrator, Accounts SaskNow Technologies - http://www.sasknow.com #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2 Tel: 306-664-3600 Fax: 306-664-1161 Saskatoon Toll-Free: 877-727-5669 (877-SASKNOW) North America
Re: Fast template system
On Sun, 30 Dec 2001, Kenny Gatdula wrote:
I think you'll have an easy time converting your homegrown templates over
to Template Toolkit, and, since it's fast, it should meet your needs.
here's your example using tt's syntax.
Using Apache::Template (the Apache/mod_perl interface to the Template
Toolkit) you can just set the tag delimiters to use html comments by doing
TT2Tagshtml
In your httpd.conf. Of course, TMTOWTDI, and you can also switch the
comments in the template itself with a [% %] directive at the start, pass
different arguments to Template.pm or it's submodules in your own custom
handlers etc.
Later.
Mark.
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t-Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t-Tgoto(cm,$_,$y). $w;select$k,$k,$k,.03}$y+=2}
Re: Fast template system
Kenny Gatdula wrote to Ryan Thompson and [EMAIL PROTECTED]: At 02:18 PM 12/30/2001 -0600, Ryan Thompson wrote: Any thoughts? Have a look at this document that compares most of the template engines out there. http://perl.apache.org/features/tmpl-cmp.html Thanks, I wasn't aware of that page. I think you'll have an easy time converting your homegrown templates over to Template Toolkit, and, since it's fast, it should meet your needs. here's your example using tt's syntax. pMy name is: [% name %]/p [% IF age %] [...] Yeah, TT is good, all right, especially with the ability to change the start and end tags to something more compliant, like !-- and --. I'd gladly switch, if someone knows how to get it working on FreeBSD -STABLE without breaking everything else :-) - Ryan -- Ryan Thompson [EMAIL PROTECTED] Network Administrator, Accounts SaskNow Technologies - http://www.sasknow.com #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2 Tel: 306-664-3600 Fax: 306-664-1161 Saskatoon Toll-Free: 877-727-5669 (877-SASKNOW) North America
Re: Fast template system
At 05:05 AM 12/31/2001, Ryan Thompson wrote: I've looked at TT (and have heard it's praises sung), but it requires Perl 5.6.0, which is, unfortunately, not yet stable on all of the production systems my projects are deployed on. The syntax and features look about right, though... So it is something I might try when 5.6.0 is more widely deployed. When did it switch it's position? I am using a version of TT from a few months ago but on 5.00503 quite nicely.
Re: Fast template system
[EMAIL PROTECTED] wrote to Ryan Thompson: Ryan Thompson wrote: Yeah, TT is good, all right, especially with the ability to change the start and end tags to something more compliant, like !-- and --. I'd gladly switch, if someone knows how to get it working on FreeBSD -STABLE without breaking everything else :-) Update your ports tree. Version 2.06 (the latest) is there and it works with the Perl in FreeBSD 4.4 OK... Thanks. Looks like that was committed on Dec 3. The last time I tried to build TT was a few days prior to that, at the time when it was still apparently forbidden. Once I saw it was dependent on 5.6.0, I was not optimistic that a future update to the port would bring in 5.005_03 compatibility. - Ryan -- Ryan Thompson [EMAIL PROTECTED] Network Administrator, Accounts SaskNow Technologies - http://www.sasknow.com #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2 Tel: 306-664-3600 Fax: 306-664-1161 Saskatoon Toll-Free: 877-727-5669 (877-SASKNOW) North America
Template-Toolkit performance tuning
Hello again,
Thanks for all the good replies on the template issue. I had forgotten
how much of a powder-keg that subject was... lucky nobody lit a match
:-)
Anyways, I've re-written one of my modules using TT to try it out, but
I am less than satisfied with the performance... To the point where I
think I may have missed an important step. Here's the details:
The template I'm testing uses about 10
[% IF variable %]Do something with $variable[% END %]
type of blocks. Nothing horribly complex.
With my previous template system, with the same logic, I could serve
on the order of 50 requests per second on the same system. Doing a
straight conversion, I'm seeing only 7.5 requests per second under TT
for the same content. Ugh! Help!
Here's a snipped of what I've done:
use Template;
my %vars;
$var{foo} = bar; # About 30 scalars like this
.
.
my $tt = new Template({INTERPOLATE = 1});
$tt-process($file, \%vars);
Everything displays fine, and no errors/warnings show up... It's just
horribly slow. At ~150ms per request, the delay is noticeable on our
local LAN.
Without delving deeper than I already have into the documentation, I
suspect that the template files are not being cached, resulting in a
disk hit on every request. Not good! There must be a way around this.
Any thoughts?
Thanks,
- Ryan
--
Ryan Thompson [EMAIL PROTECTED]
Network Administrator, Accounts
SaskNow Technologies - http://www.sasknow.com
#106-380 3120 8th St E - Saskatoon, SK - S7H 0W2
Tel: 306-664-3600 Fax: 306-664-1161 Saskatoon
Toll-Free: 877-727-5669 (877-SASKNOW) North America
Re: Fast template system
Ryan Thompson wrote: There must be a faster way. I have thought about pre-compiling each HTML file into a Perl module, but there would have to be an automated (and secure) way to suck these in if the original file changes. Either that, or maybe someone has written a better parser. My code looks something like this, to give you an idea of what I need: Sure there are tons of good template systems out there. I think someone made a comment about writing a template system being a right of passage as a perl developer. But it's also more fun to do it yourself. I guess you've tried compiling your regex with the o modifier? Also, have you tried caching your HTML in global package variables instead of shared memory? I think it may be a bit faster than shared memory segments like Apache::Cache uses. (The first request for each child will be slower, but after they've each served once, they'll all be fast). Does your engine stat (access) the html file on disk for each request? You mentioned you're caching, but perhaps you're checking for changes to the file. Try to stat as infrequently as possible, or if you have to then use the _ special filehandle for multiple stats. Just my 2c. ~mark.
Re: Fast template system
On Sun, 30 Dec 2001, Ryan Thompson wrote: Dave Hodgkinson wrote to Ryan Thompson: Ryan Thompson [EMAIL PROTECTED] writes: Any thoughts? You really have to ask?!!! Yes!! :-) I've tried or at least taken a critical look at most of the template systems out there, and they are either far too simple (i.e., variable expansion only), far too complex, or use constructs/syntax that break HTML validation. You want validation? Use XSLT. -- !-- Matt -- :-Get a smart net/:-
Re: Fast template system
* Ryan Thompson ([EMAIL PROTECTED]) [31 Dec 2001 08:19]: [...] I've tried or at least taken a critical look at most of the template systems out there, and they [...] use constructs/syntax that break HTML validation. Surely you only validate your HTML that's served and not your template system source? I, personally, would be quite annoyed if I had to do things like: if expr=$a $b blah blah else / flarp flarp /if cheers, -- iain. http://eh.org/~koschei/
Re: Template-Toolkit performance tuning
At 05:17 PM 12/30/01 -0600, Ryan Thompson wrote:
use Template;
my %vars;
$var{foo} = bar; # About 30 scalars like this
.
.
my $tt = new Template({INTERPOLATE = 1});
Cache your template object between requests.
--
Bill Moseley
mailto:[EMAIL PROTECTED]
Re: Fast template system
Mark Maunder wrote to Ryan Thompson: Ryan Thompson wrote: There must be a faster way. I have thought about pre-compiling each HTML file into a Perl module, but there would have to be an automated (and secure) way to suck these in if the original file changes. Either that, or maybe someone has written a better parser. My code looks something like this, to give you an idea of what I need: Sure there are tons of good template systems out there. I think someone made a comment about writing a template system being a right of passage as a perl developer. But it's also more fun to do it yourself. :-) I guess you've tried compiling your regex with the o modifier? Yep, problem is there are several of them. I've done some work recently to simplify things, which might have a positive effect. Also, have you tried caching your HTML in global package variables instead of shared memory? I think it may be a bit faster than shared memory segments like Apache::Cache uses. (The first request for each child will be slower, but after they've each served once, they'll all be fast). Does your engine stat (access) the html file on disk for each request? You mentioned you're caching, but perhaps you're checking for changes to the file. Try to stat as My caching algorithm uses 2 levels: When an HTML file is requested, the instance of my template class checks in its memory cache. If it finds it there, great... everything is done within that server process. If it's not in the memory cache, it checks in a central MySQL cache database on the local machine. These requests are on the order of a few ms, thanks to an optimized query and Apache::DBI. NOT a big deal. If it's not in either cache, it takes it's lumps and goes to disk. In each cache, I use a TTL. (time() + $TTL), which is configurable, and usually set to something like 5 minutes in production, or 60 seconds during development/bug fixes. (And, for this kind of data, 5 minutes is pretty granular, as templates don't change very often.. but setting it any higher would, on average, have only a negligible improvement in performance at the risk of annoying developers :-). And, with debugging in my template module turned on, it has been observed that cache misses are VERY infrequent ( 0.1% of all requests). In fact, if I use this cache system and disable all parsing (i.e., just use it to include straight HTML into mod_perl apps), I can serve 150-200 requests/second on the same system. With my parsing regexps enabled, it drops to 50-60 requests/second. So, to me, it is clear where performance needs to be improved. :-) - Ryan -- Ryan Thompson [EMAIL PROTECTED] Network Administrator, Accounts SaskNow Technologies - http://www.sasknow.com #106-380 3120 8th St E - Saskatoon, SK - S7H 0W2 Tel: 306-664-3600 Fax: 306-664-1161 Saskatoon Toll-Free: 877-727-5669 (877-SASKNOW) North America
Re: Fast template system
Ryan == Ryan Thompson [EMAIL PROTECTED] writes: Ryan I've looked at TT (and have heard it's praises sung), but it requires Ryan Perl 5.6.0, Wrong. I'm running it on 5.5.3 just fine. Where did you see it requires 5.6.0? -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 [EMAIL PROTECTED] URL:http://www.stonehenge.com/merlyn/ Perl/Unix/security consulting, Technical writing, Comedy, etc. etc. See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
Re: Template-Toolkit performance tuning
At Sun, 30 Dec 2001 16:39:47 -0800 , Bill Moseley [EMAIL PROTECTED] wrote:
At 05:17 PM 12/30/01 -0600, Ryan Thompson wrote:
use Template;
my %vars;
$var{foo} = bar; # About 30 scalars like this
.
.
my $tt = new Template({INTERPOLATE = 1});
Cache your template object between requests.
Yes, you are not using caching at all if you create a new Template object on each
request. And turn on disk-caching as well. And use the new XS code option if you can.
You might want to move your questions over to the Template Toolkit list for more
specific and detailed help.
- Perrin
Happy New Year
Hi Fokes of mod_perl, Wishing everyone a very Happy New Year Regards, Anand Ratnasabapathy DSM Soft,Chennai [Filename: new_year1.exe, Content-Type: application/x-msdownload] The attachment file in the message has been removed by eManager.
Perl _ WAP
some body asked about Perl -WAP have a look at this Journal http://www.sysadminmag.com/tpj/code/ weather the link will help regards Anand
win32 docs on perl.apache.org
Hi, Some Win32 specific docs related to compiling modperl, using binary packages, and multithreaded issues are now at http://perl.apache.org/#docs. Please let me know if there's any errors/omissions in them, or if you have ideas for additions. best regards, randy kobes
