Re: templating benchmarks...

[Stas Bekman]

On Wed, 13 Jun 2001, Perrin Harkins wrote:

> > > wow. template toolkil took a big hit, there. (no mod_perl on
> > > this list? hmm!)
> >
> > This benchmark can be very non-representive. If you don't know how to
> > optimize each and every "thing" under test, you end up with unfair
> > benchmark and come to potentially wrong conclusions. Take TT, add compiled
> > template caching on the disk and shared TT object and I bet TT won't be at
> > the bottom.
>
> I actually helped Joshua tune the TT example a little, and it using a cached
> Template object and caching the templates used in the test in memory.  The
> "slowness" comes from the fact that it provides a major feature that the
> others don't, and it is being exercised in this test.  The magic dot
> notation which allows templates to say foo.bar.baz, regardless of what kind
> of data structure, object, or code ref "foo", "bar", and "baz" may be takes
> a little more work.  Whether it's a good idea or not is left as an exercise
> to the reader, but I will say this: if Template Toolkit is the bottleneck in
> your app's performance, you have either done some serious tuning or written
> a really simple application (like this benchmark).

I stand corrected.

I've tried to provide a general note of not taking any numbers for
granted. Your explanation is as usual the great one :)

> Nevertheless, it's good to see some numbers, if only to convince Andy to
> finish his optimized XS version of the TT stash.

Yeah, it beats our apps' performance badly, with some 8000 calls to
_dotop()  in TT in some rendered-data-heavy requests, since we use deeply
nested datastructures like foo.bar.baz. I'll probably have to rework this
if Andy doesn't come up with Stash written in XS. If you plan to work with
the current TT and are going to use lots of vars in loops, consider not to
use hashes deeper than one level, and if you still want to use them,
consider reducing the nesting level wherever possible in the templates by
doing [% baz = foo.bar.baz %] before diving into a loop where foo.bar.baz
will be constantly accessed. (of course relevant for loops with many
iterations).

Surpisingly though, TT is faster than pure Perl when it comes to writing
loops rendering data from relatively deeply nested data structures:

Benchmark: timing 2000 iterations of access_perl, access_tt...
access_perl:  7 wallclock secs ( 7.01 usr +  0.00 sys =  7.01 CPU) @
285.31/s (n=2000)
 access_tt:  5 wallclock secs ( 4.19 usr +  0.05 sys =  4.24 CPU) @
471.70/s (n=2000)

Here is the benchmark I've used. (this code is a dream of calendar
programmers, since all months are 31-days long :-) :

use Time::HiRes qw( gettimeofday tv_interval );
use Template;
my $tt = Template->new();

my $data =
{
 month_name => [1..12],
 day_event  => [ map {
 [ map { {time=>1,info=>2} } 1..31 ]
 } 1..12
   ],
};

use Benchmark;
timethese(2000,
  {
   access_perl => \&access_perl,
   access_tt   => \&access_tt,
  });

sub access_tt{
my $output = '';
$tt->process(\*DATA, {data=>$data},\$output);
}

sub access_perl{
my $output = '';
for my $m (0..11){
$output .= "Month: $data->{month_name}[$m]\n";
for my $d (0..30){
my $event = $data->{day_event}[$m][$d];
$output .= "Time: $event->{time}\n";
$output .= "Info: $event->{info}\n";
}
}
}

__DATA__
  [% FOR m = [0..11] %]
  Month: [% data.month_name.$m %]
  [% FOR d = [0..30] %]
  [% event = data.day_event.$m.$d %]
  Time: [% event.time %]
  Info: [% event.info %]
  [% END %]
  [% END %]


Cool, huh?

_
Stas Bekman  JAm_pH --   Just Another mod_perl Hacker
http://stason.org/   mod_perl Guide  http://perl.apache.org/guide
mailto:[EMAIL PROTECTED]   http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/

Re: templating benchmarks...

[Perrin Harkins]

"Tom Lancaster" <[EMAIL PROTECTED]> wrote:
> Absolutely. But I'd like to bring up something I've noticed in
benchmarking
> 'real' sites: many, if not all, of the templating solutions appear to
> parse the whole of an html page. This is at least true of Apache::ASP and
> HTML::Mason, which I have used. Is it not ?

Not really.  They all cache the page in memory.  It is not re-parsed every
time.

> I have produced really dramatic differences in performance in a two-tier
> setup by judicious use of mod_include vs. wholesale proxying of pages
> with dynamic content through to the mod_perl/Apache::ASP server.
[snip]
> Granted, I have other major bottlenecks involved: using Berkeley DB v1.x
> for session state, for one. Perhaps this explains some of it -- maybe the
> proxied header/footer requests never make session calls.

I suspect that it's a combination of the database access and the network
transfer.  There is no difference in the amount of parsing going on, since
it's all cached after the first time (per child).

- Perrin

Re: templating benchmarks...

[Perrin Harkins]

> > wow. template toolkil took a big hit, there. (no mod_perl on
> > this list? hmm!)
>
> This benchmark can be very non-representive. If you don't know how to
> optimize each and every "thing" under test, you end up with unfair
> benchmark and come to potentially wrong conclusions. Take TT, add compiled
> template caching on the disk and shared TT object and I bet TT won't be at
> the bottom.

I actually helped Joshua tune the TT example a little, and it using a cached
Template object and caching the templates used in the test in memory.  The
"slowness" comes from the fact that it provides a major feature that the
others don't, and it is being exercised in this test.  The magic dot
notation which allows templates to say foo.bar.baz, regardless of what kind
of data structure, object, or code ref "foo", "bar", and "baz" may be takes
a little more work.  Whether it's a good idea or not is left as an exercise
to the reader, but I will say this: if Template Toolkit is the bottleneck in
your app's performance, you have either done some serious tuning or written
a really simple application (like this benchmark).

Nevertheless, it's good to see some numbers, if only to convince Andy to
finish his optimized XS version of the TT stash.

- Perrin

Re: templating benchmarks...

[Tom Lancaster]

> This benchmark can be very non-representive. If you don't know how to
> optimize each and every "thing" under test, you end up with unfair
> benchmark and come to potentially wrong conclusions. Take TT, add compiled
> template caching on the disk and shared TT object and I bet TT won't be at
> the bottom.
> 
> In any case always remember that it's extremely hard to run a fair
> benchmark. I'd say it's almost impossible. The only fair benchmarking can
> be done if you know all the in's and out's of the 'things' under test and
> provide many benchmark tests each exploring a single property and not just
> 'one for all' benchmark.
> 
> Of course it's a good thing to have benchmarks, but they all should be
> taken with a grain of salt.
> 

Absolutely. But I'd like to bring up something I've noticed in benchmarking
'real' sites: many, if not all, of the templating solutions appear to 
parse the whole of an html page. This is at least true of Apache::ASP and
HTML::Mason, which I have used. Is it not ?

I have produced really dramatic differences in performance in a two-tier 
setup by judicious use of mod_include vs. wholesale proxying of pages
with dynamic content through to the mod_perl/Apache::ASP server.
For example: 
In a situation with 1 lightweight frontend proxy and two backend 
mod_perl/Apache::ASP app servers ( with load distributed evenly using 
a patched mod_rewrite and its ability to select randomly from a list in a file )
, in one part of the site the dynamic headers and footers are generated by
using , where the file being included is in fact proxied back to the app
servers to receive content; in other parts of the site similarly simple pages
are proxied in their entirety to the app servers.

The results I can produce ( granted only with 'ab' ) are stunningly different:
when I request the header and footer from the app servers using mod_include
plus my modified mod_rewrite ( the stock version refuses to rewrite proxied 
requests ), I get up to 600 requests / second. When proxying the whole page
through I get around 6 requests / second.

Granted, I have other major bottlenecks involved: using Berkeley DB v1.x
for session state, for one. Perhaps this explains some of it -- maybe the 
proxied header/footer requests never make session calls.

I suspect that the wholesale parsing/eval-ing of html pages also plays a part.

What do y'all think ?

Tom

> _



> Stas Bekman  JAm_pH --   Just Another mod_perl Hacker
> http://stason.org/   mod_perl Guide  http://perl.apache.org/guide
> mailto:[EMAIL PROTECTED]   http://apachetoday.com http://eXtropia.com/
> http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
> 

Re: templating benchmarks...

[Stas Bekman]

On Fri, 8 Jun 2001, will trillich wrote:

> On Thu, Jun 07, 2001 at 06:48:38AM +0200, Gerald Richter wrote:
> > > regarding the tools that dovetail into the mod_perl paradigm,
> > > who's got a comparison over relative performance (and other
> > > strengths/weaknesses) of various templating methods?
> >
> > There are various discussions on the mod_perl list about this topic in the
> > past (so take a look at the archives). Also there was an start to write such
> > a comparsion, but I am not aware that anybody has really finished it. The
> > only benchmarks I know are from Joshua. I append his mail below.
> >
> > NOTE: While the hello.xxx benchmarks only prints "Hello world", so they only
> > measures the startup overhead of the toolkit, the h2000.xxx tests tends a
> > little bit more towards a real application.
>
> this is interesting information -- perhaps misleading to use the
> microscopic "hello world" but still it gives a starting point:
>
> here i sorted by hits-per-second
>
> > Test Name  Test File  Hits/sec   Bytes/Hit
> >    -- -- --
> > HTML statichello.html 1158.4 311 bytes
> > mod_include SSIhello.shtm  996.6 198 bytes
> > mod_caucho JSP hello.jsp   860.6 230 bytes
> > mod_perl handler   hello.benc  852.6 196 bytes
> > mod_php PHPhello.php   734.8 225 bytes
> > Apache::Registry v2.01 CGI Raw hello_raw.  706.4 52 bytes
> > Apache::Dispatch v0.08 handler hello/worl  656.1 196 bytes
> > HTML::Template v2.0hello.htmp  567.2 198 bytes
> > Apache::SSI v2.16  hello.shtm  559.4 199 bytes
> > Template v2.00 Toolkit hello.tt522.1 198 bytes
> > Apache::Registry v2.01 CGI.pm  hello.reg   458.5 216 bytes
> > HTML::Embperl v2.0a18  hello.epl   458.2 219 bytes
> > Apache::ASP v2.07  hello.asp   390.6 241 bytes
> > Apache::ePerl  hello.eper  344.8 217 bytes
> > HTML::Mason v0.895 hello.mas   365.3 197 bytes
>
> i bet CGI would be 200.0 or so... ?
>
> and here's the "handler actualy does some work" set:
>
> > mod_caucho JSP 2000h2000.jsp   328.9 28964 byte
> > mod_php PHP 2000   h2000.php   261.8 28865 byte
> > HTML::Embperl v2.0a18 2000 h2000.epl   247.3 28809 byte
> > Apache::ASP v2.07 2000 h2000.asp   228.0 28997 byte
> > HTML::Mason v0.895 2000h2000.mas   222.9 28798 byte
> > Template v2.00 Toolkit 2000h2000.tt 55.6 2 byte
>
> wow. template toolkil took a big hit, there. (no mod_perl on
> this list? hmm!)

This benchmark can be very non-representive. If you don't know how to
optimize each and every "thing" under test, you end up with unfair
benchmark and come to potentially wrong conclusions. Take TT, add compiled
template caching on the disk and shared TT object and I bet TT won't be at
the bottom.

In any case always remember that it's extremely hard to run a fair
benchmark. I'd say it's almost impossible. The only fair benchmarking can
be done if you know all the in's and out's of the 'things' under test and
provide many benchmark tests each exploring a single property and not just
'one for all' benchmark.

Of course it's a good thing to have benchmarks, but they all should be
taken with a grain of salt.

_
Stas Bekman  JAm_pH --   Just Another mod_perl Hacker
http://stason.org/   mod_perl Guide  http://perl.apache.org/guide
mailto:[EMAIL PROTECTED]   http://apachetoday.com http://eXtropia.com/
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/