Re: Apache Hello World Benchmarks Updated
Dave Rolsky wrote: On Mon, 14 Oct 2002, Josh Chamas wrote: This is interesting. I should look into upgrading to perl 5.8 on these tests see what difference there may be. You might also see if it makes a difference if you run the tests for a long enough time. I run them at least 60 seconds for these benchmarks which may make a difference. FYI, I reposted the benchmarks without the MaxRequestsPerChild 100 set for HTML::Mason Template Toolkit, as it was only Embperl 2.x that needed it. Regards, Josh Josh Chamas, Founder phone:925-552-0128 Chamas Enterprises Inc.http://www.chamas.com NodeWorks Link Checkinghttp://www.nodeworks.com
Re: Apache Hello World Benchmarks Updated
Ed wrote: Hi, (as far as i can tell after a quick peek at the code and some debugging) It looks like there is a bug w/ AxKit::run_axkit_engine() and/or Apache::AxKit::Cache::_get_stats() This is really great Ed. Adding the AxGzipOutput On config to the XSLT tests in the benchmark does seem to correctly activate the cache layer, and AxKit now dominates in speed on the larger XSLT transformation! I will leave off the config for the XSP tests since it does not seem to help at all there. I will republish benchmarks with the new configs soon, thanks so much for digging into this. Regards, Josh Josh Chamas, Founder phone:925-552-0128 Chamas Enterprises Inc.http://www.chamas.com NodeWorks Link Checkinghttp://www.nodeworks.com
Re: Apache Hello World Benchmarks Updated
FYI, I reposted the benchmarks without the MaxRequestsPerChild 100 set for HTML::Mason Template Toolkit, as it was only Embperl 2.x that needed it. Embperl 2.0b8 has still some real memory leaks. That's why it called beta. Of course they will be fixed before the final release of 2.0. Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925131 WWW:http://www.ecos.de Fax: +49 6133 925152 -
Apache Hello World Benchmarks Updated
Hey, The Apache Hello World benchmarks are updated at http://chamas.com/bench/ The changes that affect performance numbers include: Set MaxRequestsPerChild to 1000 globally for more realistic run. Set MaxRequestsPerChild to 100 for applications that seem to leak memory which include Embperl 2.0, HTML::Mason, and Template Toolkit. This is a more typical setting in a mod_perl type application that leaks memory, so should be fairly representative benchmark setting. Note that the latter change seemed to have the most benefit for Embperl 2.0, with some benefit for Template Toolkit less ( but some ) for HTML::Mason on the memory usage numbers. Regards, Josh Josh Chamas, Founder phone:925-552-0128 Chamas Enterprises Inc.http://www.chamas.com NodeWorks Link Checkinghttp://www.nodeworks.com
Re: Apache Hello World Benchmarks Updated
Josh Chamas wrote: Set MaxRequestsPerChild to 100 for applications that seem to leak memory which include Embperl 2.0, HTML::Mason, and Template Toolkit. This is a more typical setting in a mod_perl type application that leaks memory, so should be fairly representative benchmark setting. Maybe I'm more careful about memory growth than some people, but 100 sounds a bit on the low side to me. I use Apache::SizeLimit instead of MaxRequestsPerlChild, but I'm pretty sure every child serves more requests than that. Did it seem to affect the performance numbers much? Incidentally, I hate to call this stuff memory leaks, since it's usually just normal growth as memory gets used. Calling it leaks implies that memory is being wasted as a result of mistakes with pointers or some such. When people hear memory leaks they go running for stuff like Apache::Leak, thinking they can find some problem and fix it, which is usually not the case. - Perrin
Re: Apache Hello World Benchmarks Updated
On Tue, 15 Oct 2002, Perrin Harkins wrote: Josh Chamas wrote: Set MaxRequestsPerChild to 100 for applications that seem to leak memory which include Embperl 2.0, HTML::Mason, and Template Toolkit. This is a more typical setting in a mod_perl type application that leaks memory, so should be fairly representative benchmark setting. Maybe I'm more careful about memory growth than some people, but 100 sounds a bit on the low side to me. I use Apache::SizeLimit instead of MaxRequestsPerlChild, but I'm pretty sure every child serves more requests than that. Did it seem to affect the performance numbers much? Incidentally, I hate to call this stuff memory leaks, since it's usually just normal growth as memory gets used. Calling it leaks implies that memory is being wasted as a result of mistakes with pointers or some such. When people hear memory leaks they go running for stuff like Apache::Leak, thinking they can find some problem and fix it, which is usually not the case. I'm fairly sure, FWIW, that Mason does not have any memory leaks, as of 1.12. Pre-1.10 versions do have a _very_ slow memory leak, and 1.10 and 1.11 had that leak plus another, much nastier one. Also FWIW, running Joshua's tests on my machine uniformly used about 1/3 to 1/4 as much RAM, with Mason, Template Toolkit and Apache::ASP. I too am running GNU/Linux, with a 2.4 kernel, on i386 hardware. The only difference in software was that I have Perl 5.8.0 installed, but I'd be incredibly shocked if that explained the difference. I've used Mason on a lot of machines, and I've found that processed between 15-30MB (with mod_perl are relatively normal. While this isn't tiny, it isn't nearly as bad as the 60MB monster Josh found. I'm actually pretty comfortable with Mason's memory usage because I think that certain features it offers explain its memory use. Basically, any framework that offers filtering features on generated output will need to store the entire output in memory before sending it, and I suspect this is one thing I think has a big impact on Mason's memory usage. -dave /*== www.urth.org we await the New Sun ==*/
Re: Apache Hello World Benchmarks Updated
Perrin Harkins wrote: Josh Chamas wrote: Set MaxRequestsPerChild to 100 for applications that seem to leak memory which include Embperl 2.0, HTML::Mason, and Template Toolkit. This is a more typical setting in a mod_perl type application that leaks memory, so should be fairly representative benchmark setting. Maybe I'm more careful about memory growth than some people, but 100 sounds a bit on the low side to me. I use Apache::SizeLimit instead of For the purposes of the benchmark, using Apache::SizeLimit won't do much the worst results are only showing 3M unshared per process 8M total memory per process. The benchmarks run at 20 MaxClients. MaxRequestsPerlChild, but I'm pretty sure every child serves more requests than that. Did it seem to affect the performance numbers much? For Embperl 2.x it made a world of difference on the memory side, with only a slight performance impact of 5-10%, but reduced memory usage from 60M to 25M for the test. Gerald I have found real memory leaks in Embperl 2.x, where this kind of config is necessary. Incidentally, I hate to call this stuff memory leaks, since it's usually just normal growth as memory gets used. Calling it leaks Right... I looked into HTML::Mason today, and agree with you and David that for HTML::Mason there is not a memory leak, it just uses more unshared memory than other environments at runtime, and the MaxRequestsPerChild did not seem to make a difference. This might also be true of Template Toolkit. I'll look into the latter more. I just assumed if the unshared memory footprint was as high as it was for these environments, that there must have been a leak like in Embperl 2.x. I also looked into AxKit which seems to use more memory, but it too does not seem to have a leak. implies that memory is being wasted as a result of mistakes with pointers or some such. When people hear memory leaks they go running for stuff like Apache::Leak, thinking they can find some problem and fix it, which is usually not the case. Right. What I go for in the benchmarks as far as a memory leak goes is do the environments seem to use the same amount of memory at 30 seconds as they do at 60 seconds. In the case of Embperl 2.x, this was very different, but in the case of HTML::Mason, it stays the same. Thanks as always for your feedback. --Josh
Re: Apache Hello World Benchmarks Updated
Dave Rolsky wrote: I'm fairly sure, FWIW, that Mason does not have any memory leaks, as of 1.12. Pre-1.10 versions do have a _very_ slow memory leak, and 1.10 and 1.11 had that leak plus another, much nastier one. Yes, Mason seemed pretty free of leaks when I tested it more today too. Also FWIW, running Joshua's tests on my machine uniformly used about 1/3 to 1/4 as much RAM, with Mason, Template Toolkit and Apache::ASP. I too am running GNU/Linux, with a 2.4 kernel, on i386 hardware. The only difference in software was that I have Perl 5.8.0 installed, but I'd be incredibly shocked if that explained the difference. This is interesting. I should look into upgrading to perl 5.8 on these tests see what difference there may be. You might also see if it makes a difference if you run the tests for a long enough time. I run them at least 60 seconds for these benchmarks which may make a difference. I'm actually pretty comfortable with Mason's memory usage because I think that certain features it offers explain its memory use. Basically, any framework that offers filtering features on generated output will need to store the entire output in memory before sending it, and I suspect this is one thing I think has a big impact on Mason's memory usage. Yep, especially with how perl can store internal data structures relatively inefficiently, where 20K of data represented internally can easily bloat up to 100K internally depending on how its being stored, and matters are probably made worse by code data being intermingled so dirtying the data dirties the code pages to to become unshared. I think that is how Gerald achieves his low memory footprints generally in Embperl, since he is doing most of his stuff in C it seems, and still offering all the advanced features we have in the other application frameworks. Regards, Josh
Re: Apache Hello World Benchmarks Updated
On Mon, 14 Oct 2002, Josh Chamas wrote: This is interesting. I should look into upgrading to perl 5.8 on these tests see what difference there may be. You might also see if it makes a difference if you run the tests for a long enough time. I run them at least 60 seconds for these benchmarks which may make a difference. I ran the Mason 2000 test for 60 seconds and it only used about 20MB. I think that is how Gerald achieves his low memory footprints generally in Embperl, since he is doing most of his stuff in C it seems, and still offering all the advanced features we have in the other application frameworks. It might be worth trying to implement a few parts of Mason in C, particularly the Buffer class, though I'm not really sure I'm the one to do it, given my weak (and without honor) C skills. -dave /*== www.urth.org we await the New Sun ==*/
Re: Apache Hello World Benchmarks Updated
Hi,
(as far as i can tell after a quick peek at the code and some debugging)
It looks like there is a bug w/ AxKit::run_axkit_engine() and/or
Apache::AxKit::Cache::_get_stats()
run_axkit_engine() wants to create a .gzip cachefile when AxGzipOutput is off.
When AxGzipOutput is off the .gzip file is never made and _get_stats()
returns w/ !$self-{file_exists} effectivly disabling delivering cached copies.
With AxGzipOutput enabled both files are created and appropriate cached copies
are delivered as expected.
I haven't decided for myself a best fix except for just enabling AxGzipOutput.
So, I reran hello/bench.pl w/ AxGzipOutput On and sped axkit up quite a bit.
attached are some diffs and a couple of 1 sec bench.pl runs. Would be
interesting to see how axkit compares now?
Thanks,
Ed
On Mon, Oct 14, 2002 at 12:26:06AM -0700, Josh Chamas wrote:
Hey,
The Apache Hello World benchmarks are updated at
http://chamas.com/bench/
The changes that affect performance numbers include:
Set MaxRequestsPerChild to 1000 globally for more realistic run.
Set MaxRequestsPerChild to 100 for applications that seem to leak
memory which include Embperl 2.0, HTML::Mason, and Template Toolkit.
This is a more typical setting in a mod_perl type application that
leaks memory, so should be fairly representative benchmark setting.
Note that the latter change seemed to have the most benefit for Embperl 2.0,
with some benefit for Template Toolkit less ( but some ) for HTML::Mason
on the memory usage numbers.
Regards,
Josh
Josh Chamas, Founder phone:925-552-0128
Chamas Enterprises Inc.http://www.chamas.com
NodeWorks Link Checkinghttp://www.nodeworks.com
--- hello/bench.pl Sun Oct 13 04:07:35 2002
+++ hello-gz/bench.pl Tue Oct 15 00:15:48 2002
-106,7 +106,7
# FIND AB
my $httpd_dir = $HTTPD_DIR;
-$AB = $httpd_dir/ab;
+$AB = '/usr/sbin/ab'; #$httpd_dir/ab;
unless(-x $AB) {
print ab benchmark utility not found at $AB, using 'ab' in PATH\n;
$AB = 'ab';
--- hello/bench.pl Sun Oct 13 04:07:35 2002
+++ hello-gz/bench-gz.plTue Oct 15 00:16:32 2002
-106,7 +106,7
# FIND AB
my $httpd_dir = $HTTPD_DIR;
-$AB = $httpd_dir/ab;
+$AB = '/usr/sbin/ab'; #$httpd_dir/ab;
unless(-x $AB) {
print ab benchmark utility not found at $AB, using 'ab' in PATH\n;
$AB = 'ab';
-583,6 +583,7
AxAddStyleMap application/x-xpathscript
+Apache::AxKit::Language::XPathScript
AxAddProcessor text/xsl hello.xsl
AxCacheDir $TMP/axkit
+ AxGzipOutput On
}],
'AxKit XSLT Big' = ['hxsltbig.xml', qq{
-593,6 +594,7
AxAddStyleMap application/x-xpathscript
+Apache::AxKit::Language::XPathScript
AxAddProcessor text/xsl hxsltbig.xsl
AxCacheDir $TMP/axkit
+ AxGzipOutput On
}],
'AxKit XSP Hello' = ['hello.xsp', qq{
-601,6 +603,7
AxAddStyleMap application/x-xsp +Apache::AxKit::Language::XSP
AxAddProcessor application/x-xsp NULL
AxCacheDir $TMP/axkit
+ AxGzipOutput On
}],
'AxKit XSP 2000' = ['h2000.xsp', qq{
-609,6 +612,7
AxAddStyleMap application/x-xsp +Apache::AxKit::Language::XSP
AxAddProcessor application/x-xsp NULL
AxCacheDir $TMP/axkit
+ AxGzipOutput On
}],
# new Embperl 2.x series
[2002-10-15 00:16:53] Found apache web server at /usr/local/sbin/httpd_perl
[2002-10-15 00:16:53] running 1 groups of benchmarks for 1 seconds
[2002-10-15 00:16:56] testing AxKit v1.6 XSP 2000 at
http://localhost:5000/h2000.xsp?title=Hello%20World%202000integer=2000
[2002-10-15 00:17:11] testing AxKit v1.6 XSP Hello at http://localhost:5000/hello.xsp
[2002-10-15 00:17:25] testing AxKit v1.6 XSLT Hello at http://localhost:5000/hxslt.xml
[2002-10-15 00:17:40] testing AxKit v1.6 XSLT Big at http://localhost:5000/hxsltbig.xml
Test Name Test File Hits/sec # of Hits Time(sec)
secs/Hit Bytes/Hit
- - - - -
- -
AxKit v1.6 XSP 2000 h2000.xsp14.8 20 1.35
0.067600 28680
AxKit v1.6 XSP Hellohello.xsp 245.5261 1.06
0.004073 353
AxKit v1.6 XSLT Hello hxslt.xml 157.6169 1.07
0.006343 331
AxKit v1.6 XSLT Big hxsltbig.x 37.3 38 1.02
0.026816 21590
Apache Server Header Tokens
---
Apache/1.3.26
AxKit/1.6
mod_perl/1.27
PERL Versions: 5.006001
[2002-10-15 00:18:04] Found apache web server at /usr/local/sbin/httpd_perl
[2002-10-15 00:18:04] running 1 groups of benchmarks for 1 seconds
[2002-10-15 00:18:07] testing AxKit v1.6 XSP 2000 at
