Here is another freshly squeezed^H^H^H^H^H^H^H^Hmade benchmark with comments :) =head2 Reducing the Number of stat() Calls Made by Apache If you watch the system calls that your mod_perl server makes (using I<truss> or I<strace> depending on your OS), while processing a request, you will notice that a few stat() calls are made, which are quite expensive. For example when I fetch http://localhost/perl-status and I have my C<DocumentRoot> set to I</home/httpd/docs> I see: [snip] stat("/home/httpd/docs/perl-status", 0xbffff8cc) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0 [snip] If you have some dynamic content and your virtual relative URI is something like I</news/perl/mod_perl/summary> (i.e., there is no such directory on the web server, the path components are only used for requesting a specific report), this will generate five(!) stat() calls, before the C<DocumentRoot> is found. You will see something like this: stat("/home/httpd/docs/news/perl/mod_perl/summary", 0xbffff744) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs/news/perl/mod_perl", 0xbffff744) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs/news/perl", 0xbffff744) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs/news", 0xbffff744) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0 You can blame the default installed C<TransHandler> for this inefficiency. Of course you could supply your own, which will be smart enough not to look for this virtual path and immediately return C<OK>. But in cases where you have a virtual host that serves only dynamically generated documents, you can override the default C<PerlTransHandler> with this one: <VirtualHost 10.10.10.10:80> ... PerlTransHandler Apache::OK ... </VirtualHost> As you see it affects only this specific virtual host. This has the effect of short circuiting the normal C<TransHandler> processing of trying to find a file system component that matches the given URI -- no more 'stat's! Watching your server under strace/truss can often reveal more performance hits than trying to optimize the code itself! For example unless configured correctly, Apache might look for the I<.htaccess> file in many places, if you don't have one and add many open() calls. Let's start with this simple configuration, and will try to reduce the number of irrelevant system calls. DocumentRoot "/home/httpd/docs" <Location /foo/test> SetHandler perl-script PerlHandler Apache::Foo </Location> The above configuration allows us to make a request to I</foo/test> and the Perl handler() defined in C<Apache::Foo> will be executed. Notice that in the test setup there is no file to be executed (like in C<Apache::Registry>). There is no I<.htaccess> file as well. This is a typical generated trace. stat("/home/httpd/docs/foo/test", 0xbffff8fc) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs/foo", 0xbffff8fc) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0 open("/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory) open("/home/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory) open("/home/httpd/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory) open("/home/httpd/docs/.htaccess", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs/test", 0xbffff774) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0 Now we modify the C<E<lt>DirectoryE<gt>> entry and add S<AllowOverride None>, which among other things disables I<.htaccess> files and will not try to open them. <Directory /> AllowOverride None </Directory> We see that the four open() calls for I<.htaccess> have gone. stat("/home/httpd/docs/foo/test", 0xbffff8fc) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs/foo", 0xbffff8fc) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0 stat("/home/httpd/docs/test", 0xbffff774) = -1 ENOENT (No such file or directory) stat("/home/httpd/docs", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0 Let's try to shortcut the I<foo> location with: Alias /foo / Which makes Apache to look for the file in the I</> directory and not under I</home/httpd/docs/foo>. Let's run it: stat("//test", 0xbffff8fc) = -1 ENOENT (No such file or directory) Wow, we've got only one stat call left! Let's remove the last C<Alias> setting and use: PerlTransHandler Apache::OK as explained above. When we issue the request, we see no stat() calls. But this is possible only if you serve only dynamically generated documents, i.e. no CGI scripts. Otherwise you will have to write your own I<PerlTransHandler> to handle requests as desired. For example this I<PerlTransHandler> will not lookup the file on the file system if the URI starts with I</foo>, but will use the default I<PerlTransHandler> otherwise: PerlTransHandler 'sub { return shift->uri() =~ m|^/foo| \ ? Apache::OK : Apache::DECLINED;}' Let's see the same configuration using the C<E<lt>PerlE<gt>> section and a dedicated package: <Perl> package My::Trans; use Apache::Constants qw(:common); sub handler{ my $r = shift; return OK if $r->uri() =~ m|^/foo|; return DECLINED; } package Apache::ReadConfig; $PerlTransHandler = "My::Trans"; </Perl> As you see we have defined the C<My::Trans> package and implemented the handler() function. Then we have assigned this handler to the C<PerlTransHandler>. Of course you can move the code in the module into an external file, (e.g. I<My/Trans.pm>) and configure the C<PerlTransHandler> with PerlTransHandler My::Trans in the normal way (no C<E<lt>PerlE<gt>> section required. Now as usual let's run the benchmark, we will test all the above solution one by one and in groups. In order to to make the difference of the number of stat calls more prominent we will use a very light handler, that just prints something out. This is the module that we have used: News.pm ------- package News; use Apache::Constants qw(:common); sub handler{ my $r = shift; my $uri = $r->uri; my @sections = split "/",$uri; # in real code you'd do some DB lookup and return the story: # my $story = get_story(@sections); $r->send_http_header('text/plain'); print "Story matching @sections\n"; return OK; } 1; This is the URI we have tested with: /news/perl/mod_perl/summary So the URI is long enough to generate many stat() calls if you didn't prevent that. This is the main configuration: <Location /news> SetHandler perl-script PerlHandler +News </Location> Now we have tried to add different configurations and see how it influences the performance. =over =item 1 Nothing was added. i.e. <Directory /> AllowOverride ... something </Directory> =item 2 Prevent I<.htaccess> lookup: <Directory /> AllowOverride None </Directory> =item 3 Location alias short-cutting: Alias /news / =item 4 Using non-default TransHandler: <Perl> package My::Trans; use Apache::Constants qw(:common); sub handler{ my $r = shift; return OK if $r->uri() =~ m|^/news|; return DECLINED; } package Apache::ReadConfig; $PerlTransHandler = "My::Trans"; </Perl> =back And the results sorted by the Requests Per Second (rps) rate: Options | avtime completed failed rps --------|------------------------------- 2+3 | 27 5000 0 996 2+4 | 29 5000 0 988 4 | 29 5000 0 975 3 | 28 5000 0 974 2 | 32 5000 0 885 1 | 34 5000 0 827 with static arguments: concurrency : 30 connections : 5000 But this doesn't really matter, since what's important is the relative and not the absolute numbers. So we can see that preventing I<.htaccess> lookup improves the performance by about 8% (827 vs 885), using an alias short-cutting (3) or non-default TransHandler (4) gave even more performance boost since for a long URI like we have used, each directory generates a few stat() and open() system calls, the speed up is of about 15% relative to the case 1. And finally grouping the prevention of I<.htaccess> lookup plus one of the techniques (3 or 4) that don't look for the non-existing file on the file system brings the performance boost to about 18% (827 vs 996). As we have seen the number of pseudo-subdirectories is in direct relation with the number of called stat() and open() system calls. To prove it, let's call the configuration, i.e. case 1 and benchmark tree URI with a different number of sections (directories) without counting the base first section (I</news>): Sections URI --------------------------------------------- 1 /news/perl 3 /news/perl/mod_perl/summary 5 /news/perl/mod_perl/summary/foo/bar and the results are just perfect: Sections | avtime completed failed rps --------------------------------------------------- 1 | 33 5000 0 849 3 | 34 5000 0 829 5 | 35 5000 0 801 --------------------------------------------------- You can clearly see that each two sections add one more millisecond to the average processing and connection time, or about about 25 RPS degradation in performance. There is one important thing to tell. If you really think that you can improve the performance by 20% by just adding a few configuration directives, you probably wrong. Remember that in our test code we have used very very light handler, which did nothing but sending a few lines of text without doing any processing. When you use your real code whose run time is not 30-40 milliseconds, but 300-400 milliseconds the improvement of 7 milliseconds on average (the one we saw between case 1 (34ms) and 2+3 (27ms) ) might be very insignificant. This is mostly important to servers who serve millions of requests per day and where each millisecond counts. But even if your server gets a little load, you can still make it a little bit faster. Use the benchmark on the real code and see whether you win something or not. _____________________________________________________________________ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC http://singlesheaven.com http://perlmonth.com http://sourcegarden.org