Re: Apache::DBI strategy/philosophy
Part of the problem I've had is transitioning a cgi application onto mod_perl and keeping performance up on both platforms until a switch can be made. So I've done some comparisons of the various dbh schemes. Comments and corrections very welcome. I'm wary of connect_cached() under mod_perl but perhaps for no good reason. DBI Connection Tests Since connecting can be expensive, you generally just connect at the start of your program and disconnect at the end. DBI.pm 1.14 docs For programmer ease you probally want to connect inside a module, and for mod_perl speed you want to preload that module. But you can't do both AND connect once at the start. Mod_perl insists that dbh connections must not be made in the parent and then used in the child: that means that you cannot initialize your dbh in BEGIN or outside of a subroutine. The trickier problem involves a popular hack '$dbh ||= connect()', in CGI this will work effectively, however in mod_perl the variable $dbh is persistent between invocations. Should your database vanish, the connection within $dbh will be invalid, but the variable $dbh will still be a TRUE value and all of your queries will fail until you restart apache (ie: ||= doesn't call ping()). This benchmark compares some different techiques of DBI connection and discusses their merits. Each set of DBI connections is done inside a module. Each module calls a number of subroutines, all of which perform database operations. There are 14 connect and execute calls over two different database handles. Run from the command line Benchmark: timing 20 iterations of 0 my, 1 local, 2 or, 3 global, 4 cache, 5 cache my... 0 my: 55 wallclock secs ( 5.47 usr 3.44 sys + 22.36 cusr 21.31 csys = 52.58 CPU) @ 2.24/s (n=20) 1 local: 52 wallclock secs ( 5.54 usr 3.48 sys + 20.86 cusr 19.99 csys = 49.87 CPU) @ 2.22/s (n=20) 2 or: 12 wallclock secs ( 2.13 usr 0.74 sys + 2.58 cusr 3.08 csys = 8.53 CPU) @ 6.97/s (n=20) 3 global: 12 wallclock secs ( 2.11 usr 0.84 sys + 2.60 cusr 3.04 csys = 8.59 CPU) @ 6.78/s (n=20) 4 cache: 13 wallclock secs ( 2.80 usr 0.91 sys + 2.73 cusr 2.94 csys = 9.38 CPU) @ 5.39/s (n=20) 5 cache my: 13 wallclock secs ( 2.82 usr 0.89 sys + 2.87 cusr 2.77 csys = 9.35 CPU) @ 5.39/s (n=20) Run under CGI Benchmark: timing 20 iterations of 0 my, 1 local, 2 or, 3 global, 4 cache, 5 cache my... 0 my: 54 wallclock secs ( 5.31 usr 2.90 sys + 22.34 cusr 21.20 csys = 51.75 CPU) @ 2.44/s (n=20) 1 local: 51 wallclock secs ( 5.49 usr 2.88 sys + 20.78 cusr 19.96 csys = 49.11 CPU) @ 2.39/s (n=20) 2 or: 12 wallclock secs ( 2.35 usr 0.63 sys + 2.74 cusr 2.82 csys = 8.54 CPU) @ 6.71/s (n=20) 3 global: 12 wallclock secs ( 2.28 usr 0.68 sys + 2.59 cusr 2.86 csys = 8.41 CPU) @ 6.76/s (n=20) 4 cache: 13 wallclock secs ( 2.88 usr 0.76 sys + 2.82 cusr 2.86 csys = 9.32 CPU) @ 5.49/s (n=20) 5 cache my: 13 wallclock secs ( 2.99 usr 0.67 sys + 2.84 cusr 2.77 csys = 9.27 CPU) @ 5.46/s (n=20) Run under Apache::PerlRun and NO Apache::DBI Benchmark: timing 20 iterations of 0 my, 1 local, 2 or, 3 global, 4 cache, 5 cache my... 0 my: 55 wallclock secs ( 5.43 usr 3.23 sys + 21.99 cusr 22.04 csys = 52.69 CPU) @ 2.31/s (n=20) 1 local: 53 wallclock secs ( 5.49 usr 3.35 sys + 21.34 cusr 19.83 csys = 50.01 CPU) @ 2.26/s (n=20) 2 or: 12 wallclock secs ( 2.12 usr 0.67 sys + 2.85 cusr 2.82 csys = 8.46 CPU) @ 7.17/s (n=20) 3 global: 12 wallclock secs ( 2.12 usr 0.74 sys + 2.69 cusr 2.88 csys = 8.43 CPU) @ 6.99/s (n=20) 4 cache: 13 wallclock secs ( 2.89 usr 0.80 sys + 2.67 cusr 2.93 csys = 9.29 CPU) @ 5.42/s (n=20) 5 cache my: 12 wallclock secs ( 3.08 usr 0.72 sys + 2.61 cusr 2.86 csys = 9.27 CPU) @ 5.26/s (n=20) Run under Apache::PerlRun and Apache::DBI Benchmark: timing 20 iterations of 0 my, 1 local, 2 or, 3 global, 4 cache, 5 cache my... 0 my: 12 wallclock secs ( 2.29 usr 0.58 sys + 2.64 cusr 2.82 csys = 8.33 CPU) @ 6.97/s (n=20) 1 local: 12 wallclock secs ( 2.43 usr 0.61 sys + 2.61 cusr 3.02 csys = 8.67 CPU) @ 6.58/s (n=20) 2 or: 11 wallclock secs ( 2.14 usr 0.69 sys + 2.98 cusr 2.95 csys = 8.76 CPU) @ 7.07/s (n=20) 3 global: 12 wallclock secs ( 2.16 usr 0.66 sys + 2.61 cusr 2.86 csys = 8.29 CPU) @ 7.09/s (n=20) 4 cache: 12 wallclock secs ( 2.72 usr 0.74 sys + 2.80 cusr 2.81 csys = 9.07 CPU) @ 5.78/s (n=20) 5 cache my: 13 wallclock secs ( 2.97 usr 0.76 sys + 2.79 cusr 2.88 csys = 9.40 CPU) @ 5.36/s (n=20) EXPLANATION OF THE TESTS my: my dbh are created within the subroutines. CGI: slow due to reconnections. MOD_PERL: safe for preloading. local: Global dbh are initialized within the subroutines. CGI: slow due to reconnections. MOD_PERL: safe for preloading. or: Global dbh are initialized if false using ||= CGI: fast due to single variable connections. MOD_PERL: dbh
mod_perl caches compiled quotes?
Hi, I'm seeing some very disturbing behavior while running mod_perl. I'm sure there must be a workaround for this. Sorry if this is a FAQ, but I haven't seen anything discussing this. I use mod_perl to generate pages based on a user from cookies and a database (DBI/MySql). Use strict is in effect everywhere (I've checked). I print this data using syntax of the form: print qq( Your personal details are $name $salary); (Simplified obviously, but you get the idea). This works fine the first time, but the person who accesses the page frequently gets the details of the last person to access the server. Needless to say this is a Bad Thing. Is it the case that the results of the string interpolation are cached and used in the next run of the script? This would seem to be the case, as touching the script or restarting the server seems to remove the problem, until the next run. I've attached some version info below. Apache/1.3.12 (Unix) mod_perl/1.23 mod_ssl/2.6.3 OpenSSL/0.9.5 Perl: 5.005_03 Redhat6.1 However, I tried compiling up to the latest versions (including perl 5.6.0) on a development machine and saw the same behavior. What am I doing wrong? Thanks, Steve
ANNOUNCE: Apache::AuthCookie 2.011
Hi, The URL http://mathforum.com/~ken/modules/archive/Apache-AuthCookie-2.011.tar.gz has entered CPAN as file: $CPAN/authors/id/K/KW/KWILLIAMS/Apache-AuthCookie-2.011.tar.gz size: 18530 bytes md5: b4617ac3ed237e2ae5123e9c7756fc82 This release follows right on the heels of 2.010. Here's the change: Version: 2.011 Date: 2000/06/17 08:01:19 Several people have requested a mechanism for the login script to be able to tell the difference between a failed login and a first attempt at a login. Typically one can figure this out by whether a cookie has been sent or not, so I've put some information to that effect in $r-prev-subprocess_env(). See the login script docs for specifics. I also cleaned up the documentation for the login script. ------ Ken Williams Last Bastion of Euclidity [EMAIL PROTECTED]The Math Forum
Re: IPC::Open2 v5.6.0 failures.
[EMAIL PROTECTED] wrote:
Is this a know bug? Is there a bug?
Not that I know of, did you check the archives?
Is it right in front of my face?
Something seems odd to me, see below, but it shouldn't be causing you
too much grief.
First, to educate me, what's encrypt do differently than Perl's crypt?
why not
crypt $pass, salt() ;
?
PLUG MODULE="IPC::Run" AUTHOR="me" MODE="shameless"
If you want to play with it, IPC::Run's up on CPAN and this might make
your routine simpler and more debuggable. You'd do something like this to
use it (untested):
use IPC::Run qw( run ) ;
sub encrypt {
my $pass = shift;
my $crypt;
run [qw(/usr/bin/encrypt', -b), salt()], \$pass, \$crypt
or die "Can't run '/usr/bin/encrypt': $!" ;
chomp $crypt;
return $crypt ;
}
The semantic of run's return value are what you, or at least I, would
expect. Then, to get a clue about what's happening under the hood, just
add a {debug=1} to run's params:
run [qw(/usr/bin/encrypt', -b), salt()], \$pass, \$crypt, {debug=1}
or die "Can't run '/usr/bin/encrypt': $!" ;
and see what happens. Increase the debug level to 2 or 3 if need be,
and it probably will be.
This is a beta module though, so there might be gotchas. One is that I'm
soon going to remove '{'...'}' hash constructors around the debug=2.
/PLUG
Here's what seems odd:
open2(\*ENCRD, \*ENCWR, "/usr/bin/encrypt $args")
or die("Can't open2: '/usr/bin/encrypt': $!");
From the IPC::Open2 man page:
SYNOPSIS
use IPC::Open2;
$pid = open2(\*RDRFH, \*WTRFH, 'some cmd and args');
# or without using the shell
snip
open2() returns the process ID of the child process. It
doesn't return on failure: it just raises an exception
matching `/^open2:/'. However, `exec' failures in the
child are not detected. You'll have to trap SIGPIPE your-
self.
I don't see how this could this be contributing to your grief, though.
I take it there's no spoor in the error_log?
As a performance note, have you considered passing the command line as
a list like
qw(/usr/bin/encrypt -b ), salt )
in place of the string? That way the exec in open2 is not invoking
a subshell. Also, if $pass comes from a user, what if they type in
`mail [EMAIL PROTECTED] /etc/passwd`?
- Barrie
Highly optimized mod_perl ?
Hi! Does anyone know the way to highly optimize mod_perl for speed? Any hacks, advices? All standard advices read from help are done... Best regards, Vlad Safronov, mailto: [EMAIL PROTECTED]
Segfault Apache1.3.12/mod_perl1.24/Solaris2.6
Hi all, Sorry, this is a bit long. I'm working on a bunch of machines running Solaris 2.6 on which I have installed Apache+mod_perl static. The machines are set up for just a couple of virtual hosts at present for testing, eventually they will be load-balanced with mod_perl on separate machines and will host many sites. I downloaded compiled new sources for the latest (not CVS) versions of Apache 1.3.12, mod_perl 1.24, mod_macro 1.1.1 and Perl 5.005_03. The compiler is gcc 2.91.66 and was used for all the compilations. I set up a plain/mod_perl Apache pair on one of the machines. The proxying worked fine and I could get server-info and server-status OK from the mod_perl Apache. I left it at that and went on to other things. Unfortunately when the development department started working on the scripting a couple of days later (yesterday, Friday) the mod_perl Apache started segfaulting as soon as they tried to get it to serve a request from their test handler. Here's the error_log message: [Fri Jun 16 17:20:21 2000] [notice] \ child pid 22310 exit signal Segmentation Fault (11) There is no core dump and the same thing happens with the -X switch. The first Perl had thread support and dynamic linking, so I recompiled it twice, once with no thread support and once without dynamic linking but the same thing happened. Below are the makepl_args.mod_perl, the output of perl -V, httpd_perl -l, perl.conf, startup.pl and test.pm which I just noticed contains a call to Apache::Request-new() and I don't know why. I don't think that should make Apache segfault... I searched the archives and there are some Solaris segfaults but nothing seems to fit this simple case. I don't have access to the machines at the weekend so I've posted this hoping it might ring some bells somewhere before I try again next week maybe with the CVS version of mod_perl 1.24 and maybe even with 1.21 and 1.3.9 just to see what happens. I'll also get rid of that Apache::Request call and try things like Apache::Registry scripts to see what happens there. Any input gratefully received. Especially by Leo, who gets it in the neck if the machines aren't ready to go live next Friday... 73, Ged. == makepl_args.mod_perl -- USE_APACI=1 APACHE_PREFIX=/usr/local/apache/httpd_perl APACHE_SRC=../apache_1.3.12/src DO_HTTPD=1 EVERYTHING=1 ALL_HOOKS=1 PERL_SSI=1 PERL_SECTIONS=1 APACI_ARGS=--sbindir=/usr/local/sbin/httpd_perl APACI_ARGS=--sysconfdir=/usr/local/apache/httpd_perl/conf APACI_ARGS=--runtimedir=/usr/local/apache/httpd_perl/run APACI_ARGS=--logfiledir=/usr/local/apache/httpd_perl/logs APACI_ARGS=--localstatedir=/usr/local/apache/httpd_perl/stat APACI_ARGS=--proxycachedir=/usr/local/apache/httpd_perl/proxy APACI_ARGS=--enable-module=rewrite APACI_ARGS=--enable-module=include APACI_ARGS=--enable-module=info APACI_ARGS=--activate-module=src/modules/extra/mod_macro.c == Output of perl -V: -- Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration: Platform: osname=solaris, osvers=2.6, archname=sun4-solaris uname='sunos emap-sun-07.whoc.theplanet.co.uk 5.6 generic_105181-21 sun4u sparc ' hint=recommended, useposix=true, d_sigaction=define usethreads=undef useperlio=undef d_sfio=undef Compiler: cc='cc -B/usr/ccs/bin/ -B/usr/ccs/bin/', optimize='-O', gccversion=egcs-2.91.66 19990314 (egcs-1.1.2 release) cppflags='-I/usr/local/include' ccflags ='-I/usr/local/include' stdchar='unsigned char', d_stdstdio=define, usevfork=false intsize=4, longsize=4, ptrsize=4, doublesize=8 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 alignbytes=8, usemymalloc=y, prototype=define Linker and Libraries: ld='ld', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib libs=-lsocket -lnsl -lgdbm -ldl -lm -lc -lcrypt libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a Dynamic Linking: dlsrc=dl_none.xs, dlext=none, d_dlsymun=undef, ccdlflags='' cccdlflags='', lddlflags='' Characteristics of this binary (from libperl): Built under solaris Compiled at Jun 16 2000 17:02:50 @INC: /usr/local/lib/perl5/5.00503/sun4-solaris /usr/local/lib/perl5/5.00503 /usr/local/lib/perl5/site_perl/5.005/sun4-solaris /usr/local/lib/perl5/site_perl/5.005 . == httpd_perl -l -- Compiled-in modules: http_core.c mod_env.c mod_log_config.c mod_mime.c mod_negotiation.c mod_status.c mod_info.c mod_include.c mod_autoindex.c mod_dir.c mod_cgi.c mod_asis.c mod_imap.c mod_actions.c mod_userdir.c
RE: Highly optimized mod_perl ?
Hi, There was a discussion on this list about 1.5 weeks ago on how to optimize mod_perl execution when using the database. Stas Bekman basically has done some tests to show that the driver needs to be initialized in startup.pl file. There were also a discussion about the fact that as far as using templates for HTML generation the fastest approach is when a template is compiled into perl code before being executed. I do not have enough understanding of the internals to explain why though. Another discussion suggested that it is much better to precompile explicitly the functions that you need to use in modules like CGI.pm instead. The fact that you precompile specific functions and not the whole module will save memory space of apache processes that run mod_perl. Saved memory means less swaping, and more memory for cache -- that means speed. There was also a post referring to a comparasing between many scripting and compiled languages that pointed out that by far the most important factor when choosing a language to implement a taks is what algorithm is being used (ie. how smart and fast it is) and that differences in programming languages are not significant compared to that. I also just recently read in "Advanced Perl Programming" by Sriram Srinivasan that (on page 369) there are Static Typing Hints: they give hints to the interepreter for better optimization and type checking. so my Dog $spot = new Dog; should be faster then my $spot= new Dog; It actually should be faster for the consequent method calls to $spot-xyz (because perl would not have to do type checking and run-time) (this me parapharsing the example from the book Static hints will also allow for more aggressive compiler optimizations (using perlcc) (but I do thing this mod_perl relevant). This is my high level overview of what I have learned sofar from the list (I am new to Perl so I am just trying to absorb as much as possible). Regards, Vladislav P. On 17-Jun-2000 Vladislav Safronov wrote: Hi! Does anyone know the way to highly optimize mod_perl for speed? Any hacks, advices? All standard advices read from help are done... Best regards, Vlad Safronov, mailto: [EMAIL PROTECTED]
ANNOUNCE: Apache::Filter 1.010
The URL http://forum.swarthmore.edu/~ken/modules/archive/Apache-Filter-1.010.tar.gz has entered CPAN as file: $CPAN/authors/id/K/KW/KWILLIAMS/Apache-Filter-1.010.tar.gz size: 15166 bytes md5: 8d5e5dec3ffc5fce9ff7cab8d25f7d9e Changes since 1.009: Version: 1.010 Date: 2000/06/17 18:26:05 Apache::RegistryFilter now overrides Apache's header-sending routines with no-ops, so that you may leave header-sending calls in your Registry scripts. This means you can use the same code under Apache::Registry and Apache::RegistryFilter. Added a test to make sure the above header-overriding works properly. Added lots of insane crap to the 'make test' phase to support apaches built with --enable-shared=max or any DSOs. [[EMAIL PROTECTED] (Rafael Kitover)] During 'perl Makefile.PL', search around for copies of apache to use during 'make test'. Also default to user 'nobody' if the current user is root. [[EMAIL PROTECTED] (Rafael Kitover)] Added Apache::SimpleReplace to list of Filter-aware handlers. [[EMAIL PROTECTED] (Geoffrey Young)] Changed \Files to /Files in documentation. [[EMAIL PROTECTED] (Stas Bekman)] ------ Ken Williams Last Bastion of Euclidity [EMAIL PROTECTED]The Math Forum
Re: Highly optimized mod_perl ?
Vladislav Safronov wrote: Does anyone know the way to highly optimize mod_perl for speed? Any hacks, advices? All standard advices read from help are done... The Guide (http://perl.apache.org/guide/) is very comprehensive. If you've followed the perfromance tuning tips there, the main thing left is improving the speed of your database or other data access. Use Apache::DProf to find out where it hurts. - Perrin
RE: Highly optimized mod_perl ?
Hi there, On Sat, 17 Jun 2000 [EMAIL PROTECTED] wrote: Saved memory means less swaping, You don't want _any_ swapping on a mod_perl host. 73, Ged.
RE: Highly optimized mod_perl ?
Yes, you are right. I should have just said 'saves memory' -- most developers realize the implications of improved memory utilization. Vladislav. On 17-Jun-2000 G.W. Haywood wrote: Hi there, On Sat, 17 Jun 2000 [EMAIL PROTECTED] wrote: Saved memory means less swaping, You don't want _any_ swapping on a mod_perl host. 73, Ged.
Re: IPC::Open2 v5.6.0 failures.
On Sat, Jun 17, 2000 at 09:16:25AM -0400, Barrie Slaymaker wrote:
[EMAIL PROTECTED] wrote:
Is this a know bug? Is there a bug?
Not that I know of, did you check the archives?
Yes but I didn't find the answer I was looking for.
I also checked out the guide's Forking and Executing Subprocesses
and ... Apache::SubProccess.
Apache::SubProccess just hung for me, if I have more time I'll check it out
more.
Is it right in front of my face?
Something seems odd to me, see below, but it shouldn't be causing you
too much grief.
First, to educate me, what's encrypt do differently than Perl's crypt?
why not
crypt $pass, salt() ;
encrypt allows me to use the Blowfish version of crypt;
128 bits vs 12 bits.
?
PLUG MODULE="IPC::Run" AUTHOR="me" MODE="shameless"
If you want to play with it, IPC::Run's up on CPAN and this might make
your routine simpler and more debuggable. You'd do something like this to
use it (untested):
use IPC::Run qw( run ) ;
sub encrypt {
my $pass = shift;
my $crypt;
run [qw(/usr/bin/encrypt', -b), salt()], \$pass, \$crypt
or die "Can't run '/usr/bin/encrypt': $!" ;
chomp $crypt;
return $crypt ;
}
The semantic of run's return value are what you, or at least I, would
expect. Then, to get a clue about what's happening under the hood, just
add a {debug=1} to run's params:
run [qw(/usr/bin/encrypt', -b), salt()], \$pass, \$crypt, {debug=1}
or die "Can't run '/usr/bin/encrypt': $!" ;
and see what happens. Increase the debug level to 2 or 3 if need be,
and it probably will be.
This is a beta module though, so there might be gotchas. One is that I'm
soon going to remove '{'...'}' hash constructors around the debug=2.
/PLUG
Thanks, I'll check it out.
Here's what seems odd:
open2(\*ENCRD, \*ENCWR, "/usr/bin/encrypt $args")
or die("Can't open2: '/usr/bin/encrypt': $!");
From the IPC::Open2 man page:
SYNOPSIS
use IPC::Open2;
$pid = open2(\*RDRFH, \*WTRFH, 'some cmd and args');
# or without using the shell
snip
open2() returns the process ID of the child process. It
doesn't return on failure: it just raises an exception
matching `/^open2:/'. However, `exec' failures in the
child are not detected. You'll have to trap SIGPIPE your-
self.
I don't see how this could this be contributing to your grief, though.
I take it there's no spoor in the error_log?
regrardless, open2 will warn on failure. Checking 'wait' and '$?' will tell
me if the command failed. But something screwy is going on inside of mod_perl.
I'd feel better if I knew what mod__perl was doing with the handles and why
the info was being dropped on the way back.
I also tried Symbol::gensym, Apache::gensym, and autoviv ... ... no diff.
As a performance note, have you considered passing the command line as
a list like
qw(/usr/bin/encrypt -b ), salt )
in place of the string? That way the exec in open2 is not invoking
a subshell. Also, if $pass comes from a user, what if they type in
`mail [EMAIL PROTECTED] /etc/passwd`?
the $pass is checked long before it makes it to the shell.
Well time is too valuable, so my solution for right now is to have a local
socket daemon run the encrypt command for me. Just have to remember to
have the daemon running ...
Thanks for your reply and help.
--Wendell
