Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
I just wanted to thank you guys for sending this to the mailing list.
I added these lines to my startup script
use Carp;
CGI-compile(qw(my_common_functions));
DBI-install_driver('mysql');
Please note that these were already existing in my startup script.
use CGI();
use DBI();
And after testing, and running the server with the new settings all night long, it
seems that I am saving ~2M of RAM for each apache
process. Again thank you.
However, this has made me curious, and left me wondering... I have 3 scripts that I
pre-load in my startup file, would any of my
self-made modules, or other CPAN modules, benefit from including directly in the
startup script... as opposed to being loaded only
in the pre-loaded scripts themselves?
Again, thank you for this thread it has saved me 20-80M of RAM depending on my current
load.
-Jason
- Original Message -
From: "Stas Bekman" [EMAIL PROTECTED]
To: "Tim Bunce" [EMAIL PROTECTED]
Cc: "mod_perl list" [EMAIL PROTECTED]
Sent: Sunday, June 04, 2000 1:50 PM
Subject: Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code
exercising)
On Sun, 4 Jun 2000, Tim Bunce wrote:
On Sat, Jun 03, 2000 at 02:49:47AM +0300, Stas Bekman wrote:
On Fri, 2 Jun 2000, Perrin Harkins wrote:
On Sat, 3 Jun 2000, Stas Bekman wrote:
* install_driver (2):
DBI-install_driver("mysql");
I've never seen that before,
There is always a first time :)
and it isn't in the DBI perldoc.
Where do you think I've found it :) It is mentioned in DBI manpage:
DBI-connect automatically installs the driver if it
has not been installed yet. Driver installation always
returns a valid driver handle or it dies with an error
message which includes the string 'install_driver' and
the underlying problem. So, DBI-connect will die on a
driver installation failure and will only return undef
on a connect failure, for which $DBI::errstr will hold
the error.
I've been meaning to document it properly for the last couple of
years, since I knew people were using it for mod_perl.
I guess most of the mod_perl folks don't use it, since they don't
know about it. I have discovered it following the Vivek's reply.
but actualy it's the DBD:: method.
Nope. A DBI-method.
Ok
Is it safer than "use DBD::mysql;"?
"safer" is the wrong question. It's always used by DBI, but when you call
it at the server startup you get a few more K shared, compared with only
preloading of DBD::mysql.
Yes, and the semantics of install_driver are that it'll die if the driver
can't be installed, just like 'use'.
Apache::DBI-connect_on_init('DBI:mysql:test::localhost', [...]
or DBI-disconnect("Cannot connect to database: $DBI::errstr\n");
There's no such method as DBI-disconnect.
Ouch, you are right. But the first part has never failed for me, so I copy
and paste this old error for years :) Thanks for telling me!
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
RE: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
At 11:10 AM 6/6/00 -0400, Geoffrey Young wrote:
-Original Message-
From: Jason Terry [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, June 06, 2000 10:52 AM
To: [EMAIL PROTECTED]
Subject: Re: [benchmark] DBI/preload (was Re: [RFC] improving memory
mapping thru code exercising)
I just wanted to thank you guys for sending this to the mailing list.
I added these lines to my startup script
use Carp;
CGI-compile(qw(my_common_functions));
DBI-install_driver('mysql');
I tried this in my startup.pl with Oracle, and this is what I got:
DBI handle cleared whilst still active at
/home/cardlock/lib/perl5/site_perl/5.005/sun4-solaris/DBD/Oracle.pm line 82.
dbih_clearcom (h 0x54f978, com 0x3319e8):
FLAGS 0x211: COMSET Warn AutoCommit
TYPE 1
PARENT undef
KIDS 0 (0 active)
IMP_DATA undef in 'DBD::Oracle::dr'
I tried both
use DBD::Oracle
and
DBI-install_driver( 'Oracle' );
with the same results. It didn't seem to cause any problems, since
everything still worked.
Something to worry about?
Cheers!
-klm.
---
Ken Miller, Consultant
Shetland Software Services Inc.
RE: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
-Original Message-
From: Ken Miller [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, June 06, 2000 12:13 PM
To: [EMAIL PROTECTED]
Subject: RE: [benchmark] DBI/preload (was Re: [RFC] improving memory
mapping thru code exercising)
At 11:10 AM 6/6/00 -0400, Geoffrey Young wrote:
-Original Message-
From: Jason Terry [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, June 06, 2000 10:52 AM
To: [EMAIL PROTECTED]
Subject: Re: [benchmark] DBI/preload (was Re: [RFC]
improving memory
mapping thru code exercising)
I just wanted to thank you guys for sending this to the
mailing list.
I added these lines to my startup script
use Carp;
CGI-compile(qw(my_common_functions));
DBI-install_driver('mysql');
I tried this in my startup.pl with Oracle, and this is what I got:
DBI handle cleared whilst still active at
this typically means that you have opened a cursor and the $dbh has gone out
of scope before calling $sth-finish
are you doing other stuff in your startup.pl?
/home/cardlock/lib/perl5/site_perl/5.005/sun4-solaris/DBD/Orac
le.pm line 82.
dbih_clearcom (h 0x54f978, com 0x3319e8):
FLAGS 0x211: COMSET Warn AutoCommit
TYPE 1
PARENT undef
KIDS 0 (0 active)
IMP_DATA undef in 'DBD::Oracle::dr'
I tried both
use DBD::Oracle
I use() DBD::Oracle as above in my startup.pl, but I don't see any
warnings...
and
DBI-install_driver( 'Oracle' );
I don't have any experience with this method...
with the same results. It didn't seem to cause any problems, since
everything still worked.
Something to worry about?
generally no, but since lots of people use() DBD::* in the startup files, it
doesn't sound normal
--GEoff
Cheers!
-klm.
---
Ken Miller, Consultant
Shetland Software Services Inc.
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
On Mon, 5 Jun 2000, Vivek Khera wrote:
"PH" == Perrin Harkins [EMAIL PROTECTED] writes:
PH On Sat, 3 Jun 2000, Stas Bekman wrote:
* install_driver (2):
DBI- install_driver("mysql");
PH I've never seen that before, and it isn't in the DBI perldoc. Is it safer
PH than "use DBD::mysql;"?
"use DBD::mysql" doesn't really do anything, does it?
Only preloads (compiles) the module. See the benchmark.
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
On Sat, Jun 03, 2000 at 02:49:47AM +0300, Stas Bekman wrote:
On Fri, 2 Jun 2000, Perrin Harkins wrote:
On Sat, 3 Jun 2000, Stas Bekman wrote:
* install_driver (2):
DBI-install_driver("mysql");
I've never seen that before,
There is always a first time :)
and it isn't in the DBI perldoc.
Where do you think I've found it :) It is mentioned in DBI manpage:
DBI-connect automatically installs the driver if it
has not been installed yet. Driver installation always
returns a valid driver handle or it dies with an error
message which includes the string 'install_driver' and
the underlying problem. So, DBI-connect will die on a
driver installation failure and will only return undef
on a connect failure, for which $DBI::errstr will hold
the error.
I've been meaning to document it properly for the last couple of
years, since I knew people were using it for mod_perl.
but actualy it's the DBD:: method.
Nope. A DBI-method.
Is it safer than "use DBD::mysql;"?
"safer" is the wrong question. It's always used by DBI, but when you call
it at the server startup you get a few more K shared, compared with only
preloading of DBD::mysql.
Yes, and the semantics of install_driver are that it'll die if the driver
can't be installed, just like 'use'.
Apache::DBI-connect_on_init('DBI:mysql:test::localhost', [...]
or DBI-disconnect("Cannot connect to database: $DBI::errstr\n");
There's no such method as DBI-disconnect.
Tim.
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
On Sun, 4 Jun 2000, Tim Bunce wrote:
On Sat, Jun 03, 2000 at 02:49:47AM +0300, Stas Bekman wrote:
On Fri, 2 Jun 2000, Perrin Harkins wrote:
On Sat, 3 Jun 2000, Stas Bekman wrote:
* install_driver (2):
DBI-install_driver("mysql");
I've never seen that before,
There is always a first time :)
and it isn't in the DBI perldoc.
Where do you think I've found it :) It is mentioned in DBI manpage:
DBI-connect automatically installs the driver if it
has not been installed yet. Driver installation always
returns a valid driver handle or it dies with an error
message which includes the string 'install_driver' and
the underlying problem. So, DBI-connect will die on a
driver installation failure and will only return undef
on a connect failure, for which $DBI::errstr will hold
the error.
I've been meaning to document it properly for the last couple of
years, since I knew people were using it for mod_perl.
I guess most of the mod_perl folks don't use it, since they don't
know about it. I have discovered it following the Vivek's reply.
but actualy it's the DBD:: method.
Nope. A DBI-method.
Ok
Is it safer than "use DBD::mysql;"?
"safer" is the wrong question. It's always used by DBI, but when you call
it at the server startup you get a few more K shared, compared with only
preloading of DBD::mysql.
Yes, and the semantics of install_driver are that it'll die if the driver
can't be installed, just like 'use'.
Apache::DBI-connect_on_init('DBI:mysql:test::localhost', [...]
or DBI-disconnect("Cannot connect to database: $DBI::errstr\n");
There's no such method as DBI-disconnect.
Ouch, you are right. But the first part has never failed for me, so I copy
and paste this old error for years :) Thanks for telling me!
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
[benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
On Mon, 29 May 2000, Vivek Khera wrote:
"SB" == Stas Bekman [EMAIL PROTECTED] writes:
SB A while ago, a few people have mentioned that it's possible to improve the
SB way Perl data structures get mapped in memory pages, by exercising the
SB code before the child processes have been spawned, i.e. running the code
SB and not just pre-compiling it. Did anyone try this at home :) Any
SB satisfactionary results on this direction or what it just a crazy idea?
This is extremely important for DBI, since the DBD layer is not loaded
until needed. Thus, unless you exercise your Connect in DBI, the DBD
is not loaded until each child is spawned. I'm sure other modules do
some sort of initialization as well which could break sharing since
perl code and data pages are the same as far as the OS is concerned.
In fact it's not the connect, but the install_driver that makes the
difference, connect_on_init (I know you said connect, not on_init :) only
gets the connection ready for the first request for each process, it
doesn't preload the driver!!! surprise, surprise... rush and update your
startup files!
Here are the tests I've conducted. Our goal is to find the startup
environment tha will lead to the smallest "Difference" between the shared
and normal memory reports, therefore lesser total memory usage.
Here are the results (exclusively for the mod_perl list ;-) :
The startup.pl file could have:
nothing added 1
install_driver 2
connect_on_init 3
Results (sorted by Diff):
=== First request:
Test: Size SharedDiff
---
2 : 3465216 2621440 843776 (bytes)
2+3 : 3461120 2609152 851968 (bytes)
1 : 3461120 2494464 966656 (bytes)
3 : 3461120 2482176 978944 (bytes)
=== Second request (all the subsequent request showed the same results):
Test: Size Shared Diff
---
2 : 3469312 2609152 860160 (bytes)
2+3 : 3481600 2605056 876544 (bytes)
1 : 3477504 2482176 995328 (bytes)
3 : 3481600 2469888 1011712 (bytes)
Notice that after the second reload the process's memory size grows a
little bit and there are less shared memory pages, but all the subsequent
reloads have shown the same results.
As you can see that the *real* winner is the startup.pl file that has used
install_driver (2). Having only connect_on_init (3) is the worst case in
terms of shared memory usage.
Here is the test enviroment:
### startup.pl:
* Always preloaded:
Gtop() and Apache::DBI()
* install_driver (2):
DBI-install_driver("mysql");
* connect_on_init (3):
Apache::DBI-connect_on_init('DBI:mysql:test::localhost',
"",
"",
{
PrintError = 1, # warn() on errors
RaiseError = 0, # don't die on error
AutoCommit = 1, # commit executes
# immediately
}
)
or DBI-disconnect("Cannot connect to database: $DBI::errstr\n");
### httpd.conf:
MinSpareServers 1
MaxSpareServers 1
StartServers 1
MaxClients 1
MaxRequestsPerChild 100
PerlModule Apache::Registry
### The registry script used in test (unmodified in all 4 test):
preload_dbi.pl
--
use strict;
use GTop ();
use DBI ();
my $dbh = DBI-connect("DBI:mysql:test::localhost",
"",
"",
{
PrintError = 1, # warn() on errors
RaiseError = 0, # don't die on error
AutoCommit = 1, # commit executes
# immediately
}
)
or DBI-disconnect("Cannot connect to database: $DBI::errstr\n");
my $r = shift;
$r-send_http_header('text/plain');
my $do_sql = "show tables";
my $sth = $dbh-prepare($do_sql);
$sth-execute();
my @data = ();
while (my @row = $sth-fetchrow_array){
push @data, @row;
}
$dbh-disconnect();
print "Data: @data\n";
my $proc_mem = GTop-new-proc_mem($$);
my $size = $proc_mem-size;
my $share = $proc_mem-share;
my $diff = $size - $share;
printf "%8s %8s %8s\n", qw(Size Shared Diff);
printf "%8d %8d %8d (bytes)\n",$size,$share,$diff;
-
Your comments are very welcome!
Especially about other modules that allow initialization and thus
reducing the memory usage. (not CGI.pm, I know about this one)
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
On Fri, 2 Jun 2000, Perrin Harkins wrote:
On Sat, 3 Jun 2000, Stas Bekman wrote:
* install_driver (2):
DBI-install_driver("mysql");
I've never seen that before,
There is always a first time :)
and it isn't in the DBI perldoc.
Where do you think I've found it :) It is mentioned in DBI manpage:
DBI-connect automatically installs the driver if it
has not been installed yet. Driver installation always
returns a valid driver handle or it dies with an error
message which includes the string 'install_driver' and
the underlying problem. So, DBI-connect will die on a
driver installation failure and will only return undef
on a connect failure, for which $DBI::errstr will hold
the error.
but actualy it's the DBD:: method. No matter what, you get more shared
memory with it, see the updated table:
Version Size SharedDiff Test type
1 3469312 2609152 860160 install_driver
2 3481600 2605056 876544 install_driver connect_on_init
3 3469312 2576384 892928 load driver
4 3477504 2482176 995328 nothing added
5 3481600 2469888 1011712 connect_on_init
Is it safer than "use DBD::mysql;"?
"safer" is the wrong question. It's always used by DBI, but when you call
it at the server startup you get a few more K shared, compared with only
preloading of DBD::mysql.
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC
http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
Re: [benchmark] DBI/preload (was Re: [RFC] improving memory mapping thru code exercising)
but actualy it's the DBD:: method. No matter what, you get more shared memory with it, see the updated table: Version Size SharedDiff Test type 1 3469312 2609152 860160 install_driver 2 3481600 2605056 876544 install_driver connect_on_init 3 3469312 2576384 892928 preload driver 4 3477504 2482176 995328 nothing added 5 3481600 2469888 1011712 connect_on_init correction for the 3rd version (had the wrong startup), but it's almost the same. Version Size SharedDiff Test type 1 3469312 2609152 860160 install_driver 2 3481600 2605056 876544 install_driver connect_on_init 3 3469312 2588672 880640 preload driver 4 3477504 2482176 995328 nothing added 5 3481600 2469888 1011712 connect_on_init _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://perl.org http://stason.org/TULARC http://singlesheaven.com http://perlmonth.com http://sourcegarden.org
