Re: profiling DBI (was Re: apache children waits for each other?)
Perrin Harkins wrote: Malcolm J Harwood wrote: Somehow I managed to miss the existence of DBI::Profile. Has anyone gotten this working with Apache::DBI (so that it aggregates across the entire lifetime of the DB connection)? Just use DBI::ProfileDumper::Apache. I don't believe thats been ported mp2. Also doesn't look that hard to do so. -- END What doesn't kill us can only make us stronger. Nothing is impossible. Philip M. Gollucci ([EMAIL PROTECTED]) 301.254.5198 Consultant / http://p6m7g8.net/Resume/ Senior Developer / Liquidity Services, Inc. http://www.liquidityservicesinc.com http://www.liquidation.com http://www.uksurplus.com http://www.govliquidation.com http://www.gowholesale.com
Re: profiling DBI (was Re: apache children waits for each other?)
Malcolm J Harwood wrote: Somehow I managed to miss the existence of DBI::Profile. Has anyone gotten this working with Apache::DBI (so that it aggregates across the entire lifetime of the DB connection)? Just use DBI::ProfileDumper::Apache. - Perrin
profiling DBI (was Re: apache children waits for each other?)
> >You can look at the tuning information on http://modperlbook.org/ for > >some advice as well. And don't forget that there is a profiler for DBI > >queries that comes with DBI. Somehow I managed to miss the existence of DBI::Profile. Has anyone gotten this working with Apache::DBI (so that it aggregates across the entire lifetime of the DB connection)? -- "You heard Mr. Garibaldi. You did the right thing." "Darling, I did the necessary thing. That is not always the same as the right thing." - Janice and Laura Rosen in Babylon 5:"The Quality of Mercy"
Re: apache children waits for each other?
> I don't know it's even possible in mysql. Its reference docs doesn't say > that it is (http://dev.mysql.com/doc/mysql/en/join.html). > Yes it does : http://dev.mysql.com/doc/mysql/en/identifier-qualifiers.html Hmmm... Excellent!!! I'll use it then...Will let you know how it goes... Thanks, mate... --- Badai Aqrandista Cheepy (?) _ Single? Start dating at Lavalife. Try our 7 day FREE trial! http://lavalife9.ninemsn.com.au/clickthru/clickthru.act?context=an99&locale=en_AU&a=19179
Re: apache children waits for each other?
> I don't know it's even possible in mysql. Its reference docs doesn't say > that it is (http://dev.mysql.com/doc/mysql/en/join.html). > Yes it does : http://dev.mysql.com/doc/mysql/en/identifier-qualifiers.html Clinton Gormley [EMAIL PROTECTED] www.TravelJury.com - For travellers, By travellers
Re: apache children waits for each other?
Apache::DProf uses the debugger hooks. You can also try Sam Tregar's Devel::Profiler::Apache which is somewhat slower but easier to use since it doesn't use the debugger. I'll look into that... Thanks for your suggestions... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: apache children waits for each other?
Badai Aqrandista wrote: use lib '/i4u/web/elres-mp'; #require Apache::DB; #Apache::DB->init; PerlRequire /i4u/web/elres-mp/etc/startup.pl PerlModule Apache::DProf PerlModule ELRes::ELRes PerlModule ELRes::ApacheHandler PerlTransHandler ELRes::ApacheHandler->trans_handler PerlLogHandler ELRes::ApacheHandler->log_handler PerlCleanupHandler ELRes::ApacheHandler->cleanup_handler This looks okay, assuming those lines were not commented out when you ran it. the startup.pl script loads and require's all modules under ELRes:: namespace on startup time, as suggested in mod_perl documentation... Why do you have those two PerlModule calls then? I suggest taking them out. If that breaks anything, it means your startup.pl is not doing what you think it is. the object that $app refer to is dynamically decided based on the hostname, for example: a.example.com ---> $app = ELRes::Property->new() b.example.com ---> $app = ELRes::Property->new() c.example.com ---> $app = ELRes::Distributor->new() both ELRes::Property and ELRes::Distributor are children of ELRes::TopLevelEntity and have run() method... does this prevents the debugger to know which subroutine should be profiled? No, it won't make a difference. Hmmm... Where can I learn more about the debugger? There is information about it in the perl man pages (man perldebug), in Programming Perl, and in various articles on the web. I didn't know profiling needs debugger, not until I was told about it in this mailinglist... Apache::DProf uses the debugger hooks. You can also try Sam Tregar's Devel::Profiler::Apache which is somewhat slower but easier to use since it doesn't use the debugger. - Perrin
Re: apache children waits for each other?
It does look odd. Maybe you are still loading some code before calling
Apache::DB->init(). Can you post your httpd.conf, or at least the mod_perl
part of it?
start of mod_perl configuration ...
use lib '/i4u/web/elres-mp';
#require Apache::DB;
#Apache::DB->init;
PerlRequire /i4u/web/elres-mp/etc/startup.pl
PerlModule Apache::DProf
PerlModule ELRes::ELRes
PerlModule ELRes::ApacheHandler
PerlTransHandler ELRes::ApacheHandler->trans_handler
PerlLogHandler ELRes::ApacheHandler->log_handler
PerlCleanupHandler ELRes::ApacheHandler->cleanup_handler
... end mod_perl configuration ...
the startup.pl script loads and require's all modules under ELRes::
namespace on startup time, as suggested in mod_perl documentation...
eval { $app->run }; if ($@) { $r->log_error("Server error: $@");
$app->send_error(err => $@); }
That's why I think you are loading some code before initializing the
debugger -- when it doesn't know about those other subs, they don't show up
in the profile.
the object that $app refer to is dynamically decided based on the hostname,
for example:
a.example.com ---> $app = ELRes::Property->new()
b.example.com ---> $app = ELRes::Property->new()
c.example.com ---> $app = ELRes::Distributor->new()
both ELRes::Property and ELRes::Distributor are children of
ELRes::TopLevelEntity and have run() method...
does this prevents the debugger to know which subroutine should be profiled?
all the test that I've done is on the hostname whose object is of class
ELRes::Property...
Anyway,
Hmmm... Where can I learn more about the debugger?
I didn't know profiling needs debugger, not until I was told about it in
this mailinglist...
Cheers...
---
Badai Aqrandista
Cheepy (?)
_
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: apache children waits for each other?
Badai Aqrandista wrote:
As I said previously, the handler itself tops everything... I don't know
why it happens...
It does look odd. Maybe you are still loading some code before calling
Apache::DB->init(). Can you post your httpd.conf, or at least the
mod_perl part of it?
If I use Devel::Timer to profile its parts, I found
that this block takes the most time to run (99.98%):
eval { $app->run };
if ($@) {
$r->log_error("Server error: $@");
$app->send_error(err => $@);
}
That's pretty much the whole thing, isn't it?
But the call should not be included in the handler's ExclSec time,
shouldn't it?
That's why I think you are loading some code before initializing the
debugger -- when it doesn't know about those other subs, they don't show
up in the profile.
- Perrin
Re: apache children waits for each other?
Okay, send it again with the top 10, sorted by -r then. Maybe we can
make more suggestions.
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
20.5 14.30 15.222 1 14.305 15.222 ELRes::ApacheHandler::handler
0.37 0.261 0.261 99576 0. 0. Date::Simple::as_iso
0.22 0.151 0.153 20364 0. 0.
Params::Validate::_check_regex_fro
m_xs
0.17 0.117 0.117 6580 0. 0. Storable::mretrieve
0.13 0.093 0.093 27854 0. 0. Date::Simple::_add
0.09 0.065 0.065 2900 0. 0. ELRes::Entity::DESTROY
0.08 0.058 0.058 34766 0. 0. Date::Simple::DESTROY
0.08 0.054 0.054 41678 0. 0. Date::Simple::_compare
0.05 0.035 0.035 12984 0. 0. Date::Simple::__ANON__
0.03 0.020 0.020 36 0.0006 0.0005 Template::Context::filter
0.03 0.019 0.019102 0.0002 0.0002 DBI::common::DESTROY
0.01 0.010 0.010 1 0.0100 0.0100 DynaLoader::bootstrap
0.01 0.010 0.010 1 0.0100 0.0100 ELRes::App::Session::DESTROY
0.01 0.010 -0.000 2 0.0050 -
ELRes::ApacheHandler::trans_handle
0. r
0.01 0.010 0.010 25 0.0004 0.0004 DBD::mysql::db::_login
As I said previously, the handler itself tops everything... I don't know why
it happens... If I use Devel::Timer to profile its parts, I found that this
block takes the most time to run (99.98%):
eval { $app->run };
if ($@) {
$r->log_error("Server error: $@");
$app->send_error(err => $@);
}
But the call should not be included in the handler's ExclSec time, shouldn't
it? The eval doesn't make the call included in the ExclSec time because I've
tested without it and it still use 99.98% of the time...
Wierd eh?
Thanks a lot...
---
Badai Aqrandista
Cheepy (?)
_
Sell your car for $9 on carpoint.com.au
http://www.carpoint.com.au/sellyourcar
Re: apache children waits for each other?
On Wed, 2005-08-17 at 10:15 +1000, Badai Aqrandista wrote: > >Was the previous output sorted with the -r flag though? > > No. Okay, send it again with the top 10, sorted by -r then. Maybe we can make more suggestions. - Perrin
Re: apache children waits for each other?
Was the previous output sorted with the -r flag though? No. There's no trick, it's the same as in any language: find out what the slow parts are by using a profiler and tinker with them until they are faster. You can look at the tuning information on http://modperlbook.org/ for some advice as well. And don't forget that there is a profiler for DBI queries that comes with DBI. I've done the database profiling as well... Thanks for your help... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: apache children waits for each other?
On Wed, 2005-08-17 at 09:23 +1000, Badai Aqrandista wrote: > I got the previous output because I put in the debugger initialization... Was the previous output sorted with the -r flag though? It just seems so unlikely that Params::Validate would take significant time. If it was sorted with -r, it looks like your use of memcached is working against you. Maybe you have a slow network connection to the memcached server, or are just storing too much in it. I'd suggest you look into using a local cache instead, like Cache::FastMmap. This will be faster, although the cache will not be shared between machines. > Without initialization (Apache::DB->init) No point in looking at that. It's missing most of your code. > How should I interpret these outputs and what should I do about it to > improve my app's speed? This is the first time for me to optimize a large > perl codebase... So I need a bit of guidance here... There's no trick, it's the same as in any language: find out what the slow parts are by using a profiler and tinker with them until they are faster. You can look at the tuning information on http://modperlbook.org/ for some advice as well. And don't forget that there is a profiler for DBI queries that comes with DBI. - Perrin
Re: apache children waits for each other?
I do all of my memcached stuff , including sessions, with mysql failover.
its barely more code - i just make every public function address two
private functions.
ie:
sub save {
$_[0]->_save_memcached();
$_[0]->_save_mysql();
}
sub load {
if ( !$_[0]->_load_memcached() )
{
$_[0]->_load_mysql();
}
}
I thought about the same thing last night, but is there any race condition
problem if I use it for session?
---
Badai Aqrandista
Cheepy (?)
_
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: apache children waits for each other?
I also suspect that you didn't initialize the debugger before compiling your code, since none of your code appears in this output. I got the previous output because I put in the debugger initialization... Without initialization (Apache::DB->init), this is the top 5 output before the cache is filled (with dprofpp -r): Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 33.5 23.74 5.224 1 23.747 5.2236 ELRes::ApacheHandler::handler 1.09 0.772 0.772 7869 0.0001 0.0001 Storable::mretrieve 0.87 0.615 0.615 1418 0.0004 0.0004 Storable::net_mstore 0.70 0.495 0.495 1524 0.0003 0.0003 DBD::mysql::db::_login 0.54 0.380 0.380 2902 0.0001 0.0001 DBI::_new_sth 0.51 0.359 0.388 8852 0. 0. DBI::common::DESTROY and this one is after it's filled: Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 16.5 13.85 14.780 1 13.857 14.780 ELRes::ApacheHandler::handler 0.21 0.174 0.174 34766 0. 0. Date::Simple::DESTROY 0.20 0.170 0.170 6580 0. 0. Storable::mretrieve 0.19 0.162 0.162 99576 0. 0. Date::Simple::as_iso 0.14 0.115 0.115 27854 0. 0. Date::Simple::_add 0.11 0.090 0.090 25 0.0036 0.0036 DBD::mysql::db::_login How should I interpret these outputs and what should I do about it to improve my app's speed? This is the first time for me to optimize a large perl codebase... So I need a bit of guidance here... Thanks a lot... --- Badai Aqrandista Cheepy (?) _ Your opinion counts..for your chance to win a Mini Cooper click here http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: apache children waits for each other?
Hi, I'm assuming that each client database is contained within the same MySQL server, and that you're not running 300 MySQL servers on different machines or different ports? Yes, why would I do that? If so, you can reuse the same connections, and just reference the different database in the query.So : select * from db1.table1 select * from db2.table1 When you connect and specify a database name, you're just specifying the default database. I don't know it's even possible in mysql. Its reference docs doesn't say that it is (http://dev.mysql.com/doc/mysql/en/join.html). Thanks... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: apache children waits for each other?
On Aug 16, 2005, at 11:17 AM, Perrin Harkins wrote: Good plan. This would make a nice addition (as a separate module) to the Apache::Session::Memcached distribution. I'll see if I can figure out how to make: Apache::Session::Memcached::WithFailover where it does that behavior, but then allows you to specify a specific Apache::Session datastore as being the failover device it shouldn't be all that hard - mostly aliasing function calls to existing modules.
Re: apache children waits for each other?
Jonathan Vanasco wrote: I do all of my memcached stuff , including sessions, with mysql failover. Good plan. This would make a nice addition (as a separate module) to the Apache::Session::Memcached distribution. - Perrin
Re: apache children waits for each other?
On Aug 16, 2005, at 9:49 AM, Perrin Harkins wrote:
If you can use Apache::Session::Memcached, why can't you use
Apache::Session::MySQL with a common database login? That will mean
one connection per apache process. You can also raise the connection
limit on your database server if it isn't dying under the load.
I do all of my memcached stuff , including sessions, with mysql
failover. its barely more code - i just make every public function
address two private functions.
ie:
sub save {
$_[0]->_save_memcached();
$_[0]->_save_mysql();
}
sub load {
if ( !$_[0]->_load_memcached() )
{
$_[0]->_load_mysql();
}
}
Re: apache children waits for each other?
Badai Aqrandista wrote: Yes, I'm aware of that. The problem is that the database structure is a reminiscent of the old version, which creates one database per client. It used to be only 40 clients and one web server. Now we have 300+ clients (=300+ databases) and 2 web servers. I always get 'Too many connections' error when I test the application with more than 100 concurrent request. Any other ideas? If you can use Apache::Session::Memcached, why can't you use Apache::Session::MySQL with a common database login? That will mean one connection per apache process. You can also raise the connection limit on your database server if it isn't dying under the load. Now, This is the top 5 output of 'dprofpp tmon.out' before data is cached: You should run dprofpp with the -r flag. This output doesn't even show the database work in the top 5, and that is normally the top area where time gets spent. I also suspect that you didn't initialize the debugger before compiling your code, since none of your code appears in this output. Params::Validate::_validate is very fast and wouldn't be showing up this high if the rest of your code was getting profiled. - Perrin
Re: apache children waits for each other?
> Yes, I'm aware of that. The problem is that the database structure is a > reminiscent of the old version, which creates one database per client. It > used to be only 40 clients and one web server. Now we have 300+ clients > (=300+ databases) and 2 web servers. I always get 'Too many connections' > error when I test the application with more than 100 concurrent request. Any > other ideas? > I'm assuming that each client database is contained within the same MySQL server, and that you're not running 300 MySQL servers on different machines or different ports? If so, you can reuse the same connections, and just reference the different database in the query.So : select * from db1.table1 select * from db2.table1 When you connect and specify a database name, you're just specifying the default database. Clinton Gormley [EMAIL PROTECTED] www.TravelJury.com - For travellers, By travellers
Re: apache children waits for each other?
On 16 Aug 2005, at 06:55, Badai Aqrandista wrote: Badai Aqrandista wrote: My mod_perl web app uses memcached to cache most of the (MySQL) database query results and as the session storage (Apache::Session::Memcached). Would it be a problem for your application if you suddenly lost all of your session data? That could happen with memcached. Yes, I'm aware of that. The problem is that the database structure is a reminiscent of the old version, which creates one database per client. It used to be only 40 clients and one web server. Now we have 300+ clients (=300+ databases) and 2 web servers. I always get 'Too many connections' error when I test the application with more than 100 concurrent request. Any other ideas? Yes, really read an understand the chapters in the mod_perl guide about setting up two_tier servers and what the implications are. You may need to do some MySQL tuning too if you haven't already. There are parameters in there to do with the number of concurrent open databases.
Re: apache children waits for each other?
Badai Aqrandista wrote: My mod_perl web app uses memcached to cache most of the (MySQL) database query results and as the session storage (Apache::Session::Memcached). Would it be a problem for your application if you suddenly lost all of your session data? That could happen with memcached. Yes, I'm aware of that. The problem is that the database structure is a reminiscent of the old version, which creates one database per client. It used to be only 40 clients and one web server. Now we have 300+ clients (=300+ databases) and 2 web servers. I always get 'Too many connections' error when I test the application with more than 100 concurrent request. Any other ideas? As for how to find what is making your application slow, use Apache::DProf. That is the best way to find the bottleneck. I have, and I have optimised it (with caches and indexes). Now, This is the top 5 output of 'dprofpp tmon.out' before data is cached: Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 14.4 4.189 9.431 16511 0.0003 0.0006 Cache::Memcached::_load_multi 12.5 3.639 3.639693 0.0053 0.0053 Compress::Zlib::deflateStream::def late 8.94 2.589 3.018 46501 0.0001 0.0001 Params::Validate::_validate 7.68 2.225 4.025 17582 0.0001 0.0002 Cache::Memcached::__ANON__[/usr/sh are/perl5/Cache/Memcached.pm:667] 4.20 1.217 1.217 16511 0.0001 0.0001 Cache::Memcached::__ANON__[/usr/sh are/perl5/Cache/Memcached.pm:691] ... and this is after the cache is filled up ... Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 22.5 3.752 6.911 15141 0.0002 0.0005 Cache::Memcached::_load_multi 15.6 2.602 2.939 38569 0.0001 0.0001 Params::Validate::_validate 9.56 1.595 2.242 15173 0.0001 0.0001 Cache::Memcached::__ANON__[/usr/sh are/perl5/Cache/Memcached.pm:667] 5.50 0.917 0.917 15141 0.0001 0.0001 Cache::Memcached::__ANON__[/usr/sh are/perl5/Cache/Memcached.pm:691] 4.05 0.676 0.676 102787 0. 0. Class::Accessor::get So, as you see, the subroutines that takes most of the time are realy needed (to speed up the code and to enforce code correctness). But I still need more speed (yes, I'm obsessed with speed :p... Hehe... It's just the old version -a bunch of cgi scripts put together- runs twice as fast as the new version -a modular OO structure-, and my boss wasn't pleased to hear that...) Any ideas? Anyway, thanks for your replies... --- Badai Aqrandista Cheepy (?) _ Dating? Try Lavalife get 7 days FREE! Sign up NOW. http://www.lavalife.com/clickthru/clickthru.act?id=ninemsn&context=an99&a=20233&locale=en_AU&_t=33473
Re: apache children waits for each other?
Badai Aqrandista wrote: My mod_perl web app uses memcached to cache most of the (MySQL) database query results and as the session storage (Apache::Session::Memcached). Would it be a problem for your application if you suddenly lost all of your session data? That could happen with memcached. As for how to find what is making your application slow, use Apache::DProf. That is the best way to find the bottleneck. - Perrin
RE: apache children waits for each other? (was: Re: web application speed problem)
UPDATE: I hadn't done much testing when I wrote this... Well, after a bit more testing they don't wait for each other apparently... Sorry for filling up your mailbox... But the actual question remains: How to optimize this web app? I'll send more questions later with more details... Thanks for your time... Hi all, My mod_perl web app uses memcached to cache most of the (MySQL) database query results and as the session storage (Apache::Session::Memcached). When doing performance tests with httperf, I found that apache processes waits for each other. I mean: all requests are accepted (apache forks off lots of children), then processed (no process finishes), and after a while, all the processes finish almost at the same time... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
apache children waits for each other? (was: Re: web application speed problem)
Hi all, My mod_perl web app uses memcached to cache most of the (MySQL) database query results and as the session storage (Apache::Session::Memcached). When doing performance tests with httperf, I found that apache processes waits for each other. I mean: all requests are accepted (apache forks off lots of children), then processed (no process finishes), and after a while, all the processes finish almost at the same time... If I put warns in the apache handler's start and end, it gives this output (edited for brevity, md5 hash in the end is the session ids, just disregard them): ... output from 90+ of other warns... Start process 30324: Tue, 16 Aug 2005 11:00:23 EST,af263d03f531584c4d05c7f3df55c7d4 Start process 30326: Tue, 16 Aug 2005 11:00:23 EST,931f8a3a2de409fb3285c3bd038ec7b6 Start process 30325: Tue, 16 Aug 2005 11:00:23 EST,48ca7271dab9037f3a7426a6a68789e5 Start process 30314: Tue, 16 Aug 2005 11:00:23 EST,259709a683879491498cb81055466948 ... after a while, suddenly the following output comes out ... End process 29803: Tue, 16 Aug 2005 11:05:47 EST,dc7e7c893f0ee15b0b6c94d061f9493c End process 30337: Tue, 16 Aug 2005 11:05:49 EST,73f520865a13715e02d183c5cbb920d4 End process 30002: Tue, 16 Aug 2005 11:05:49 EST,cbfb00260afade004d3cc3086f425272 End process 30003: Tue, 16 Aug 2005 11:05:51 EST,b84da97b68df5f54e336a7767d64a1f8 ... output of other 90+ warns ... It's like they are not processed concurrently. I wonder why does it happen? Is it memcached? Or is it how apache works? Or is it because of the benchmarking application that I use? Thanks... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
