Apache::Session's session size
Hi All, I am experiencing intermittently missing session data. After weeks of investigation, we suspect the size of the data we put in the session as the culprit: it is too big. So I am trying to fix it by adding Storable hooks for objects stored in the session to create smaller serialized data. I am just wondering has anyone experience problems with the size of the data put in the session using Apache::Session? The problem is so critical yet so intermittent that we can only fix it by trial and error. Thank you... --- Badai Aqrandista Cheepy (?) _ Start something musical - 15 free ninemsn Music downloads! http://ninemsn.com.au/share/redir/adTrack.asp?mode=clickclientID=667referral=HotmailTaglineNovURL=http://www.ninemsn.com.au/startsomething
Re: Apache::Session's session size
Hi Perrin, What database are you using? I thought the default data types for most of them would hold a very large amount of data before having trouble. I suspect you are having locking problems or session data is not being written because the session object doesn't go out of scope. I am using MySQL 4.1 on debian sarge. The type of the field that hold the session data is 'longtext'. Patrick Michaud pointed me off the list that i should also bump max_packet_size up. It seems that this problem occurs in peak period, so it works in thousands of hits except once or twice. So it can't be scoping issue. Even if the size of your session data isn't causing this lost data, it is a serious problem. When your session data is large, it makes everything slow, holds the locks longer, and hurts the performance of your site. You should consider putting that data into database tables instead and only loading it as needed. Yes, that's why I am using Storable hooks to only serialize as small information as possible. --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
Re: Detecting request cancellation behind proxy servers
On Wed, 2005-11-16 at 10:11 +1000, Badai Aqrandista wrote: But I need to know if a request has been cancelled. The problem is that I need to release the session lock when someone click submit button before the page is fully loaded, otherwise the next page won't load because it is waiting for the session lock to be released by the previous process that has been cancelled before releasing it. Why are you using locking with your sessions? Are you doing something that truly requires it? You already have atomic updates if you use database-backed session storage. Actually, the locking I meant is the Apache::Session's lock. I am guessing that A::S lock blocks the second page to be loaded if the first page hasn't finished loading. In other words, look for a way to avoid the root of the problem: doing a slow operation and keeping the session locked the whole time. I'll look into that. Thanks... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Detecting request cancellation behind proxy servers
Hi All, I have backend apache-mod_perl processes running behind frontend apache-proxy servers. I am aware that it is impossible for the backend processes to know if a request has been cancelled, according to: http://perl.apache.org/docs/1.0/guide/debug.html#Handling_the__User_pressed_Stop_button__case But I need to know if a request has been cancelled. The problem is that I need to release the session lock when someone click submit button before the page is fully loaded, otherwise the next page won't load because it is waiting for the session lock to be released by the previous process that has been cancelled before releasing it. Does anyone know a workaround or a way to combat this? Thanks... --- Badai Aqrandista Cheepy (?) _ Start something musical - 15 free ninemsn Music downloads! http://ninemsn.com.au/share/redir/adTrack.asp?mode=clickclientID=667referral=HotmailTaglineNovURL=http://www.ninemsn.com.au/startsomething
Re: mod_perl: How pass variables (objects) from page to page?
On 2005-11-15, at 03.11, Peter1 Alvin wrote: Using mod_perl, how do you keep Perl objects in RAM from page to page? I don't want to re-instantiate my objects on every page request, and I don't want the overhead of serializing my objects to a persistent store from page to page (I use A LOT of objects). IMHO, if I create a lot of objects, I don't want to put them all in RAM. I'd rather create the objects on every request, i.e.: only store the needed objects in RAM. It is a CPU-Memory trade off. As someone has already noted in this thread, storing everything in RAM isn't scalable. I've done extensive reseach and I've not found any application space to store objects indefinitly. To store objects indefinitely, you must serialize it into persistent storage. What happen when the computer goes down? I can't believe Perl doesn't support keeping objects in RAM. This would de-qualify Perl for half of the projects I need to develop. Objects only exist in RAM and nowhere else. If you want to make them to persist between request, you need to put them somewhere that can be accessed by different processes (and different servers, in the future). Those places are: filesystem, database, or memcached. Or at the least: shared memory. Have looked around like you for a solution to this, without finding one and after discussing it internally, we thought that the optimal can be to write a custom reversed modproxy server that talks directly to the perl code ... perhaps as a open software project What do you mean by 'a custom reversed modproxy server that talks directly to the perl code'? --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: Class::DBI and CGI::Application under mod_perl
Clinton Gormley wrote: I have recently had a bit of exposure to Class:DBI and CGI::Application in a project running under CGI. [...] What are the experiences/opinions of using these modules under mod_perl 2 on high traffic sites? I'm currently using them on mod_perl 1, but it shouldn't matter. CGI::Application is lightweight and fast. It will not give you any trouble. Class::DBI is significantly (about an order of magnitude) slower than hand-coded DBI objects, but saves you a lot of development time, so my current approach is to use it and replace things that end up being too slow. If you want to try something else, the developer of Rose::DB::Object has a benchmark that shows his module falling somewhere in between Class::DBI and straight DBI calls, although it's still much slower than DBI. - Perrin Sorry for the very late reply... You might want to check this out: http://www.perlmonks.org/?node_id=504724 I quite agree with it... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
RE: Masquerading requests as HTTPS
Hi Mark,
From my limited knowledge, SSL handshake is processed prior doing the HTTP
request-response. Therefore, when apache or mod_perl accepts HTTPS requests,
it can't redirect it over HTTPS unless you create another HTTPS request with
LWP or WWW::Mechanize, for example.
But if you want to connect to a backend server, why do you need a secure
connection anyway?
However, other probably know better...
---
Badai Aqrandista
Cheepy (?)
From: Mark Moseley [EMAIL PROTECTED]
Reply-To: [EMAIL PROTECTED]
To: modperl@perl.apache.org
Subject: Masquerading requests as HTTPS
Date: Thu, 15 Sep 2005 17:05:34 -0700
Greetings. I've been scouring the list and the net for a solution for this
but my apologies in advance if I didn't get the search terms right and
missed a RTFM answer.
I work for a web hosting company and we recently purchased a pair of
BigIPs.
These have the ability to terminate SSL connections and then send regular
HTTP to the backend servers (running Apache 1.3.29/mod_perl 1.29 and making
heavy use of transhandlers).
My question for the list is this:
Is there any mod_perl-ish to pretend that a request is coming through SSL?
The main issue I'm facing is that I've got a couple hundred thousands
users,
many using mod_rewrite in .htaccess files (and a potentially very very
large
number and out of my control so modifying them is not an option). Since the
BigIP is retransmitting the request as HTTP, the scheme that the backend
server is using is 'http', not 'https'. So if a redirect is generated via
mod_rewrite, it's redirecting to http://the/url/etc, not
https://the/url/etc.
Presumably people doing redirects via PHP would have the same issue.
However
the URL scheme looks to be somewhat inaccessible from $r and if I parse it
with Apache::URI and use the scheme method there, I don't know of a $r
method to set the URI to the modified URI object.
I've tried setting $ENV{ HTTPS } to 'on', but that didn't help.
Anybody know of a method I might be missing that would help me out?
Basically what I'm trying to accomplish is if I see a request coming in on
port 443, I want to set $r-method( https ) -- and I know there's no
method called this, but for explanation sake ;)
Thanks!
_
Sell your car for $9 on carpoint.com.au
http://www.carpoint.com.au/sellyourcar
Re: Crontab entry from perl
I want to schedule a perl script in crontab using a CGI script. If i try to append in the file ie /var/spool/cron/username then apache user does not have rights to modify this. If i give root rights to apache then it could be very harmful . how can i make entry into cron from script. You could have the CGI write out a file in /tmp or other and then have a permenant root run cron or trivial dameon poll this file and add to a users crontab based on the contents. I don't see what this has to do with modperl though :) or roll your own cron: http://search.cpan.org/~roland/Schedule-Cron-0.9/Cron.pm --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size
My app is an online hotel booking system. That part is the part where it searches availabilities through all combination of dates, number of guests, rooms, packages, allotments, pricing structures, and more. This doesn't sound like a search at all. Typically individuals know at least SOMETHING about the booking they want to make. How many times do you book a hotel, but don't care whether it's in Barbados or Brazil? Yes, the user provides the dates and the number of guests (and probably a coupon number). Then based on that, the system searches for available room and package combinations on the particular hotel, based on various constraints. --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: maintaining shared memory size
POE sounds good to start with, right? For a daemon to monitor a queue? Probably not. I'd just write a simple script that polls a database table and forks a new process to handle any jobs it finds. Parallel::ForkManager is handy for this kind of thing. Yes. POE looks overkill compare to Parallel::ForkManager. Thanks. --- Badai Aqrandista Cheepy (?) _ Low rate ANZ MasterCard. Apply now! http://clk.atdmt.com/MAU/go/msnnkanz003006mau/direct/01/ Must be over 18 years.
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Hi, I see from an earlier post on the mason-users list that your app is using HTML::Mason and Apache::Session::Memcached. It seems like you've got quite a few variables to juggle in your performance bottleneck analysis. Actually, I am using TT and Mason in this application. TT is used to support templates from the old version that we did not have time to port to Mason. The part that needs the speed is the part that uses TT. Have you tried switching to, say, Apache::Session::MySQL to see if there is any difference? I used to use A::S::MySQL, but it created 2 connections for every process. This caused 'Too many connections' error. So I tried to use memcached. Now that I know how to make one connection per process (using database.table identifier on all sql queries), I'll probably try to use A::S::MySQL again. One cheap way to check for memory leaks (most probably introduced by your perl code if they exist) is to run httpd -X (non-forking mode) and watch the memory usage of httpd while you send some requests. You may also want to strace the process: # httpd -X # strace -o /tmp/strace.out -p `pidof httpd` # tail -f /tmp/strace.out Then hit the server with some requests and watch the strace output. This is especially useful for finding I/O or IPC bottlenecks. Hmmm... Interesting... I'll give it a try... good luck Thanks... --- Badai Aqrandista Cheepy (?) _ Dating? Try Lavalife get 7 days FREE! Sign up NOW. http://www.lavalife.com/clickthru/clickthru.act?id=ninemsncontext=an99a=20233locale=en_AU_t=33473
Re: maintaining shared memory size (was: Re:swampedwithconnection?)
Hi, Rewriting things in C is a last resort. If you must do it, try Inline::C. Yes... Inline::C looks good... Your requests take 110 seconds each? What is your application doing? My app is an online hotel booking system. That part is the part where it searches availabilities through all combination of dates, number of guests, rooms, packages, allotments, pricing structures, and more. I must admit there is too much to handle in one mod_perl process. But I don't know any better way to design and code it. In the future, I'll probably move that search feature to a new process, so all the combinations can be calculated in parallel. Thanks for helping... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
You *do* have KeepAlive off in your httpd, right? No... I mean in the backend Apache, not the frontend whatever. Yes, I understand... I put it in the backend... When you're happering your server, is the CPU on the server running at or near 100%? If not, you have other problems. Almost 100%, around 90%... --- Badai Aqrandista Cheepy (?) _ Your opinion counts..for your chance to win a Mini Cooper click here http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: maintaining shared memory size
Okay, so it's slow because it does a lot of work. How does it do this search? Is it a bunch of SQL queries, or a request to a remote server of some other kind, or an internal calculation? Is there some possible caching you can do? Lots of SQL queries and calculation based on the query results. I've cached most of the query results, but the calculation results can't be cached because they depend on user inputs. I would probably use some sort of queue system here. I would have the mod_perl process push requests onto a queue (probably in a database) and then have a separate daemon, maybe on a different machine, do the work and return the answer. It would allow you to split the search into multiple pieces that happen in parallel, and handle them on multiple machines simultaneously. It would also separate the web request, which is fast, from the slow search request. That lets your server scale to handling many more clients at once, even though some of them are just waiting for their turn. Good idea!!! Thanks... You've been a great help... POE sounds good to start with, right? --- Badai Aqrandista Cheepy (?) _ Dating? Try Lavalife get 7 days FREE! Sign up NOW. http://www.lavalife.com/clickthru/clickthru.act?id=ninemsncontext=an99a=20233locale=en_AU_t=33473
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
On 29 Aug 2005, at 01:15, Badai Aqrandista wrote: I think I have to write some of the code in C. I can't find any other places in the code to optimize (or probably I uncounciously don't want to make changes because I don't have any test suites)... Then how do you know what to write in C? The part that is called the most... I have localized it to one or two subroutines... But I haven't done so... My C skill is very2 rusty... Franky, if your code is CPU bound (rather than waiting for the database) then a MaxClients of 3 will serve your purposes. I'll give it a try... You *do* have KeepAlive off in your httpd, right? No... Thanks a lot for the enlightment... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Then how do you know what to write in C? I have localized one subroutine that has been heavily called in the search function. That should be the way to pick the candidate, shouldn't it? I have never done this and I am worried that writing it in C would push the project's deadline a little bit further. However, does anyone have any hint on doing this? What are the recommended readings? Anyhow, I've decrease the MaxClients to 3 and it worked!!! The speed significantly increases, from average of 110 sec/request to 85 sec/request. But, that's not enough. I need less than 10 sec/request. Is that possible without changing hardware? --- Badai Aqrandista Cheepy (?) _ Sell your car for $9 on carpoint.com.au http://www.carpoint.com.au/sellyourcar
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
top - 17:24:27 up 34 days, 9:01, 4 users, load average: 20.67, 12.84, 9.26 Tasks: 142 total, 7 running, 135 sleeping, 0 stopped, 0 zombie Cpu(s): 88.7% us, 7.6% sy, 0.0% ni, 0.0% id, 2.0% wa, 0.0% hi, 1.7% si Mem:906736k total, 359464k used, 547272k free, 6184k buffers Swap: 3036232k total, 111564k used, 2924668k free,17420k cached Which processes are swapping? Have you tried setting MaxClients to say 10 (or some other low number that'll ensure you don't run out of memory). I just did, and cleared the swap before doing another testing. The result: Nothing swapped, but the average response time goes down by 20 seconds. I guess Perrin was right, with 30 MaxClients, the processes are competing the CPU cycles. I think I have to write some of the code in C. I can't find any other places in the code to optimize (or probably I uncounciously don't want to make changes because I don't have any test suites)... --- Badai Aqrandista Cheepy (?) _ Low rate ANZ MasterCard. Apply now! http://clk.atdmt.com/MAU/go/msnnkanz003006mau/direct/01/ Must be over 18 years.
Re: maintaining shared memory size (was: Re:swampedwithconnection?)
If the performance you get is not enough for you, look for the bottleneck. This could be running out of CPU, running out of memory, or contention for a shared resource like a database. Then you fix it, by changing code or buying hardware. I think I've got the point now: To reduce memory consumption - look for ways to do more things on compile time To reduce CPU usage - write the most called subroutine in C To reduce database contention - cache as much as possible Thanks for all your help... The journey just begins... --- Badai Aqrandista Cheepy (?) _ Dating? Try Lavalife get 7 days FREE! Sign up NOW. http://www.lavalife.com/clickthru/clickthru.act?id=ninemsncontext=an99a=20233locale=en_AU_t=33473
Re: maintaining shared memory size (was: Re: swampedwithconnection?)
Does this sound like fixing the wrong problem? Yes. Put a reverse proxy in front of your server, tune MaxClients so you won't go into swap, and then benchmark to see how much load you can handle. Then think about tuning. Thanks for replying... I have put a reverse procy in front of my mod_perl servers and I have set MaxClient to 30. I have tried setting it to 50, but it slows down the response time. This is what top gave me when I hammered the test server with httperf: -- top - 17:24:27 up 34 days, 9:01, 4 users, load average: 20.67, 12.84, 9.26 Tasks: 142 total, 7 running, 135 sleeping, 0 stopped, 0 zombie Cpu(s): 88.7% us, 7.6% sy, 0.0% ni, 0.0% id, 2.0% wa, 0.0% hi, 1.7% si Mem:906736k total, 359464k used, 547272k free, 6184k buffers Swap: 3036232k total, 111564k used, 2924668k free,17420k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 22802 www-data 15 0 39664 35m 7968 S 30.7 4.0 0:05.02 apache-perl 22370 www-data 16 0 39664 35m 7968 S 12.8 4.0 0:03.45 apache-perl 13604 www-data 17 0 40096 35m 7968 R 5.7 4.0 0:30.93 apache-perl 15424 root 15 0 32060 7668 1560 S 3.8 0.8 6:49.47 memcached 13611 www-data 15 0 40036 35m 7968 S 3.5 4.0 0:17.13 apache-perl 22804 www-data 17 0 39664 35m 7968 R 3.1 4.0 0:03.01 apache-perl 13612 www-data 16 0 40032 35m 7968 S 0.7 4.0 0:33.38 apache-perl I ran httperf to create 50 connections. This is the result: 1 connections per second - avg reply time = 103953.7 ms 10 connections per second - avg reply time = 123167.2 ms 20 connections per second - avg reply time = 121483.7 ms 30 connections per second - avg reply time = 114411.3 ms 40 connections per second - avg reply time = 130168.7 ms 50 connections per second - avg reply time = 130457.4 ms When only creating 1 connection, the avg reply time = 3289.4 ms When creating 10 connections, with 1 conn per second, the avg reply time = 25929.7 ms I have no idea where to start fixing this. It seems that the more connection there is, the higher the avg reply time. Does anyone ever experienced this? Why does it happen? Basically I just want to make my webapp ready to launch... (doesn't anyone?) THANK YOU... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
Re: maintaining shared memory size (was: Re:swampedwithconnection?)
Hi all, If the performance you get is not enough for you, look for the bottleneck. This could be running out of CPU, running out of memory, or contention for a shared resource like a database. Then you fix it, by changing code or buying hardware. I'll try that... Thanks for helping me out... Especially perrin... I'll let you know how it goes... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
maintaining shared memory size (was: Re: swamped with connection?)
I think I have to reword the question: How do I maintain the size of the shared memory between apache children? What cause a memory page to be copied (not shared) from perl's point of view? This brings the question of how to increase shared memory usage. I've tried to load every modules upfront. But even before any request comes in, the shared memory is only 7 Mb. What makes it so small? Thanks... --- Badai Aqrandista Cheepy (?) _ Your opinion counts..for your chance to win a Mini Cooper click here http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: maintaining shared memory size (was: Re: swamped withconnection?)
On Tue, 2005-08-23 at 17:23 +1000, Badai Aqrandista wrote: How do I maintain the size of the shared memory between apache children? What cause a memory page to be copied (not shared) from perl's point of view? Anything that writes to memory -- modifying any variable (even just reading one in a different context) or compiling some code are the most common things. There's a bit more here: http://modperlbook.com/html/ch10_01.html Oh my, I should've kept memory consumption in mind when designing the code in the first place. Anyway, to fix this, I'm trying to make my onw shared memory with Apache::SharedMem. But it seems that shared memory is just a memory area that any process can read from or write to, not a memory that can become part of any process's memory space. I'd like to put the read only data in a memory area that can become part of any process's memory space, like threads sharing memory space. Is there any way I can do that? How about mmap? I have never looked into that. Does this sound like fixing the wrong problem? Thank you... --- Badai Aqrandista Cheepy (?) _ On the road to retirement? Check out MSN Life Events for advice on how to get there! http://lifeevents.msn.com/category.aspx?cid=Retirement
swamped with connection?
Hi all, it's me again :D I am still trying to improve my company's webapp performance. I'm testing it with httperf and autobench. The application seems to be able to respond when hammered by 20 connections per second and 10 calls per connection. But then, it doesn't respond to any request when the connection rate is raised to 40 (with 10 calls per connection) and above. What does the apache treats as a request (hence forks another child)? - every new connection, or - every http request regardless of it comes with a new connection or comes through an existing connection Does anyone have any idea what's going on? My only guess is because the connections goes over the MaxClients count while the existing apache children are still processing the previous requests. But why, then, none of the requests after 40 conn/sec is responded? I'll give more detail if requested. THANKS A LOT!!! --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: swamped with connection?
Did you run out of memory? Is the CPU pegged? Give us something to go on here... Have you read the performance tuning docs on the mod_perl site? Yes, ages ago. I just read it again and did the calculation again. Apparently, yes, it runs out of memory and holds up the connection so no other requests can be processed. So, here's the variables: RAM = 700 Mb Per process total size = 40 Mb Shared memory = 7 Mb So, the number of processes = (700 - 7) / 33 = 21 processes So, does that mean it can only accept up to 21 connections? This brings the question of how to increase shared memory usage. I've tried to load every modules upfront. But even before any request comes in, the shared memory is only 7 Mb. What makes it so small? Thank you... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: Scope Question
So, let's reverse this.
I have a module that will be used in perl.
I want all subs in the module to share the stagte of one variable...in my
case @context.
I also want any changes to @context to only effect the current user/request
under mod_perl.
What's the best approach?
1. declare @context with 'my' in the package
2. add this to your module:
sub clean_context { @context = () }
3. add this to your config file:
PerlCleanupHandler MyApp::clean_context
I think that'll do the trick...
---
Badai Aqrandista
Cheepy (?)
_
REALESTATE: biggest buy/rent/share listings
http://ninemsn.realestate.com.au
Re: apache children waits for each other?
I don't know it's even possible in mysql. Its reference docs doesn't say that it is (http://dev.mysql.com/doc/mysql/en/join.html). Yes it does : http://dev.mysql.com/doc/mysql/en/identifier-qualifiers.html Hmmm... Excellent!!! I'll use it then...Will let you know how it goes... Thanks, mate... --- Badai Aqrandista Cheepy (?) _ Single? Start dating at Lavalife. Try our 7 day FREE trial! http://lavalife9.ninemsn.com.au/clickthru/clickthru.act?context=an99locale=en_AUa=19179
Re: Clear Registry
I use Apache::StatINC It works fine for me, I haven't noticed needing to restart with it Except you have two modules A and B, where A's BEGIN block depends on B's BEGIN block being executed beforehand, and you only change module A. In that case you definitely must restart. Got bitten by that on the last project... Cheers... --- Badai Aqrandista Cheepy (?) _ Low rate ANZ MasterCard. Apply now! http://clk.atdmt.com/MAU/go/msnnkanz003006mau/direct/01/ Must be over 18 years.
Re: apache children waits for each other?
Hi, I'm assuming that each client database is contained within the same MySQL server, and that you're not running 300 MySQL servers on different machines or different ports? Yes, why would I do that? If so, you can reuse the same connections, and just reference the different database in the query.So : select * from db1.table1 select * from db2.table1 When you connect and specify a database name, you're just specifying the default database. I don't know it's even possible in mysql. Its reference docs doesn't say that it is (http://dev.mysql.com/doc/mysql/en/join.html). Thanks... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: apache children waits for each other?
I also suspect that you didn't initialize the debugger before compiling your code, since none of your code appears in this output. I got the previous output because I put in the debugger initialization... Without initialization (Apache::DB-init), this is the top 5 output before the cache is filled (with dprofpp -r): Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 33.5 23.74 5.224 1 23.747 5.2236 ELRes::ApacheHandler::handler 1.09 0.772 0.772 7869 0.0001 0.0001 Storable::mretrieve 0.87 0.615 0.615 1418 0.0004 0.0004 Storable::net_mstore 0.70 0.495 0.495 1524 0.0003 0.0003 DBD::mysql::db::_login 0.54 0.380 0.380 2902 0.0001 0.0001 DBI::_new_sth 0.51 0.359 0.388 8852 0. 0. DBI::common::DESTROY and this one is after it's filled: Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 16.5 13.85 14.780 1 13.857 14.780 ELRes::ApacheHandler::handler 0.21 0.174 0.174 34766 0. 0. Date::Simple::DESTROY 0.20 0.170 0.170 6580 0. 0. Storable::mretrieve 0.19 0.162 0.162 99576 0. 0. Date::Simple::as_iso 0.14 0.115 0.115 27854 0. 0. Date::Simple::_add 0.11 0.090 0.090 25 0.0036 0.0036 DBD::mysql::db::_login How should I interpret these outputs and what should I do about it to improve my app's speed? This is the first time for me to optimize a large perl codebase... So I need a bit of guidance here... Thanks a lot... --- Badai Aqrandista Cheepy (?) _ Your opinion counts..for your chance to win a Mini Cooper click here http://www.qualifiedopinions.com/joinup.php?source=hotmail
Re: apache children waits for each other?
I do all of my memcached stuff , including sessions, with mysql failover.
its barely more code - i just make every public function address two
private functions.
ie:
sub save {
$_[0]-_save_memcached();
$_[0]-_save_mysql();
}
sub load {
if ( !$_[0]-_load_memcached() )
{
$_[0]-_load_mysql();
}
}
I thought about the same thing last night, but is there any race condition
problem if I use it for session?
---
Badai Aqrandista
Cheepy (?)
_
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: apache children waits for each other?
Was the previous output sorted with the -r flag though? No. There's no trick, it's the same as in any language: find out what the slow parts are by using a profiler and tinker with them until they are faster. You can look at the tuning information on http://modperlbook.org/ for some advice as well. And don't forget that there is a profiler for DBI queries that comes with DBI. I've done the database profiling as well... Thanks for your help... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: apache children waits for each other?
Okay, send it again with the top 10, sorted by -r then. Maybe we can
make more suggestions.
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
20.5 14.30 15.222 1 14.305 15.222 ELRes::ApacheHandler::handler
0.37 0.261 0.261 99576 0. 0. Date::Simple::as_iso
0.22 0.151 0.153 20364 0. 0.
Params::Validate::_check_regex_fro
m_xs
0.17 0.117 0.117 6580 0. 0. Storable::mretrieve
0.13 0.093 0.093 27854 0. 0. Date::Simple::_add
0.09 0.065 0.065 2900 0. 0. ELRes::Entity::DESTROY
0.08 0.058 0.058 34766 0. 0. Date::Simple::DESTROY
0.08 0.054 0.054 41678 0. 0. Date::Simple::_compare
0.05 0.035 0.035 12984 0. 0. Date::Simple::__ANON__
0.03 0.020 0.020 36 0.0006 0.0005 Template::Context::filter
0.03 0.019 0.019102 0.0002 0.0002 DBI::common::DESTROY
0.01 0.010 0.010 1 0.0100 0.0100 DynaLoader::bootstrap
0.01 0.010 0.010 1 0.0100 0.0100 ELRes::App::Session::DESTROY
0.01 0.010 -0.000 2 0.0050 -
ELRes::ApacheHandler::trans_handle
0. r
0.01 0.010 0.010 25 0.0004 0.0004 DBD::mysql::db::_login
As I said previously, the handler itself tops everything... I don't know why
it happens... If I use Devel::Timer to profile its parts, I found that this
block takes the most time to run (99.98%):
eval { $app-run };
if ($@) {
$r-log_error(Server error: $@);
$app-send_error(err = $@);
}
But the call should not be included in the handler's ExclSec time, shouldn't
it? The eval doesn't make the call included in the ExclSec time because I've
tested without it and it still use 99.98% of the time...
Wierd eh?
Thanks a lot...
---
Badai Aqrandista
Cheepy (?)
_
Sell your car for $9 on carpoint.com.au
http://www.carpoint.com.au/sellyourcar
Re: apache children waits for each other?
It does look odd. Maybe you are still loading some code before calling
Apache::DB-init(). Can you post your httpd.conf, or at least the mod_perl
part of it?
start of mod_perl configuration ...
Perl
use lib '/i4u/web/elres-mp';
#require Apache::DB;
#Apache::DB-init;
/Perl
PerlRequire /i4u/web/elres-mp/etc/startup.pl
PerlModule Apache::DProf
PerlModule ELRes::ELRes
PerlModule ELRes::ApacheHandler
PerlTransHandler ELRes::ApacheHandler-trans_handler
PerlLogHandler ELRes::ApacheHandler-log_handler
PerlCleanupHandler ELRes::ApacheHandler-cleanup_handler
... end mod_perl configuration ...
the startup.pl script loads and require's all modules under ELRes::
namespace on startup time, as suggested in mod_perl documentation...
eval { $app-run }; if ($@) { $r-log_error(Server error: $@);
$app-send_error(err = $@); }
That's why I think you are loading some code before initializing the
debugger -- when it doesn't know about those other subs, they don't show up
in the profile.
the object that $app refer to is dynamically decided based on the hostname,
for example:
a.example.com --- $app = ELRes::Property-new()
b.example.com --- $app = ELRes::Property-new()
c.example.com --- $app = ELRes::Distributor-new()
both ELRes::Property and ELRes::Distributor are children of
ELRes::TopLevelEntity and have run() method...
does this prevents the debugger to know which subroutine should be profiled?
all the test that I've done is on the hostname whose object is of class
ELRes::Property...
Anyway,
Hmmm... Where can I learn more about the debugger?
I didn't know profiling needs debugger, not until I was told about it in
this mailinglist...
Cheers...
---
Badai Aqrandista
Cheepy (?)
_
SEEK: Over 80,000 jobs across all industries at Australia's #1 job site.
http://ninemsn.seek.com.au?hotmail
Re: apache children waits for each other?
Apache::DProf uses the debugger hooks. You can also try Sam Tregar's Devel::Profiler::Apache which is somewhat slower but easier to use since it doesn't use the debugger. I'll look into that... Thanks for your suggestions... --- Badai Aqrandista Cheepy (?) _ REALESTATE: biggest buy/rent/share listings http://ninemsn.realestate.com.au
Re: web application speed problem with OO modules
First, give dprofpp the flag that makes it sort by wall time, not by CPU time. Otherwise, things that take most of the time in the average program, like I/O, will appear to be very fast. Second, make sure you initialize the debugger before loading any of the code that you want to profile, like this: Perl require Apache::DB; Apache::DB-init; /Perl Any code that you compile (e.g. use from startup.pl) before this will not get profiled at all. - Perrin Thanks for the replies... I'll give it a try... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
apache children waits for each other? (was: Re: web application speed problem)
Hi all, My mod_perl web app uses memcached to cache most of the (MySQL) database query results and as the session storage (Apache::Session::Memcached). When doing performance tests with httperf, I found that apache processes waits for each other. I mean: all requests are accepted (apache forks off lots of children), then processed (no process finishes), and after a while, all the processes finish almost at the same time... If I put warns in the apache handler's start and end, it gives this output (edited for brevity, md5 hash in the end is the session ids, just disregard them): ... output from 90+ of other warns... Start process 30324: Tue, 16 Aug 2005 11:00:23 EST,af263d03f531584c4d05c7f3df55c7d4 Start process 30326: Tue, 16 Aug 2005 11:00:23 EST,931f8a3a2de409fb3285c3bd038ec7b6 Start process 30325: Tue, 16 Aug 2005 11:00:23 EST,48ca7271dab9037f3a7426a6a68789e5 Start process 30314: Tue, 16 Aug 2005 11:00:23 EST,259709a683879491498cb81055466948 ... after a while, suddenly the following output comes out ... End process 29803: Tue, 16 Aug 2005 11:05:47 EST,dc7e7c893f0ee15b0b6c94d061f9493c End process 30337: Tue, 16 Aug 2005 11:05:49 EST,73f520865a13715e02d183c5cbb920d4 End process 30002: Tue, 16 Aug 2005 11:05:49 EST,cbfb00260afade004d3cc3086f425272 End process 30003: Tue, 16 Aug 2005 11:05:51 EST,b84da97b68df5f54e336a7767d64a1f8 ... output of other 90+ warns ... It's like they are not processed concurrently. I wonder why does it happen? Is it memcached? Or is it how apache works? Or is it because of the benchmarking application that I use? Thanks... --- Badai Aqrandista Cheepy (?) _ SEEK: Over 80,000 jobs across all industries at Australia's #1 job site. http://ninemsn.seek.com.au?hotmail
Re: web application speed problem with OO modules
Thanks for replying... I have done some profiling with Apache::DProfile, and this is the output of 'dprofpp tmon.out': Total Elapsed Time = 160.2141 Seconds User+System Time = 13.07412 Seconds Exclusive Times %Time ExclSec CumulS #Calls sec/call Csec/c Name 92.7 12.12 13.575 1 12.127 13.574 ELRes::ApacheHandler::handler 3.13 0.409 0.409 90360 0. 0. Date::Simple::as_iso 2.20 0.288 0.366 20364 0. 0. Params::Validate::_check_regex_fro m_xs 1.53 0.200 0.200 29966 0. 0. Date::Simple::DESTROY 1.35 0.177 0.177 6565 0. 0. Storable::mretrieve 0.66 0.086 0.086 36878 0. 0. Date::Simple::_compare 0.58 0.076 0.076 1761 0. 0. DBD::_::db::prepare_cached 0.46 0.060 0.060 24782 0. 0. Date::Simple::_add 0.32 0.042 0.042 2900 0. 0. ELRes::Entity::DESTROY 0.25 0.033 0.033 3700 0. 0. DBI::common::FETCH 0.18 0.023 0.023 3692 0. 0. DBI::st::fetch 0.12 0.016 0.016 1756 0. 0. ELRes::DB::DESTROY 0.08 0.011 0.011 9528 0. 0. Date::Simple::__ANON__ 0.08 0.010 0.010 1 0.0100 0.0100 DynaLoader::bootstrap 0.08 0.010 0.010 2 0.0050 0.0050 DBD::mysql::db::_login I doubt the ApacheHandler::handler subroutine runs that long, because it only shows the accumulated time used by other subroutines it calls. That makes the profile isn't really accurate. How do I make it to show the statistics of the subroutines called from the handler? Thank you... --- Badai Aqrandista Cheepy (?) From: Philip M. Gollucci [EMAIL PROTECTED] To: Badai Aqrandista [EMAIL PROTECTED] CC: modperl@perl.apache.org Subject: Re: web application speed problem with OO modules Date: Mon, 15 Aug 2005 00:32:09 -0400 (I only try to improve the speed of the code, so the use of proxy shouldn't be relevant here - I think, I could be wrong). You might try using Apache::DProf and/or Apache::SmallProf available here to see where your code is spending its time http://search.cpan.org/~fwiles/Apache-DB _ Low rate ANZ MasterCard. Apply now! http://clk.atdmt.com/MAU/go/msnnkanz003006mau/direct/01/ Must be over 18 years.
