Re: Re: Debugging a long process
Hi Fred, I like the idea of Apache may be blocking on other processes but the Devel::NYTProf docs don't seem to indicate if it would be helpful in tracking down the reason for a 30 second execution under mod_perl versus a 9 second execution from the shell. Also there seems to be some issues WRT Devel::NYTProf and virtual machines, which I believe my EC2 instance is one of. Is there anywhere to point me to find out more about the blocking issue and how to resolve it? Thanks! Tosh On 7/22/64 8:59 PM, Fred Moyer wrote: On Wed, Jun 8, 2011 at 12:50 PM, Tosh Cooeyt...@1200group.com wrote: I have a mod_perl Registry script that is taking about 30 seconds to execute. I am trying to debug it but when I run the script from the command line or the debugger it only takes 9 seconds to execute. Sounds like Apache may be blocking on other processes? Have you tried Devel::NYTProf with your program on the command line? I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0 Is there something obvious I should be looking at, and does anyone have any tips for debugging something like this? Thanks! Tosh -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/ -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Debugging a long process
Hey Tosh From my experience, I would guess you may seeing interactions between the major components of your system as load grows. If my hunch is true, then following a single request may not show the issue. You may need a way to load your system to the point where it shows the issue. If all these guesses are true, NYTProf, while an excellent tool, didn't really help me and may not be much help to you. You mention you may be running in an EC2 instance, which adds another layer at which your code may be getting suspended. When I faced a similar problem, I resolved it by: - create a test environment that I could push to show the problem - instrument major blocks of the code with markers and tv_interval() calls so I could see where time was being spent. - write a row of timing data for each request - post process the timing data spread over, in my case, many many individual requests so I could see how response time changed as load increased This took me a few days to get worked out, but once I had it in place, I was able to tune my system more effectively. I could 'see' where the bottlenecks were occurring as load increased. In my case, it pointed to a single component in my stack which was the bottleneck. So I fixed that one, then fixed the next hot spot ... you get the picture. There are a lot of assumptions here that may or may not actually be true in your case... HTH, -- jeff On 6/15/11 1:29 PM, Tosh Cooey t...@1200group.com wrote: Hi Fred, I like the idea of Apache may be blocking on other processes but the Devel::NYTProf docs don't seem to indicate if it would be helpful in tracking down the reason for a 30 second execution under mod_perl versus a 9 second execution from the shell. Also there seems to be some issues WRT Devel::NYTProf and virtual machines, which I believe my EC2 instance is one of. Is there anywhere to point me to find out more about the blocking issue and how to resolve it?
Re: Re: Debugging a long process
On Wed, Jun 15, 2011 at 1:29 PM, Tosh Cooey t...@1200group.com wrote: Hi Fred, I like the idea of Apache may be blocking on other processes but the Devel::NYTProf docs don't seem to indicate if it would be helpful in tracking down the reason for a 30 second execution under mod_perl versus a 9 second execution from the shell. Also there seems to be some issues WRT Devel::NYTProf and virtual machines, which I believe my EC2 instance is one of. Have you tried running your program under Devel::NYTProf in your development environment and using that to identify bottlenecks?
Re: Debugging a long process
Hi Jeff, this is very sensible and will likely have to be what gets done. To respond to Fred, my DEV environment is also a small EC2 instance since I want to mirror my production environment. This line of thought prodded me to check my environments and it seems that the PROD Apache is running the worker MPM while DEV is running prefork, and boy is there a performance difference. I am waiting to hear back from my server admin before I kill him, but it seems that maybe there's some issues with my Ubuntu and the worker MPM on a small EC2 instance. Thank-you both for your insights! Tosh On 6/15/11 11:00 PM, McCarrell, Jeff wrote: Hey Tosh From my experience, I would guess you may seeing interactions between the major components of your system as load grows. If my hunch is true, then following a single request may not show the issue. You may need a way to load your system to the point where it shows the issue. If all these guesses are true, NYTProf, while an excellent tool, didn't really help me and may not be much help to you. You mention you may be running in an EC2 instance, which adds another layer at which your code may be getting suspended. When I faced a similar problem, I resolved it by: - create a test environment that I could push to show the problem - instrument major blocks of the code with markers and tv_interval() calls so I could see where time was being spent. - write a row of timing data for each request - post process the timing data spread over, in my case, many many individual requests so I could see how response time changed as load increased This took me a few days to get worked out, but once I had it in place, I was able to tune my system more effectively. I could 'see' where the bottlenecks were occurring as load increased. In my case, it pointed to a single component in my stack which was the bottleneck. So I fixed that one, then fixed the next hot spot ... you get the picture. There are a lot of assumptions here that may or may not actually be true in your case... HTH, -- jeff On 6/15/11 1:29 PM, Tosh Cooeyt...@1200group.com wrote: Hi Fred, I like the idea of Apache may be blocking on other processes but the Devel::NYTProf docs don't seem to indicate if it would be helpful in tracking down the reason for a 30 second execution under mod_perl versus a 9 second execution from the shell. Also there seems to be some issues WRT Devel::NYTProf and virtual machines, which I believe my EC2 instance is one of. Is there anywhere to point me to find out more about the blocking issue and how to resolve it? -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Debugging a long process
On 15 Jun 2011, at 22:53, Tosh Cooey wrote: I am waiting to hear back from my server admin before I kill him, but it seems that maybe there's some issues with my Ubuntu and the worker MPM on a small EC2 instance. I've been doing this stuff since web 1.0 and have always found prefork to be sensible and deterministic. I'd go with the defenestration option.
Re: Debugging a long process
I haven't tried Devel::NYTProf, but the blocking seems to have been the keyword here... root 2661 0.0 1.5 37900 27492 ?Ss Jun08 0:00 /usr/sbin/apache2 -k start www-data 3875 0.0 1.3 37264 23300 ?S06:25 0:00 /usr/sbin/apache2 -k start www-data 3882 3.9 8.7 395156 153644 ? Sl 06:25 6:15 /usr/sbin/apache2 -k start As you can see I've only got three Apache+mod_perl processes running, which I suspect is causing the blocking issues since there aren't enough processes being spawned to handle the incoming requests. Oddly enough I used to have the problem that this application was spawning too many processes, now I have the opposite problem. Anyone have sweet spots for Apache settings running on an EC2 small instance? Thanks Fred! Tosh On 6/8/11 10:15 PM, Fred Moyer wrote: On Wed, Jun 8, 2011 at 12:50 PM, Tosh Cooeyt...@1200group.com wrote: I have a mod_perl Registry script that is taking about 30 seconds to execute. I am trying to debug it but when I run the script from the command line or the debugger it only takes 9 seconds to execute. Sounds like Apache may be blocking on other processes? Have you tried Devel::NYTProf with your program on the command line? I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0 Is there something obvious I should be looking at, and does anyone have any tips for debugging something like this? Thanks! Tosh -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/ -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Debugging a long process
On 9 Jun 2011, at 17:13, Tosh Cooey wrote: I haven't tried Devel::NYTProf, but the blocking seems to have been the keyword here... root 2661 0.0 1.5 37900 27492 ?Ss Jun08 0:00 /usr/sbin/apache2 -k start www-data 3875 0.0 1.3 37264 23300 ?S06:25 0:00 /usr/sbin/apache2 -k start www-data 3882 3.9 8.7 395156 153644 ? Sl 06:25 6:15 /usr/sbin/apache2 -k start Very much looks like you don't have a startup.pl that loads the world in the parent process. That will cause a masive inhale on the first hit. Anyone have sweet spots for Apache settings running on an EC2 small instance? Whatever works for you after measurement. Read the mod_perl tuning guide and settle on a strategy of MaxRequestsPerClient or a fixed Apache2::SizeLimit. Also, are you running stock apache/perl/mod_perl? Seriously consider spending time making your own build. Even recompiling the .srpms with gentoo-safe flags for that $arch could give a healthy boost. Or pay someone *cough* *cough* to do it for you :)
Debugging a long process
I have a mod_perl Registry script that is taking about 30 seconds to execute. I am trying to debug it but when I run the script from the command line or the debugger it only takes 9 seconds to execute. I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0 Is there something obvious I should be looking at, and does anyone have any tips for debugging something like this? Thanks! Tosh -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Debugging a long process
On Wed, Jun 8, 2011 at 12:50 PM, Tosh Cooey t...@1200group.com wrote: I have a mod_perl Registry script that is taking about 30 seconds to execute. I am trying to debug it but when I run the script from the command line or the debugger it only takes 9 seconds to execute. Sounds like Apache may be blocking on other processes? Have you tried Devel::NYTProf with your program on the command line? I am running Apache/2.2.11 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.0 Is there something obvious I should be looking at, and does anyone have any tips for debugging something like this? Thanks! Tosh -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
