High CPU utilization on RHEL5.6/CentOS5.6
I have a MJPEG streaming system which uses mod_perl in the web interface to supply the final stream to the client. I am seeing high cpu utilization under RHEL5.6 which I don't see on FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on different hardware, but not that different. And on EL5 the httpd session is way higher (cpu wise) than the supplying proxy. On FC14 on my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which can handle multiplexing etc and 5.5-8% for the mod_perl httpd session. I think this is an issue with that platform, but I haven't seen any/many reports. Does anyone have a sec to look at the source? I don't think I have implemented this is a horrible inefficient way but who knows. It is true, I could have byte markers and then know the size of the JPEG I am expecting rather than just splitting on the boundary field, but in tests (and production) it seemed to work ok. http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl Any comments or critiques would be appreciated, Christopher -- On FC14: Requires: httpd >= 2.2.17-1 Requires: mod_perl >= 2.0.4-11 Requires: mod_ssl >= 1:2.2.17-1 Requires: perl >= 5.12.3-141 Requires: perl-CGI >= 3.51-1 Requires: perl-Digest-SHA1 >= 2.12-4 Requires: perl-XML-Filter-BufferText >= 1.01-9 Requires: perl-XML-Simple >= 2.18-7 Requires: perl-XML-Validator-Schema >= 1.10-5 Requires: perl-Tree-DAG_Node >= 1.06-8 Requires: perl-LDAP >= 0.40-2 Requires: perl-Crypt-SSLeay >= 0.58-1 Requires: ffmpeg >= 0.6-4 On EL5: Requires: httpd >= 2.2.3-45 Requires: mod_perl >= 2.0.4-6 Requires: mod_ssl >= 2.2.3-45 Requires: perl >= 5.8.8-32 Requires: perl-Digest-SHA1 >= 2.11-1 Requires: perl-XML-Filter-BufferText >= 1.01 Requires: perl-XML-Simple >= 2.14-4 Requires: perl-XML-Validator-Schema >= 1.10-1 Requires: perl-Tree-DAG_Node >= 1.06 Requires: perl-LDAP >= 0.33-3 Requires: perl-Crypt-SSLeay >= 0.51-11 Requires: ffmpeg >= 0.6.1-1
Re: High CPU utilization on RHEL5.6/CentOS5.6
This may not be related, but when I was working with Centos 5 in a Vmware environment, I ran into an issue of high cpu since the default clock rate with Linux 2.6 is 1000Hz. I changed to a 100Hz clock rate and got much lower cpu usage. I'm not sure if that is your issue, perhaps FC14 was using less than a 1000Hz clock rate. You might try a tickless kernel, or dig further into what clock rate your kernel is using. I don't know where the authoritative source for this information is, but here's what I found with a search: http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton wrote: > I have a MJPEG streaming system which uses mod_perl in the web > interface to supply the final stream to the client. > > I am seeing high cpu utilization under RHEL5.6 which I don't see on > FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on > different hardware, but not that different. And on EL5 the httpd > session is way higher (cpu wise) than the supplying proxy. On FC14 on > my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which > can handle multiplexing etc and 5.5-8% for the mod_perl httpd session. > > I think this is an issue with that platform, but I haven't seen > any/many reports. Does anyone have a sec to look at the source? I > don't think I have implemented this is a horrible inefficient way but > who knows. It is true, I could have byte markers and then know the > size of the JPEG I am expecting rather than just splitting on the > boundary field, but in tests (and production) it seemed to work ok. > > http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl > > Any comments or critiques would be appreciated, > Christopher > > -- > > On FC14: > Requires: httpd >= 2.2.17-1 > Requires: mod_perl >= 2.0.4-11 > Requires: mod_ssl >= 1:2.2.17-1 > Requires: perl >= 5.12.3-141 > Requires: perl-CGI >= 3.51-1 > Requires: perl-Digest-SHA1 >= 2.12-4 > Requires: perl-XML-Filter-BufferText >= 1.01-9 > Requires: perl-XML-Simple >= 2.18-7 > Requires: perl-XML-Validator-Schema >= 1.10-5 > Requires: perl-Tree-DAG_Node >= 1.06-8 > Requires: perl-LDAP >= 0.40-2 > Requires: perl-Crypt-SSLeay >= 0.58-1 > Requires: ffmpeg >= 0.6-4 > > On EL5: > Requires: httpd >= 2.2.3-45 > Requires: mod_perl >= 2.0.4-6 > Requires: mod_ssl >= 2.2.3-45 > Requires: perl >= 5.8.8-32 > Requires: perl-Digest-SHA1 >= 2.11-1 > Requires: perl-XML-Filter-BufferText >= 1.01 > Requires: perl-XML-Simple >= 2.14-4 > Requires: perl-XML-Validator-Schema >= 1.10-1 > Requires: perl-Tree-DAG_Node >= 1.06 > Requires: perl-LDAP >= 0.33-3 > Requires: perl-Crypt-SSLeay >= 0.51-11 > Requires: ffmpeg >= 0.6.1-1 >
Re: High CPU utilization on RHEL5.6/CentOS5.6
Neither install is running within VMWare. On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer wrote: > This may not be related, but when I was working with Centos 5 in a > Vmware environment, I ran into an issue of high cpu since the default > clock rate with Linux 2.6 is 1000Hz. I changed to a 100Hz clock rate > and got much lower cpu usage. > > I'm not sure if that is your issue, perhaps FC14 was using less than a > 1000Hz clock rate. You might try a tickless kernel, or dig further > into what clock rate your kernel is using. > > I don't know where the authoritative source for this information is, > but here's what I found with a search: > > http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare > > On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton > wrote: >> I have a MJPEG streaming system which uses mod_perl in the web >> interface to supply the final stream to the client. >> >> I am seeing high cpu utilization under RHEL5.6 which I don't see on >> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on >> different hardware, but not that different. And on EL5 the httpd >> session is way higher (cpu wise) than the supplying proxy. On FC14 on >> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which >> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session. >> >> I think this is an issue with that platform, but I haven't seen >> any/many reports. Does anyone have a sec to look at the source? I >> don't think I have implemented this is a horrible inefficient way but >> who knows. It is true, I could have byte markers and then know the >> size of the JPEG I am expecting rather than just splitting on the >> boundary field, but in tests (and production) it seemed to work ok. >> >> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl >> >> Any comments or critiques would be appreciated, >> Christopher >> >> -- >> >> On FC14: >> Requires: httpd >= 2.2.17-1 >> Requires: mod_perl >= 2.0.4-11 >> Requires: mod_ssl >= 1:2.2.17-1 >> Requires: perl >= 5.12.3-141 >> Requires: perl-CGI >= 3.51-1 >> Requires: perl-Digest-SHA1 >= 2.12-4 >> Requires: perl-XML-Filter-BufferText >= 1.01-9 >> Requires: perl-XML-Simple >= 2.18-7 >> Requires: perl-XML-Validator-Schema >= 1.10-5 >> Requires: perl-Tree-DAG_Node >= 1.06-8 >> Requires: perl-LDAP >= 0.40-2 >> Requires: perl-Crypt-SSLeay >= 0.58-1 >> Requires: ffmpeg >= 0.6-4 >> >> On EL5: >> Requires: httpd >= 2.2.3-45 >> Requires: mod_perl >= 2.0.4-6 >> Requires: mod_ssl >= 2.2.3-45 >> Requires: perl >= 5.8.8-32 >> Requires: perl-Digest-SHA1 >= 2.11-1 >> Requires: perl-XML-Filter-BufferText >= 1.01 >> Requires: perl-XML-Simple >= 2.14-4 >> Requires: perl-XML-Validator-Schema >= 1.10-1 >> Requires: perl-Tree-DAG_Node >= 1.06 >> Requires: perl-LDAP >= 0.33-3 >> Requires: perl-Crypt-SSLeay >= 0.51-11 >> Requires: ffmpeg >= 0.6.1-1 >> >
Re: High CPU utilization on RHEL5.6/CentOS5.6
On Mon, Jul 25, 2011 at 4:02 PM, Christopher Stanton wrote: > Neither install is running within VMWare. Might be worthwhile trying a different rate though, or going tickless. Perl 5.8.8 isn't that much (if at all) slower than 5.12. Looks like the FC14 kernel is tickless: http://docs.fedoraproject.org/en-US/Fedora/14/pdf/Power_Management_Guide/Fedora-14-Power_Management_Guide-en-US.pdf > On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer wrote: >> This may not be related, but when I was working with Centos 5 in a >> Vmware environment, I ran into an issue of high cpu since the default >> clock rate with Linux 2.6 is 1000Hz. I changed to a 100Hz clock rate >> and got much lower cpu usage. >> >> I'm not sure if that is your issue, perhaps FC14 was using less than a >> 1000Hz clock rate. You might try a tickless kernel, or dig further >> into what clock rate your kernel is using. >> >> I don't know where the authoritative source for this information is, >> but here's what I found with a search: >> >> http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare >> >> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton >> wrote: >>> I have a MJPEG streaming system which uses mod_perl in the web >>> interface to supply the final stream to the client. >>> >>> I am seeing high cpu utilization under RHEL5.6 which I don't see on >>> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on >>> different hardware, but not that different. And on EL5 the httpd >>> session is way higher (cpu wise) than the supplying proxy. On FC14 on >>> my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which >>> can handle multiplexing etc and 5.5-8% for the mod_perl httpd session. >>> >>> I think this is an issue with that platform, but I haven't seen >>> any/many reports. Does anyone have a sec to look at the source? I >>> don't think I have implemented this is a horrible inefficient way but >>> who knows. It is true, I could have byte markers and then know the >>> size of the JPEG I am expecting rather than just splitting on the >>> boundary field, but in tests (and production) it seemed to work ok. >>> >>> http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl >>> >>> Any comments or critiques would be appreciated, >>> Christopher >>> >>> -- >>> >>> On FC14: >>> Requires: httpd >= 2.2.17-1 >>> Requires: mod_perl >= 2.0.4-11 >>> Requires: mod_ssl >= 1:2.2.17-1 >>> Requires: perl >= 5.12.3-141 >>> Requires: perl-CGI >= 3.51-1 >>> Requires: perl-Digest-SHA1 >= 2.12-4 >>> Requires: perl-XML-Filter-BufferText >= 1.01-9 >>> Requires: perl-XML-Simple >= 2.18-7 >>> Requires: perl-XML-Validator-Schema >= 1.10-5 >>> Requires: perl-Tree-DAG_Node >= 1.06-8 >>> Requires: perl-LDAP >= 0.40-2 >>> Requires: perl-Crypt-SSLeay >= 0.58-1 >>> Requires: ffmpeg >= 0.6-4 >>> >>> On EL5: >>> Requires: httpd >= 2.2.3-45 >>> Requires: mod_perl >= 2.0.4-6 >>> Requires: mod_ssl >= 2.2.3-45 >>> Requires: perl >= 5.8.8-32 >>> Requires: perl-Digest-SHA1 >= 2.11-1 >>> Requires: perl-XML-Filter-BufferText >= 1.01 >>> Requires: perl-XML-Simple >= 2.14-4 >>> Requires: perl-XML-Validator-Schema >= 1.10-1 >>> Requires: perl-Tree-DAG_Node >= 1.06 >>> Requires: perl-LDAP >= 0.33-3 >>> Requires: perl-Crypt-SSLeay >= 0.51-11 >>> Requires: ffmpeg >= 0.6.1-1 >>> >> >
Re: High CPU utilization on RHEL5.6/CentOS5.6
Before I start playing mix-and-match with kernels, I would prefer some feedback on mod_perl and the script's code. It doesn't sleep in the main loop, but it should block on my @ready = $select->can_read(1); and not block on if ( $sock->read(my $data, POSIX::BUFSIZ*10) ){ since the socket is set with Blocking => 0. A JPEG (in a 25fps stream) arrives every 40,000 microseconds. Christopher On Mon, Jul 25, 2011 at 6:43 PM, Fred Moyer wrote: > On Mon, Jul 25, 2011 at 4:02 PM, Christopher Stanton > wrote: >> Neither install is running within VMWare. > > Might be worthwhile trying a different rate though, or going tickless. > Perl 5.8.8 isn't that much (if at all) slower than 5.12. Looks like > the FC14 kernel is tickless: > > http://docs.fedoraproject.org/en-US/Fedora/14/pdf/Power_Management_Guide/Fedora-14-Power_Management_Guide-en-US.pdf > > >> On Mon, Jul 25, 2011 at 5:55 PM, Fred Moyer wrote: >>> This may not be related, but when I was working with Centos 5 in a >>> Vmware environment, I ran into an issue of high cpu since the default >>> clock rate with Linux 2.6 is 1000Hz. I changed to a 100Hz clock rate >>> and got much lower cpu usage. >>> >>> I'm not sure if that is your issue, perhaps FC14 was using less than a >>> 1000Hz clock rate. You might try a tickless kernel, or dig further >>> into what clock rate your kernel is using. >>> >>> I don't know where the authoritative source for this information is, >>> but here's what I found with a search: >>> >>> http://yate.null.ro/pmwiki/index.php?n=Main.YateAndVMWare >>> >>> On Mon, Jul 25, 2011 at 3:43 PM, Christopher Stanton >>> wrote: I have a MJPEG streaming system which uses mod_perl in the web interface to supply the final stream to the client. I am seeing high cpu utilization under RHEL5.6 which I don't see on FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on different hardware, but not that different. And on EL5 the httpd session is way higher (cpu wise) than the supplying proxy. On FC14 on my test rig I see (for a 25fps stream) 15-18% CPU for the proxy which can handle multiplexing etc and 5.5-8% for the mod_perl httpd session. I think this is an issue with that platform, but I haven't seen any/many reports. Does anyone have a sec to look at the source? I don't think I have implemented this is a horrible inefficient way but who knows. It is true, I could have byte markers and then know the size of the JPEG I am expecting rather than just splitting on the boundary field, but in tests (and production) it seemed to work ok. http://svn.codaxus.com/flexTPS/2.x.x/trunk/portal/2.4.x/site/perl/nph-mjpeg_stream.pl Any comments or critiques would be appreciated, Christopher -- On FC14: Requires: httpd >= 2.2.17-1 Requires: mod_perl >= 2.0.4-11 Requires: mod_ssl >= 1:2.2.17-1 Requires: perl >= 5.12.3-141 Requires: perl-CGI >= 3.51-1 Requires: perl-Digest-SHA1 >= 2.12-4 Requires: perl-XML-Filter-BufferText >= 1.01-9 Requires: perl-XML-Simple >= 2.18-7 Requires: perl-XML-Validator-Schema >= 1.10-5 Requires: perl-Tree-DAG_Node >= 1.06-8 Requires: perl-LDAP >= 0.40-2 Requires: perl-Crypt-SSLeay >= 0.58-1 Requires: ffmpeg >= 0.6-4 On EL5: Requires: httpd >= 2.2.3-45 Requires: mod_perl >= 2.0.4-6 Requires: mod_ssl >= 2.2.3-45 Requires: perl >= 5.8.8-32 Requires: perl-Digest-SHA1 >= 2.11-1 Requires: perl-XML-Filter-BufferText >= 1.01 Requires: perl-XML-Simple >= 2.14-4 Requires: perl-XML-Validator-Schema >= 1.10-1 Requires: perl-Tree-DAG_Node >= 1.06 Requires: perl-LDAP >= 0.33-3 Requires: perl-Crypt-SSLeay >= 0.51-11 Requires: ffmpeg >= 0.6.1-1 >>> >> >
Re: High CPU utilization on RHEL5.6/CentOS5.6
On Mon, Jul 25, 2011 at 6:43 PM, Christopher Stanton wrote: > I am seeing high cpu utilization under RHEL5.6 which I don't see on > FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on > different hardware, but not that different. Well, you've changed hardware and lots of software versions, so there are many things it could be. The only one that seems unlikely is your code, since that didn't change. I would suspect the major downgrade in Perl as the culprit. You went from 5.12 to 5.8. To test this, I suggest you install 5.12 on your new system and compile mod_perl against it. If you don't want to touch the system mod_perl and apache you can install your own in a different location. I use perlbrew to try out new Perl versions locally. - Perrin
Re: High CPU utilization on RHEL5.6/CentOS5.6
I will see if I can get Perl 5.12 and an associated mod_perl up and running on the problem machine and see if the CPU utilization goes down. Long long ago in a install far far away I did not run into this issue on FC4 which has very similar packages other than apache (2.0 rather than 2.2): Requires: httpd >= 2.0.54-0; perl >= 5.8.6-0; perl-Digest-SHA1 >= 2.10-0; mod_perl >= 2.0.1-0; On Tue, Jul 26, 2011 at 12:43 PM, Perrin Harkins wrote: > On Mon, Jul 25, 2011 at 6:43 PM, Christopher Stanton > wrote: >> I am seeing high cpu utilization under RHEL5.6 which I don't see on >> FC14. We are talking sub 10% on FC14 vs 80% on EL5. This is on >> different hardware, but not that different. > > Well, you've changed hardware and lots of software versions, so there > are many things it could be. The only one that seems unlikely is your > code, since that didn't change. > > I would suspect the major downgrade in Perl as the culprit. You went > from 5.12 to 5.8. To test this, I suggest you install 5.12 on your > new system and compile mod_perl against it. If you don't want to > touch the system mod_perl and apache you can install your own in a > different location. I use perlbrew to try out new Perl versions > locally. > > - Perrin >
Re: High CPU utilization on RHEL5.6/CentOS5.6
On 26 Jul 2011, at 18:58, Christopher Stanton wrote: > I will see if I can get Perl 5.12 and an associated mod_perl up and > running on the problem machine and see if the CPU utilization goes > down. Is benchmarking out of the question?
Re: High CPU utilization on RHEL5.6/CentOS5.6
Which benchmark module would work best with mod_perl, Perl 5.8 and Perl 5.12? On Wed, Jul 27, 2011 at 7:22 AM, Dave Hodgkinson wrote: > > On 26 Jul 2011, at 18:58, Christopher Stanton wrote: > >> I will see if I can get Perl 5.12 and an associated mod_perl up and >> running on the problem machine and see if the CPU utilization goes >> down. > > Is benchmarking out of the question? > >
Re: High CPU utilization on RHEL5.6/CentOS5.6
On Wed, Jul 27, 2011 at 8:36 AM, Christopher Stanton wrote: > Which benchmark module would work best with mod_perl, Perl 5.8 and Perl 5.12? For web apps, I usually write my own test with LWP (or HTTP::Async if you want to simulate a lot of users). To see where things are slow, use Devel::NYTProf. - Perrin
Re: High CPU utilization on RHEL5.6/CentOS5.6
Wait. Is this the one with the horrible bug that knackers OO? On 25 Jul 2011, at 23:43, Christopher Stanton wrote: > Requires: perl >= 5.8.8-32