Re: Linux Hello World Benchmarks - 11/19/2001
Perrin Harkins wrote: > > on 11/19/01 8:05 PM, Joshua Chamas at [EMAIL PROTECTED] wrote: > > It has been a while, but here's a new set of Hello World benchmarks! > > There was a recent announcement of HTML::Template::JIT, and Template Toolkit > has an XS option now. Any chance you could put those into the next round? > - Perrin OK, I upgraded Template Toolkit to version 2.06 with XS option on for all templates, and performance is much improved!! However, HTML::Template::JIT will likely not work on my system any time soon, because it was written for perl 5.6, and I only run 5.00503 currently. Someone else can submit a test & config specifically for HTML::Template::JIT, I'd suggest the h2000 type where speed matters, and I'll add it to the test suite even it I can't run the benchmark myself. [hello]# ./bench.pl -type=2000 -ram -time=60 Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB) - - - - - - - - Apache::ASP 2000h2000.asp 209.4 12566 60.020.004776 28998 33200 Apache::Registry 2000 mod_perl API h2000.reg 327.0 19622 60.020.003059 28179 15300 HTML::Embperl 2000 h2000.epl 107.5 6465 60.120.009299 28841 20048 HTML::Mason 2000h2000.mas 80.8 4849 60.000.012374 28799 110292 HTML::Template 2000 h2000.htm 95.4 5724 60.010.010484 29152 49932 mod_php PHP 2000h2000.php 247.6 14862 60.020.004038 28866 10384 Template Toolkit 2000 h2000.tt 125.5 7533 60.020.007967 28889 55748 -- Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks Founder Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: Linux Hello World Benchmarks - 11/19/2001
on 11/19/01 8:05 PM, Joshua Chamas at [EMAIL PROTECTED] wrote: > It has been a while, but here's a new set of Hello World benchmarks! There was a recent announcement of HTML::Template::JIT, and Template Toolkit has an XS option now. Any chance you could put those into the next round? - Perrin
Linux Hello World Benchmarks - 11/19/2001
Hey, [[ NUMBERS ARE BELOW ]] It has been a while, but here's a new set of Hello World benchmarks! What took me so long in getting these out is that the java web environments that I had set up would keep crashing during the tests in ways that would not only render their benchmarks meaningless, but also skewed the results for other tests, say when a java thread would spin out of control! You can download the latest benchmark suite to run on your system at: http://www.chamas.com/bench/hello.tar.gz It seems to run on linux & solaris just fine, though the tests that are run depend on what environments you have set up. Please see the distribution README on how to run the suite for yourself. --Josh METHODOLOGY: These latest numbers come from my running the pubbench.sh script: ]# cat pubbench.sh #!/bin/bash ./bench.pl -version --ram --init-exec="perl javainit.pl -restart" -time=180 --store=results perl ./javainit.pl -stop The javainit.pl will kill any of my existing java environments, and if the current test needs java will start them up. The test run for 180 seconds total in 60 second batches, and for each 60 second run the javainit.pl script will be executed. The --ram flag attempts to calculate the RAM used by each test on my linux system during each 60 second test with the results averaged across each, so for applications with memory leaks, the RAM used only shows 60 seconds worth of them. The RAM calculation looks at RAM used & SWAP used from the linux /proc/meminfo file... as it looks at RAM before the httpd is fired up, but after the init-exec script executes, the RAM does not take into account the java web server environments being started, just memory usage that occurs after they start being used. DISCLAIMER: These numbers are just that, numbers. Please don't let them upset you. If you have a positive contribution that you would like to make, you may download the source code and submit patches ( preferably as diff -u ) to the test suite. This benchmark suite has evolved over years of benchmarking, and takes the point of view that web environments can only be well compared when run on the exact same system configuration... this is why the suite is done is such a way that you can run it on _your_ system giving you results relevant to you! NUMBERS: And without further adieu ... ( see NOTES below ) Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB) - - - - - - - - Apache::ASP v2.29 2000 h2000.asp 216.8 39042180.090.004613 28998 33632 Apache::Registry v2.01 2000 mod_per h2000.reg 341.3 61442180.040.002930 28179 16081 HTML::Embperl v1.3.0 2000 h2000.epl 113.2 20391180.120.008833 28841 20418 HTML::Mason v1.03 2000 h2000.mas 84.0 15115180.030.011911 28799 112220 HTML::Template v2.4 2000h2000.htm 99.1 17848180.030.010087 29152 50292 mod_caucho JSP 2000 h2000.jsp 97.6 17617180.540.010248 28965 14057 mod_jserv JSP 2000 h2000.jsp 148.7 26773180.040.006725 29408 29613 mod_php PHP 2000h2000.php 258.2 46477180.020.003873 28866 10212 Template v2.04 Toolkit 2000 h2000.tt53.3 9604180.030.018745 28889 55629 Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit Mem(KB) - - - - - - - - Apache::ASP v2.29 hello.asp 349.8 62985180.050.002859 242 29601 Apache::Dispatch v0.09 handler hello/wor 585.8105443180.010.001707 197 9205 Apache::ePerl hello.epe 347.1 62508180.080.002881 218 17477 Apache::Registry v2.01 CGI Raw hello.cgi 677.7122003180.030.001476 5213061 Apache::Registry v2.01 CGI.pm hello.cgi 449.7 80967180.050.002224 217 20717 Apache::SSI v2.16 hello.sht 546.7 98426180.030.001829 200 13061 CGI::SpeedyCGI CGI Raw hello.scg 159.6 28722180.000.006267 197 9748 CGI::SpeedyCGI CGI.pm hello.scg 145.1 26117180.010.006892 217 10648 HTML static hello.htm 879.215170.610.001137 312 1982 HTML::Embperl v1.3.0hello.epl 463.6 83471180.050.002157 221 17333 HTML::Mason v1.03 hello.mas 368.5 66348180.040.002714 198 29009 HTML::Template v2.4
Re: [bordering on OT] Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
On Thu, Jan 04, 2001 at 09:55:39AM -0500, Blue Lang wrote: > Eh, ab isn't really made as anything other than the most coarsely-grained > of benchmarks. Concurrency testing is useless because it will measure the > ratio of requests/second/processor, not the scalability of requests from > single to multiple processors. Yeah, I agree 'ab' is a pretty coarse benchmark. However, it does in a way measure how much the various processors are helping, because running ab with -c 1 should pretty much ensure that apache only uses one processor at a time (except for a slight overlap while one process does the logging and another could be reading the next request from another processor), and similarily -c 2 should let apache use 2 processors at one given time. All approximately, of course. Anyway, on that 4way server it works that way; the requests per second increase quickly with the concurrency up to 4, but don't increase anymore after that. That is serving relatively slow dynamic pages; with static content I'd expect more rapidly diminishing returns. -- Roger Espel Llima, [EMAIL PROTECTED] http://www.iagora.com/~espel/index.html
[bordering on OT] Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
On Thu, 4 Jan 2001, Roger Espel Llima wrote: > JR Mayberry <[EMAIL PROTECTED]> wrote: > > Linux does serious injustice to mod_perl. Anyone who uses Linux knows > > how horrible it is on SMP, I think some tests showed it uses as litle as > > 25% of the second processor.. > > A simple benchmark with 'ab' shows the number of requests per second > almost double when the concurrency is increased from 1 to 2. With a > concurrency of 4, the number of requests per second increases to > about 3.2 times the original, which is not bad at all considering > that these are dynamic requests with DB queries. Eh, ab isn't really made as anything other than the most coarsely-grained of benchmarks. Concurrency testing is useless because it will measure the ratio of requests/second/processor, not the scalability of requests from single to multiple processors. IOW, you would see almost exactly that same increase in req/second on a single processor, most likely, unless you have a really slow machine. You'd have to tune your load to give you one req/second/processor and then go from there for it to mean anything. Of course the original poster's statement on linux using only 25% of a second CPU is a fuddy and false generalization, but that's a different story. :P -- Blue Lang, Unix Voodoo Priest 202 Ashe Ave, Apt 3, Raleigh, NC. 919 835 1540 "I was born in a city of sharks and sailors!" - June of 44
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
JR Mayberry <[EMAIL PROTECTED]> wrote: > The Modperl handler benchmark, which was done on a dual P3 500mhz on > Linux does serious injustice to mod_perl. Anyone who uses Linux knows > how horrible it is on SMP, I think some tests showed it uses as litle as > 25% of the second processor.. It's an old post, but I simply cannot let this one pass uncommented. I run a busy Apache/mod_perl server on a 4-way SMP Linux box (kernel 2.2.13 from VA Linux), and it sure seems to be using all CPUs quite effectively. A simple benchmark with 'ab' shows the number of requests per second almost double when the concurrency is increased from 1 to 2. With a concurrency of 4, the number of requests per second increases to about 3.2 times the original, which is not bad at all considering that these are dynamic requests with DB queries. Anyway, I wouldn't expect the OS's SMP to be the limiting factor on Apache's dynamic page performance. Apache uses multiple processes, and dynamic page generation is generally CPU bound, not I/O bound. -- Roger Espel Llima, [EMAIL PROTECTED] http://www.iagora.com/~espel/index.html
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
[EMAIL PROTECTED] wrote: > > On Mon, Dec 18, 2000 at 10:37:16AM -0800, Joshua Chamas wrote: > > > > Please feel free to run the tests yourself, and if you give > > me the results, I'll be sure to post them at a later date > > at http://www.chamas.com/bench/ . You can grab the benchmarks > > from http://www.chamas.com/bench/hello.tar.gz > > > > I downloaded the tar ball just to fool around on my system. It will > be nice to have a README file with a short instruction set. > > Do you compile apache for each test or do you just use > one size fits all apache with everything compiled in? One size fits all is the idea, but only the relevant parts of that benchmark will be loaded into a custom httpd.conf startup file for apache. The benchmarks get run at the same time on the same apache, so any overhead due to modules registering themselves into apache for handler stages will be passed to all the tests. Yes, I'll provide a README, and --help/-h help command line docs in the next release. -- Josh
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
On Mon, Dec 18, 2000 at 10:37:16AM -0800, Joshua Chamas wrote: > > Please feel free to run the tests yourself, and if you give > me the results, I'll be sure to post them at a later date > at http://www.chamas.com/bench/ . You can grab the benchmarks > from http://www.chamas.com/bench/hello.tar.gz > I downloaded the tar ball just to fool around on my system. It will be nice to have a README file with a short instruction set. Do you compile apache for each test or do you just use one size fits all apache with everything compiled in?
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
JR Mayberry wrote: > > I strongly dislike the benchmarks on the below URL, its very > misleading.. > > The Modperl handler benchmark, which was done on a dual P3 500mhz on > Linux does serious injustice to mod_perl. Anyone who uses Linux knows > how horrible it is on SMP, I think some tests showed it uses as litle as > 25% of the second processor.. > Please feel free to run the tests yourself, and if you give me the results, I'll be sure to post them at a later date at http://www.chamas.com/bench/ . You can grab the benchmarks from http://www.chamas.com/bench/hello.tar.gz Its only with data from multiple platforms that people will get a comfortable look at the environments cross platform exactly as you say. > I don't think the benchmarks can be taken for anything, you have > benchmarks running on different Architectures other then PC based... 300 > mhz on Sun is totally different then 300 mhz on PC based products > and the real number that drives this benchmark is HITS/S/MHZ.. > Yes, I've done these 300Mhz tests before on Solaris & WinNT and the results are at http://www.chamas.com/bench/ ... got to http://www.chamas.com/bench/hello_bysystem.html for anything useful. > I think for any real test that means anything, there needs to be some > controlled factors, ie: the hardware and operating system.. > The new thing this round of testing is a portable test suite that anyone can run and control their factors as they please. --Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks >> free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
I strongly dislike the benchmarks on the below URL, its very misleading.. The Modperl handler benchmark, which was done on a dual P3 500mhz on Linux does serious injustice to mod_perl. Anyone who uses Linux knows how horrible it is on SMP, I think some tests showed it uses as litle as 25% of the second processor.. I don't think the benchmarks can be taken for anything, you have benchmarks running on different Architectures other then PC based... 300 mhz on Sun is totally different then 300 mhz on PC based products and the real number that drives this benchmark is HITS/S/MHZ.. I think for any real test that means anything, there needs to be some controlled factors, ie: the hardware and operating system.. Gunther Birznieks wrote: > > For the raw benchmarks... > > OK, I finally got a little time to download and read some the hello.tar.gz. > > It's good to see TT is fairly fast. But it's a shame that the only way to > get faster than PHP is to write a raw Mod_perl handler according to the > benchmarks. All the other mod_perl tools seem slower. > > JSP seems to also blow away mod_perl and PHP (except being almost > equivalent to mod_perl handler speed). I assume Resin is precompiling JSP > to Java classes and that maybe the JRE you are using does some very good > hotspot on-the-fly machine-code compiling type technology? > > How does this benchmark stuff compare to the tests run at > > http://www.chamas.com/bench/ > > I notice that JSPs take quite a beating there but are running on a lower > end machine on that set of tests. I presume the below tests are intended to > replace the tests run on these various disparate machines. > > You also seem to have taken out tests? So you are no longer testing > servlets only? It would be interesting to see if Servlet -> JSP dispatching > (with is the recommended model of coding Java Servlets/JSPs these days) > results in any slow down. > > At 02:15 PM 12/16/2000 -0800, Joshua Chamas wrote: > >Hey, > > > >Still very rough, the hello world benchmark suite is available > >for download at: http://www.chamas.com/bench/hello.tar.gz > >You may run it like: > > > > # to get started, see what tests will run, note you > > # may need some CPAN modules installed to get this far > > perl ./bench.pl -test > > > > # to run tests for 1 minute ... shut down your programs > > # and walk away for best results. > > perl ./bench.pl -time=60 > > > >Here are my latest results, having added Resin/caucho/JSP > >with a J2RE 1.3.0 IBM java engine, which other benchmarks > >say is the fastest java on linux overall, & from previous > >testing resin seems the fastest JSP. > > > >I changed the SSI tests to look more like the others, which > >also sped them considerably. Finally, I added tests for PHP, > >mine is 4.0.3, & ePerl. > > > >Test Name Test File Hits/sec Total Hits Total Time > >sec/Hits Bytes/Hit > > -- -- -- -- > >-- -- > >Apache::ASP hello.asp 414.3 24857 hits 60.00 > >sec 0.002414 179 bytes > >Apache::Dispatch handler hello/worl 689.5 41375 hits 60.01 > >sec 0.001450 134 bytes > >Apache::Registry CGI Raw hello_raw. 725.2 43514 hits 60.00 > >sec 0.001379 52 bytes > >Apache::Registry CGI.pm hello.reg 491.5 29492 hits 60.00 > >sec 0.002035 154 bytes > >Apache::SSI hello.shtm 584.6 35080 hits 60.01 > >sec 0.001711 137 bytes > >Apache::ePerl hello.eper 359.8 21588 hits 60.00 > >sec 0.002780 155 bytes > >HTML static hello.html 1195.2 5 hits 41.83 > >sec 0.000837 249 bytes > >HTML::Embperl hello.epl 510.8 30647 hits 60.00 > >sec 0.001958 158 bytes > >HTML::Mason hello.mas 383.8 23030 hits 60.00 > >sec 0.002605 134 bytes > >Template Toolkit hello.tt553.6 33221 hits 60.01 > >sec 0.001806 136 bytes > >mod_caucho JSPhello.jsp 859.9 5 hits 58.15 > >sec 0.001163 156 bytes > >mod_include SSI hello.shtm 1008.0 5 hits 49.60 > >sec 0.000992 136 bytes > >mod_perl handler hello.benc 886.3 5 hits 56.42 > >sec 0.001128 134 bytes > >mod_php PHP hello.php 750.8 45050 hits 60.00 > >sec 0.001332 163 bytes > > > >As has been noted, my static html is probably slower than yours > >relatively. I have a dual CPU system & have most apache modules > >enabled by default, thus creating huge headers for static html. > > > >I think the dual CPU nature of my system means my system will > >spend more time waiting on SMP & network locking as the request > >rate gets faster, but I don't know much about these things, so if > >there is something to be gained here, please feel free to clarify > >how this might impact the results. > > > >--Josh > > > >_ > >Joshua Chamas
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
Gunther Birznieks wrote: > But it's a shame that the only way to > get faster than PHP is to write a raw Mod_perl handler according to the > benchmarks. All the other mod_perl tools seem slower. It makes sense though. All the other tools do more setup work on each request: parsing input, manipulating it into objects, etc. > JSP seems to also blow away mod_perl and PHP (except being almost > equivalent to mod_perl handler speed). Hmm... To me that means it's equivalent, not blowing away mod_perl. > I assume Resin is precompiling JSP > to Java classes and that maybe the JRE you are using does some very good > hotspot on-the-fly machine-code compiling type technology? The IBM JDK has a JIT, but no Hotspot equivalent. They have the fastest JVM on x86 though. Check out the scores here: http://www.volano.com/report.html One thing I discovered when testing Caucho's JSP last year is that they do a really good job of generating efficient servlets from JSP. I was not able to write a faster "hello world" servlet than the one they auto-generated from a JSP page. By the way, as a cautionary tale on "hello world" tests: These tests appear to show mod_include handling SSI pages twice as fast as Apache::SSI. However, in tests we've done with real production pages (30-60K, half dozen or so includes per page) Apache::SSI was actually a little bit faster. Of course this was about a year ago so the tables may have turned since then. - Perrin
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
Hi all, On Sun, 17 Dec 2000, Gerald Richter wrote: > there are so many factors, so they are very difficult to compare. True. But nevertheless I think it's a very useful bit of work because the thing that stands out is that all (server) dynamic content comes at a high cost in processor cycles. This won't come as a surprise to most of you. Be aware too that even the fastest of these benchmarks is not an outstandingly fast setup. The lesson is obvious: if you're looking for high throughput, think long and hard about your system architecture, in the widest possible sense, and keep server-generated dynamic stuff to a minimum. Having said that, most of us won't be pushing the limits most of the time (and if we are we're probably secretly quite pleased about it:). 73, Ged.
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
> For the raw benchmarks... > > OK, I finally got a little time to download and read some the hello.tar.gz. > > It's good to see TT is fairly fast. But it's a shame that the only way to > get faster than PHP is to write a raw Mod_perl handler according to the > benchmarks. All the other mod_perl tools seem slower. > Everybody looking at this tests should be aware that it only test the startup/setup overhead for a "null" request. The Hello World actualy does nothing, so it highly depends on what features are initialized at the start of the request, it even makes a difference if different systems sends a different number of response http headers. For this reason this benchmarks are _not_ a comparision of the speed of the different systems which would manifest in real life. To get some usefull numbers there must be tests for very different situations, at least with some dynamic content in it (Joshua is working on that), another one with database access and for example some session management to have persistent data. There are more factors, for example the possibility of some systems to cache the generated output, the possibility to build your page out of many components... But even if somebody will generate such a huge benchmark suite there are so many factors, so they are very difficult to compare. Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
Gunther Birznieks wrote: > > For the raw benchmarks... > > OK, I finally got a little time to download and read some the hello.tar.gz. > > It's good to see TT is fairly fast. But it's a shame that the only way to > get faster than PHP is to write a raw Mod_perl handler according to the > benchmarks. All the other mod_perl tools seem slower. > I'm preparing a Hello World 2000 test which will test the runtime properties of an environment. It will be relevant to the dynamic template environments like ASP, Mason, Embperl, TT, JSP, PHP... early results show the oft stated perl is faster than the rest to be true. More later for RFC. > JSP seems to also blow away mod_perl and PHP (except being almost > equivalent to mod_perl handler speed). I assume Resin is precompiling JSP > to Java classes and that maybe the JRE you are using does some very good > hotspot on-the-fly machine-code compiling type technology? > The IBM JDK I believe does hotspot... blow away? The diff I see between mod_perl & mod_caucho JSP is .35 sec/hit of my system time. Little differences seem to make a bigger impact when things are running this fast. Note, resin does a lot of work to compile a JSP, really poor compile time, but you won't see that in this test. > How does this benchmark stuff compare to the tests run at > http://www.chamas.com/bench/ Almost identical, except the SSI test is changed to be more fair/similar to the rest. Note that I didn't run all those benchmarks which is why the numbers are much different. To see the old results grouped by the system/who did them, check out: http://www.chamas.com/bench/hello_bysystem.html > I notice that JSPs take quite a beating there but are running on a lower > end machine on that set of tests. I presume the below tests are intended to > replace the tests run on these various disparate machines. > I don't know what to do with those old benchmarks. I might replace them with results that come in from people running the bench.pl program. But it takes a lot of time to set things up, so I don't know if I'll just lose all of it. Its a lot of data to display in a relevant way. > You also seem to have taken out tests? So you are no longer testing > servlets only? It would be interesting to see if Servlet -> JSP dispatching > (with is the recommended model of coding Java Servlets/JSPs these days) > results in any slow down. > I never tested servlets before, that was someone else. I could set up a servlet test too, but I'm a java newbie, but I do think the resin mod_caucho stuff does it too, so it should be shortly forthcoming. -- Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks >> free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: Linux Hello World Benchmarks: +PHP,JSP,ePerl
For the raw benchmarks... OK, I finally got a little time to download and read some the hello.tar.gz. It's good to see TT is fairly fast. But it's a shame that the only way to get faster than PHP is to write a raw Mod_perl handler according to the benchmarks. All the other mod_perl tools seem slower. JSP seems to also blow away mod_perl and PHP (except being almost equivalent to mod_perl handler speed). I assume Resin is precompiling JSP to Java classes and that maybe the JRE you are using does some very good hotspot on-the-fly machine-code compiling type technology? How does this benchmark stuff compare to the tests run at http://www.chamas.com/bench/ I notice that JSPs take quite a beating there but are running on a lower end machine on that set of tests. I presume the below tests are intended to replace the tests run on these various disparate machines. You also seem to have taken out tests? So you are no longer testing servlets only? It would be interesting to see if Servlet -> JSP dispatching (with is the recommended model of coding Java Servlets/JSPs these days) results in any slow down. At 02:15 PM 12/16/2000 -0800, Joshua Chamas wrote: >Hey, > >Still very rough, the hello world benchmark suite is available >for download at: http://www.chamas.com/bench/hello.tar.gz >You may run it like: > > # to get started, see what tests will run, note you > # may need some CPAN modules installed to get this far > perl ./bench.pl -test > > # to run tests for 1 minute ... shut down your programs > # and walk away for best results. > perl ./bench.pl -time=60 > >Here are my latest results, having added Resin/caucho/JSP >with a J2RE 1.3.0 IBM java engine, which other benchmarks >say is the fastest java on linux overall, & from previous >testing resin seems the fastest JSP. > >I changed the SSI tests to look more like the others, which >also sped them considerably. Finally, I added tests for PHP, >mine is 4.0.3, & ePerl. > >Test Name Test File Hits/sec Total Hits Total Time >sec/Hits Bytes/Hit > -- -- -- -- >-- -- >Apache::ASP hello.asp 414.3 24857 hits 60.00 >sec 0.002414 179 bytes >Apache::Dispatch handler hello/worl 689.5 41375 hits 60.01 >sec 0.001450 134 bytes >Apache::Registry CGI Raw hello_raw. 725.2 43514 hits 60.00 >sec 0.001379 52 bytes >Apache::Registry CGI.pm hello.reg 491.5 29492 hits 60.00 >sec 0.002035 154 bytes >Apache::SSI hello.shtm 584.6 35080 hits 60.01 >sec 0.001711 137 bytes >Apache::ePerl hello.eper 359.8 21588 hits 60.00 >sec 0.002780 155 bytes >HTML static hello.html 1195.2 5 hits 41.83 >sec 0.000837 249 bytes >HTML::Embperl hello.epl 510.8 30647 hits 60.00 >sec 0.001958 158 bytes >HTML::Mason hello.mas 383.8 23030 hits 60.00 >sec 0.002605 134 bytes >Template Toolkit hello.tt553.6 33221 hits 60.01 >sec 0.001806 136 bytes >mod_caucho JSPhello.jsp 859.9 5 hits 58.15 >sec 0.001163 156 bytes >mod_include SSI hello.shtm 1008.0 5 hits 49.60 >sec 0.000992 136 bytes >mod_perl handler hello.benc 886.3 5 hits 56.42 >sec 0.001128 134 bytes >mod_php PHP hello.php 750.8 45050 hits 60.00 >sec 0.001332 163 bytes > >As has been noted, my static html is probably slower than yours >relatively. I have a dual CPU system & have most apache modules >enabled by default, thus creating huge headers for static html. > >I think the dual CPU nature of my system means my system will >spend more time waiting on SMP & network locking as the request >rate gets faster, but I don't know much about these things, so if >there is something to be gained here, please feel free to clarify >how this might impact the results. > >--Josh > >_ >Joshua Chamas Chamas Enterprises Inc. >NodeWorks >> free web link monitoring Huntington Beach, CA USA >http://www.nodeworks.com1-714-625-4051 __ Gunther Birznieks ([EMAIL PROTECTED]) eXtropia - The Web Technology Company http://www.extropia.com/
Linux Hello World Benchmarks: +PHP,JSP,ePerl
Hey, Still very rough, the hello world benchmark suite is available for download at: http://www.chamas.com/bench/hello.tar.gz You may run it like: # to get started, see what tests will run, note you # may need some CPAN modules installed to get this far perl ./bench.pl -test # to run tests for 1 minute ... shut down your programs # and walk away for best results. perl ./bench.pl -time=60 Here are my latest results, having added Resin/caucho/JSP with a J2RE 1.3.0 IBM java engine, which other benchmarks say is the fastest java on linux overall, & from previous testing resin seems the fastest JSP. I changed the SSI tests to look more like the others, which also sped them considerably. Finally, I added tests for PHP, mine is 4.0.3, & ePerl. Test Name Test File Hits/sec Total Hits Total Time sec/Hits Bytes/Hit -- -- -- -- -- -- Apache::ASP hello.asp 414.3 24857 hits 60.00 sec 0.002414 179 bytes Apache::Dispatch handler hello/worl 689.5 41375 hits 60.01 sec 0.001450 134 bytes Apache::Registry CGI Raw hello_raw. 725.2 43514 hits 60.00 sec 0.001379 52 bytes Apache::Registry CGI.pm hello.reg 491.5 29492 hits 60.00 sec 0.002035 154 bytes Apache::SSI hello.shtm 584.6 35080 hits 60.01 sec 0.001711 137 bytes Apache::ePerl hello.eper 359.8 21588 hits 60.00 sec 0.002780 155 bytes HTML static hello.html 1195.2 5 hits 41.83 sec 0.000837 249 bytes HTML::Embperl hello.epl 510.8 30647 hits 60.00 sec 0.001958 158 bytes HTML::Mason hello.mas 383.8 23030 hits 60.00 sec 0.002605 134 bytes Template Toolkit hello.tt553.6 33221 hits 60.01 sec 0.001806 136 bytes mod_caucho JSPhello.jsp 859.9 5 hits 58.15 sec 0.001163 156 bytes mod_include SSI hello.shtm 1008.0 5 hits 49.60 sec 0.000992 136 bytes mod_perl handler hello.benc 886.3 5 hits 56.42 sec 0.001128 134 bytes mod_php PHP hello.php 750.8 45050 hits 60.00 sec 0.001332 163 bytes As has been noted, my static html is probably slower than yours relatively. I have a dual CPU system & have most apache modules enabled by default, thus creating huge headers for static html. I think the dual CPU nature of my system means my system will spend more time waiting on SMP & network locking as the request rate gets faster, but I don't know much about these things, so if there is something to be gained here, please feel free to clarify how this might impact the results. --Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks >> free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: Linux Hello World Benchmarks...
Joshua Chamas <[EMAIL PROTECTED]> writes: > Joe Schaefer wrote: > > > > IME, simple mod_perl handlers typically run around 50% as fast as > > HTML static pages. Your hello world benchmark seems to be slightly > > misleading in this respect, since the content-length is small > > relative to the header size. > > > > I'll send you my benchmark suite separately so you can > submit your results for http://www.chamas.com/bench. I have > never seen modperl handler faster than static HTML. Me neither - what I said was that modperl handlers are about twice as slow as static pages. That's what I meant, but I guess didn't make it clear. Best. -- Joe Schaefer
Re: Linux Hello World Benchmarks...
Joe Schaefer wrote: > > IME, simple mod_perl handlers typically run around 50% as fast as > HTML static pages. Your hello world benchmark seems to be slightly > misleading in this respect, since the content-length is small > relative to the header size. > I'll send you my benchmark suite separately so you can submit your results for http://www.chamas.com/bench. I have never seen modperl handler faster than static HTML. > For your HTML Static, I would guess that the headers delivered are > significantly larger than the ones returned by your modperl handler. > Hence for 5 hits, there is a significant discrepancy in the > total bytes delivered. This skews the hits/sec numbers in the > favor of content handlers that deliver shorter headers. > Yes, each environment delivers different headers, but that's just what they do. The Hello World test is to show the overhead of running in a particular environment. Generally people only add code from there, and not optimize HelloWorld further. -- Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks >> free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: Linux Hello World Benchmarks...
Joshua Chamas <[EMAIL PROTECTED]> writes: > RESULTS: > > [hello]# ./bench.pl -time=60 > ... > Test Name Test FileHits/sec Total Hits Total Time Total >Bytes > > > HTML Static hello.html 1211.6 5 hits 41.27 sec >12450249 byt > ModPerl Handler hello.bench 888.9 5 hits 56.25 sec >6700268 byte [...] IME, simple mod_perl handlers typically run around 50% as fast as HTML static pages. Your hello world benchmark seems to be slightly misleading in this respect, since the content-length is small relative to the header size. For your HTML Static, I would guess that the headers delivered are significantly larger than the ones returned by your modperl handler. Hence for 5 hits, there is a significant discrepancy in the total bytes delivered. This skews the hits/sec numbers in the favor of content handlers that deliver shorter headers. To use HTML Static as a baseline comparison for dynamic content handlers, I think you should ensure that the headers delivered are comparable to those that are dynamically generated. Best. -- Joe Schaefer
Re: Linux Hello World Benchmarks...
"Alexander Farber (EED)" wrote: > > Hi Joshua, > > you sort the table at http://www.chamas.com/bench/ by Hits/s, > but the ModPerl Handler was tested on PIII-500 x 2 and the Java > thingies below - only PII-266. > > Is it an intended joke or do I misunderstand something? > The first page is marked "for fun" because there are no relevant sortings that you can do between systems & benchmarkers, as there are too many variables. Its sorted by speed to encourage people to submit new results with their shiny new GHz CPUs. You can try to derive some real meaning from tests done on the same system by the same person using the same methodology ... for these go to http://www.chamas.com/bench/hello_bysystem.html --Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks >> free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: Linux Hello World Benchmarks...
Hi Joshua, Joshua Chamas wrote: > Note, this is the first benchmark that I've run of Apache::ASP on > Linux, which is nice to see because Linux is one of the faster OS's, > and it now looks bit more of a player, compared to what's listed at > http://www.chamas.com/bench/ when I benched it on Solaris x86. you sort the table at http://www.chamas.com/bench/ by Hits/s, but the ModPerl Handler was tested on PIII-500 x 2 and the Java thingies below - only PII-266. Is it an intended joke or do I misunderstand something? Regards Alex
Re: Linux Hello World Benchmarks...
Gunther Birznieks wrote: > > Then it seems odd that there is such a huge discrepency between CGI.pm and > no CGI.pm. If you preload CGI.pm in startup.pl does the difference go away? > I did preload CGI.pm. I'll send you the hello world suite separately since you seem curious. Note that at 500 requests / second, the difference of 50 requests / sec is really 1/500 - 1/550 = .00018 CPU seconds. The point is that as these systems get more efficient, minor differences in performance seem to make more of a difference. The same .0018 CPU sec at 100 hits/sec would only make 102 hits / sec. --Josh
Re: Linux Hello World Benchmarks...
Then it seems odd that there is such a huge discrepency between CGI.pm and no CGI.pm. If you preload CGI.pm in startup.pl does the difference go away? At 02:56 AM 12/11/2000 -0800, Joshua Chamas wrote: >Gunther Birznieks wrote: > > > > Is CGI Raw decoding the get/post yourself? Or using the Apache::args, > > Apache::Request::param mechanism? > > > >In the hello world scripts, there is no get/post processing as >part of the benchmark. Here's the code that's run: > > http://www.chamas.com/bench/#perlrawcgi > >The general language scripts are at http://www.chamas.com/bench/ >and are also in the hello.tar.gz bundle that I could send. >The bundle also has the app configs in bench.pl. Its a whole 8K >gzipped though, and I'm afraid Ask would hurt me if I posted it ;) > >-- Josh > >_ >Joshua Chamas Chamas Enterprises Inc. >NodeWorks >> free web link monitoring Huntington Beach, CA USA >http://www.nodeworks.com1-714-625-4051 __ Gunther Birznieks ([EMAIL PROTECTED]) eXtropia - The Web Technology Company http://www.extropia.com/
Re: Linux Hello World Benchmarks...
On Mon, 11 Dec 2000, Joshua Chamas wrote: > Lastly, I was unable to get AxKit to run without segfaulting ... http://axkit.org/faq.xml Either you're running PHP on that server, or you have an Apache with expat included. Do "nm /path/to/apache/bin/httpd | grep -i XML" to find out if the latter is true. -- /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: Linux Hello World Benchmarks...
Gunther Birznieks wrote: > > Is CGI Raw decoding the get/post yourself? Or using the Apache::args, > Apache::Request::param mechanism? > In the hello world scripts, there is no get/post processing as part of the benchmark. Here's the code that's run: http://www.chamas.com/bench/#perlrawcgi The general language scripts are at http://www.chamas.com/bench/ and are also in the hello.tar.gz bundle that I could send. The bundle also has the app configs in bench.pl. Its a whole 8K gzipped though, and I'm afraid Ask would hurt me if I posted it ;) -- Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks >> free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: Linux Hello World Benchmarks...
Is CGI Raw decoding the get/post yourself? Or using the Apache::args, Apache::Request::param mechanism? At 02:13 AM 12/11/2000 -0800, Joshua Chamas wrote: >Hey, > >I have automated a portable Hello World test suite, but its not >CPAN ready, so if any would like to contribute, run, and comment >on the sources, give me a holler & I'll send them to you. What >it does is fire up a lean apache on a high port with only the >config necessary to run the benchmark.
Linux Hello World Benchmarks...
Hey,
I have automated a portable Hello World test suite, but its not
CPAN ready, so if any would like to contribute, run, and comment
on the sources, give me a holler & I'll send them to you. What
it does is fire up a lean apache on a high port with only the
config necessary to run the benchmark.
RESULTS:
[hello]# ./bench.pl -time=60
...
Test Name Test FileHits/sec Total Hits Total Time Total
Bytes
Apache::ASP hello.asp 416.9 25016 hits 60.00 sec4478401
byte
Apache::Registry CGI Raw hello_raw.re 736.9 44214 hits 60.00 sec2299284
byte
Apache::Registry CGI.pm hello.reg 501.3 30080 hits 60.00 sec4632628
byte
HTML Static hello.html 1211.6 5 hits 41.27 sec12450249
byt
HTML::Embperl hello.epl 510.7 30647 hits 60.01 sec4843016
byte
HTML::Mason hello.mas 382.2 22934 hits 60.00 sec3086640
byte
ModPerl Handler hello.bench 888.9 5 hits 56.25 sec6700268
byte
Template Toolkit hello.tt 181.6 10895 hits 60.01 sec1481992
byte
**AxKit --> need someone to run benchmark suite with this working
Apache::ASP v2.07
HTML::Embperl 1.3.0
HTML::Mason 0.89
Template Toolkit 2.0
mod_perl 1.24
apache 1.3.14
linux 2.2.14
PIII 450x2
localhost interface
Note, this is the first benchmark that I've run of Apache::ASP on
Linux, which is nice to see because Linux is one of the faster OS's,
and it now looks bit more of a player, compared to what's listed at
http://www.chamas.com/bench/ when I benched it on Solaris x86.
Also, Mason seems to have improved significantly since I last
benched it. Good job Jonathan.
Lastly, I was unable to get AxKit to run without segfaulting ...
I had wanted to get at least a XPathScript benchmark, and maybe
Sablotron driven XSLT. The simple XSLT hello world is at
http://www.chamas.com/bench/#xml. The XPathScript HelloWorld
that I came up with was: <% print "Hello "; %>World ... but
maybe it should have been more like the XSLT hello world?
-- Josh
_
Joshua Chamas Chamas Enterprises Inc.
NodeWorks >> free web link monitoring Huntington Beach, CA USA
http://www.nodeworks.com1-714-625-4051
