Re: [ANNOUNCE] Hello World Benchmarks, updated
Joshua Chamas wrote: > > mod_caucho > used to look a lot faster, but my testing methodology changed. > I used to take the results of the second benchmark run, and > publish those, but this time only ran the -test for minor > caching after starting resin ( & tomcat ). So, I'm guessing > that mod_caucho compiles aggresively in the beginning, killing > performance for a dry run ( even 60 seconds! ). To improve > the numbers for mod_caucho using this methodology might require > and longer test cycle than 60 seconds. > Perrin convinced me that getting "steady-state" numbers was important enough to ignore the compile penalties for such apps as mod_caucho which does runtime java compilation. To this end, I have added a -prime switch to the bench.pl which will run all the tests for a couple seconds, before doing the real benchmark where the results are scored. This allows any system caching to be done before the numbers start counting. However, despite the new -prime setting, I was still not getting reproducable resin/mod_caucho results, varying from 80 hits/sec one run to 280 hits/sec the next for the Hello World 2000 JSP benchmark. What I found is that in order for the mod_caucho results to be reproducable to 5% variation from one run to the next, I need to run the benchmark for 10 minutes! At 3-5 minutes, there was a 10-15% variation, which seems too high. Benchmarks < 120 seconds, which were my benchmarks from before seem next to meaningless now. Here's some #s to show what I mean: Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit - - - - - - - mod_caucho JSP 2000 h2000.jsp75.4 9069 120.33 0.013269 28965 mod_caucho JSP 2000 h2000.jsp 249.4 2505 10.05 0.004010 28965 mod_caucho JSP 2000 h2000.jsp 274.5 4118 15.00 0.003643 28965 mod_caucho JSP 2000 h2000.jsp 191.9 5760 30.01 0.005211 28965 mod_caucho JSP 2000 h2000.jsp 133.7 8022 60.01 0.007481 28965 mod_caucho JSP 2000 h2000.jsp 104.8 12591 120.10 0.009539 28965 mod_caucho JSP 2000 h2000.jsp96.0 11581 120.66 0.010419 28965 mod_caucho JSP 2000 h2000.jsp 113.6 19937 175.51 0.008803 28965 mod_caucho JSP 2000 h2000.jsp 103.9 18283 175.98 0.009625 28965 mod_caucho JSP 2000 h2000.jsp91.4 27146 297.16 0.010947 28965 mod_caucho JSP 2000 h2000.jsp 105.0 31335 298.57 0.009528 28965 mod_caucho JSP 2000 h2000.jsp91.4 27210 297.76 0.010943 28965 mod_caucho JSP 2000 h2000.jsp88.8 52440 590.56 0.011262 28965 mod_caucho JSP 2000 h2000.jsp93.5 55129 589.64 0.010696 28965 The numbers seem to indicate that the resin/mod_caucho java environment gets slow the longer tests are run for, but also that the results stabilize the longer tests are run for. As a result, when I post benchmarks, it will be with 10 minute tests run in the future. The idea here is stable, repeatable benchmarks we can use. Here were the 60 second #s I had posted before just to show how far off the mod_caucho #s were. > Test Name Test File Hits/sec Total Hits Total Time >sec/Hits Bytes/Hit > -- -- -- -- >-- -- > Apache::ASP v2.19 2000 h2000.asp 226.2 13578 hits 60.02 sec >0.004421 28998 byte > Apache::Registry v2.01 2000 mod_per h2000.reg 339.5 20376 hits 60.02 sec >0.002945 28179 byte > HTML::Embperl v1.3.0 2000 h2000.epl 111.3 6677 hits 60.00 sec >0.008987 28841 byte > HTML::Mason v1.03 2000 h2000.mas83.5 5014 hits 60.02 sec >0.011969 28799 byte > HTML::Template v2.3 2000h2000.htmp 98.2 5892 hits 60.00 sec >0.010183 29152 byte > mod_caucho JSP 2000 h2000.jsp76.6 4595 hits 60.01 sec >0.013060 28965 byte --Josh
Re: [ANNOUNCE] Hello World Benchmarks, updated
On Wed, 11 Jul 2001, Philip Mak wrote: > And sorry for my newbie-ish question, but what is the difference > between "mod_perl handler" and "Apache::Registry mod_perl"? http://perl.apache.org/guide/performance.html#Apache_Registry_PerlHandler_vs_ including the benchmarks _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://eXtropia.com/ http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: [ANNOUNCE] Hello World Benchmarks, updated
Perrin Harkins wrote: > > > I do feel that compile time matters, but really with 60 seconds > > and high MaxRequestsPerChild, these systems are getting plenty > > of compiling caching. > > The thing is, if mod_caucho takes 5 seconds the first time it hits each > template, but is the fastest afterwards, these numbers don't give a very > accurate picture of that. Most people expect a hit on the first access. > I get it, but I don't like it... why should an app get to spend ( hypothetically ) 60 seconds compiling a template into highly optimized assembly at runtime, and not have to show this cost in the benchmark. I'd be willing to run a test for longer, or run it multiple times, but to entirely throw out the compilation phase results just seems wrong to me. I'd like to see some middle ground here. > No need for anything that fancy. I'd say just run the test once as a primer > and throw away the results, like you were doing before. It could be for 10 > seconds instead of 60. > Another reason not to throw out the results is that it represents a legitimate web request as part of the apps life cycle. There might be some user sitting at the end of that 5 second delay. But then if we talk about throwing out highs & lows this starts to sound almost scientific. Like run 10 time slices, throw out the highest & lowest times? > I wouldn't worry about them interacting so much, but I did suggest running > multiple times and averaging. I think it helps smooth out random bad runs, > which do happen now and then. > > Any numbers on the new Apache::ASP CGI mode? > Horrid. Won't set up a benchmark yet, but its something like 3 hits/sec on my system compared with mod_cgi CGI.pm which is 11 hits/sec. ASP won't be optimized for mod_cgi type execution any time soon, which would require not loading in all the code per request which it does now for best mod_perl use. --Josh
Re: [ANNOUNCE] Hello World Benchmarks, updated
> I do feel that compile time matters, but really with 60 seconds > and high MaxRequestsPerChild, these systems are getting plenty > of compiling caching. The thing is, if mod_caucho takes 5 seconds the first time it hits each template, but is the fastest afterwards, these numbers don't give a very accurate picture of that. Most people expect a hit on the first access. > if we wanted to do away with compile > time entirely, we'd make sure that each system got to precompile > all its templates like Apache::RegistryLoader, Apache::ASP->Loader() > & Embperl's Execute() in the parent httpd. No need for anything that fancy. I'd say just run the test once as a primer and throw away the results, like you were doing before. It could be for 10 seconds instead of 60. > compile time is a very real problem for some > types of apps like large web sites, and there is an increased > burden on optimizing each benchmark, which implies an expertise > in the development environment that many users may not > have normally. I have been trying for more out of box benchmarks, > and not highly optimized benchmarks, using mostly the shipping > config for a system. I feel like allowing the templates to compile isn't tuning, just ignoring startup costs. > People have suggested before ( you? ) to do two runs, and > average the results, and I think that this is a fair approach, > and might be better than just doubling the test time because > the systems might interact, better yet, do them in one order, > and then reverse them, to average out sequential interactions > of the tests on a system ( ??? ) I wouldn't worry about them interacting so much, but I did suggest running multiple times and averaging. I think it helps smooth out random bad runs, which do happen now and then. Any numbers on the new Apache::ASP CGI mode? - Perrin
Re: [ANNOUNCE] Hello World Benchmarks, updated
Perrin Harkins wrote: > > > mod_caucho > > used to look a lot faster, but my testing methodology changed. > > I used to take the results of the second benchmark run, and > > publish those, but this time only ran the -test for minor > > caching after starting resin ( & tomcat ). So, I'm guessing > > that mod_caucho compiles aggresively in the beginning, killing > > performance for a dry run ( even 60 seconds! ). To improve > > the numbers for mod_caucho using this methodology might require > > and longer test cycle than 60 seconds. > > Ouch! I would think it's worth doing one full run to prime each system. Or > do you feel a need to include the initial compilation time? > I do feel that compile time matters, but really with 60 seconds and high MaxRequestsPerChild, these systems are getting plenty of compiling caching., these are thousands of requests we are talking about, what if we had lower MaxRequests, or big sites with lots of templates ... if we wanted to do away with compile time entirely, we'd make sure that each system got to precompile all its templates like Apache::RegistryLoader, Apache::ASP->Loader() & Embperl's Execute() in the parent httpd. Gerald suggested this before & I think it could be good, but for two reasons: compile time is a very real problem for some types of apps like large web sites, and there is an increased burden on optimizing each benchmark, which implies an expertise in the development environment that many users may not have normally. I have been trying for more out of box benchmarks, and not highly optimized benchmarks, using mostly the shipping config for a system. I have thought about having lower MaxClients as an option to the bench to help test compile times, but this doesn't affect the java engines which effectively have their own backend web servers running, like mod_proxy/mod_perl dual httpds. People have suggested before ( you? ) to do two runs, and average the results, and I think that this is a fair approach, and might be better than just doubling the test time because the systems might interact, better yet, do them in one order, and then reverse them, to average out sequential interactions of the tests on a system ( ??? ) --Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks Founder Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: [ANNOUNCE] Hello World Benchmarks, updated
Good work as usual, Joshua. > mod_caucho > used to look a lot faster, but my testing methodology changed. > I used to take the results of the second benchmark run, and > publish those, but this time only ran the -test for minor > caching after starting resin ( & tomcat ). So, I'm guessing > that mod_caucho compiles aggresively in the beginning, killing > performance for a dry run ( even 60 seconds! ). To improve > the numbers for mod_caucho using this methodology might require > and longer test cycle than 60 seconds. Ouch! I would think it's worth doing one full run to prime each system. Or do you feel a need to include the initial compilation time? - Perrin
Re: [ANNOUNCE] Hello World Benchmarks, updated
Good work as usual, Joshua. > mod_caucho > used to look a lot faster, but my testing methodology changed. > I used to take the results of the second benchmark run, and > publish those, but this time only ran the -test for minor > caching after starting resin ( & tomcat ). So, I'm guessing > that mod_caucho compiles aggresively in the beginning, killing > performance for a dry run ( even 60 seconds! ). To improve > the numbers for mod_caucho using this methodology might require > and longer test cycle than 60 seconds. Ouch! I would think it's worth doing one full run to prime each system. Or do you feel a need to include the initial compilation time? - Perrin
Re: [ANNOUNCE] Hello World Benchmarks, updated
Philip Mak wrote: > > One thing caught my eye; how come "mod_perl handler" (808.4 hits per > second) performed better than "HTML static" (768.2 hits per second)? > Here are my comments on this from the original post: HTML static for the first time, looks slower on my system than mod_perl. It may be that the default 1.3.20 apache install/config does more work on the headers. Also this is a dual proc system which has slowed down the HTML static test before, relative to single CPU systems. > And sorry for my newbie-ish question, but what is the difference > between "mod_perl handler" and "Apache::Registry mod_perl"? Check out the full code in the benchmarks at: http://www.chamas.com/bench/hello.tar.gz For the hello tests, one is an Apache::Registry run CGI script, and the other a small mod_perl module handler written for the hello bench. For the 2000 benchmark, the Registry mod_perl is a .pl script, but uses the modperl API for things like headers. -- Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks Founder Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: [ANNOUNCE] Hello World Benchmarks, updated
One thing caught my eye; how come "mod_perl handler" (808.4 hits per second) performed better than "HTML static" (768.2 hits per second)? And sorry for my newbie-ish question, but what is the difference between "mod_perl handler" and "Apache::Registry mod_perl"?
[ANNOUNCE] Hello World Benchmarks, updated
Hey, The latest Hello World benchmarks at available at: http://www.chamas.com/bench/hello.tar.gz To reproduce the BELOW results on your platform, for whatever tests are available, run: ./bench.pl -test ./bench.pl -version -time=60 --Josh DISCLAIMER: these benchmarks test only what they test for, and I try to make no assertions about the fitness of any system for your needs. They are open source, feel free to critique the tests as you like, and provide constructive feedback. CHANGES: New HTML::Template 2000 test. My system is newly built with apache 1.3.20, mod_perl 1.25. NOTES: mod_perl The mod_perl environments are starting to look faster than PHP. Maybe something in the default mod_perl 1.25 gave a performance boost? mod_caucho used to look a lot faster, but my testing methodology changed. I used to take the results of the second benchmark run, and publish those, but this time only ran the -test for minor caching after starting resin ( & tomcat ). So, I'm guessing that mod_caucho compiles aggresively in the beginning, killing performance for a dry run ( even 60 seconds! ). To improve the numbers for mod_caucho using this methodology might require and longer test cycle than 60 seconds. HTML::Embperl 2.0 series is much faster ( than PHP & ASP too), but 1.3.0 is stable, & installable from CPAN, so I run that. HTML::Mason < 1.0 version used to be much faster on 2000 benchmark mod_jserv Why is the hello bench slower than the 2000 bench? I don't know. It may be a funny caching strategy. It may be the JVMs were cleaning up from the 2000 test during the hello test. Speculation only, as I don't know much about java. HTML static for the first time, looks slower on my system than mod_perl. It may be that the default 1.3.20 apache install/config does more work on the headers. Also this is a dual proc system which has slowed down the HTML static test before, relative to single CPU systems. Test Name Test File Hits/sec Total Hits Total Time sec/Hits Bytes/Hit -- -- -- -- -- -- Apache::ASP v2.19 2000 h2000.asp 226.2 13578 hits 60.02 sec 0.004421 28998 byte Apache::Registry v2.01 2000 mod_per h2000.reg 339.5 20376 hits 60.02 sec 0.002945 28179 byte HTML::Embperl v1.3.0 2000 h2000.epl 111.3 6677 hits 60.00 sec 0.008987 28841 byte HTML::Mason v1.03 2000 h2000.mas83.5 5014 hits 60.02 sec 0.011969 28799 byte HTML::Template v2.3 2000h2000.htmp 98.2 5892 hits 60.00 sec 0.010183 29152 byte mod_caucho JSP 2000 h2000.jsp76.6 4595 hits 60.01 sec 0.013060 28965 byte mod_jserv JSP 2000 h2000.jsp 150.0 9002 hits 60.02 sec 0.006667 29408 byte mod_php PHP 2000h2000.php 228.4 13705 hits 60.01 sec 0.004379 28866 byte Template v2.04 Toolkit 2000 h2000.tt 52.9 3176 hits 60.02 sec 0.018900 28889 byte Apache::ASP v2.19 hello.asp 378.3 22706 hits 60.02 sec 0.002643 242 bytes Apache::Dispatch v0.09 handler hello/worl 587.6 35259 hits 60.01 sec 0.001702 197 bytes Apache::ePerl hello.eper 345.5 20742 hits 60.03 sec 0.002894 218 bytes Apache::Registry v2.01 CGI Raw hello_raw. 669.9 40196 hits 60.00 sec 0.001493 52 bytes Apache::Registry v2.01 CGI.pm hello.cgi 448.6 26924 hits 60.02 sec 0.002229 217 bytes Apache::SSI v2.16 hello.shtm 533.8 32029 hits 60.00 sec 0.001873 200 bytes HTML static hello.html 768.2 46120 hits 60.04 sec 0.001302 312 bytes HTML::Embperl v1.3.0hello.epl 459.8 27595 hits 60.01 sec 0.002175 221 bytes HTML::Mason v1.03 hello.mas 373.3 22406 hits 60.02 sec 0.002679 198 bytes HTML::Template v2.3 hello.htmp 535.9 32165 hits 60.02 sec 0.001866 199 bytes mod_caucho JSP hello.jsp88.1 5321 hits 60.37 sec 0.011345 231 bytes mod_cgi CGI Raw hello_raw. 153.5 9210 hits 60.00 sec 0.006515 197 bytes mod_cgi CGI.pm hello.cgi11.0 662 hits 60.02 sec 0.090660 217 bytes mod_include SSI hello.shtm 223.2 13396 hits 60.03 sec 0.004481 199 bytes mod_jserv JSP hello.jsp53.5 3223 hits 60.29 sec 0.018706 358 bytes mod_perl handlerhello.benc 808.4 48522 hits 60.02 sec 0.001237 197 bytes mod_php PHP hello.php 665.9 39954 hits 60.00 sec 0.001502 226 bytes mod_speedycgi hello.
