Re: [ANNOUNCE] Hello World Benchmarks, updated
Joshua Chamas wrote: mod_caucho used to look a lot faster, but my testing methodology changed. I used to take the results of the second benchmark run, and publish those, but this time only ran the -test for minor caching after starting resin ( tomcat ). So, I'm guessing that mod_caucho compiles aggresively in the beginning, killing performance for a dry run ( even 60 seconds! ). To improve the numbers for mod_caucho using this methodology might require and longer test cycle than 60 seconds. Perrin convinced me that getting steady-state numbers was important enough to ignore the compile penalties for such apps as mod_caucho which does runtime java compilation. To this end, I have added a -prime switch to the bench.pl which will run all the tests for a couple seconds, before doing the real benchmark where the results are scored. This allows any system caching to be done before the numbers start counting. However, despite the new -prime setting, I was still not getting reproducable resin/mod_caucho results, varying from 80 hits/sec one run to 280 hits/sec the next for the Hello World 2000 JSP benchmark. What I found is that in order for the mod_caucho results to be reproducable to 5% variation from one run to the next, I need to run the benchmark for 10 minutes! At 3-5 minutes, there was a 10-15% variation, which seems too high. Benchmarks 120 seconds, which were my benchmarks from before seem next to meaningless now. Here's some #s to show what I mean: Test Name Test File Hits/sec # of Hits Time(sec) secs/Hit Bytes/Hit - - - - - - - mod_caucho JSP 2000 h2000.jsp75.4 9069 120.33 0.013269 28965 mod_caucho JSP 2000 h2000.jsp 249.4 2505 10.05 0.004010 28965 mod_caucho JSP 2000 h2000.jsp 274.5 4118 15.00 0.003643 28965 mod_caucho JSP 2000 h2000.jsp 191.9 5760 30.01 0.005211 28965 mod_caucho JSP 2000 h2000.jsp 133.7 8022 60.01 0.007481 28965 mod_caucho JSP 2000 h2000.jsp 104.8 12591 120.10 0.009539 28965 mod_caucho JSP 2000 h2000.jsp96.0 11581 120.66 0.010419 28965 mod_caucho JSP 2000 h2000.jsp 113.6 19937 175.51 0.008803 28965 mod_caucho JSP 2000 h2000.jsp 103.9 18283 175.98 0.009625 28965 mod_caucho JSP 2000 h2000.jsp91.4 27146 297.16 0.010947 28965 mod_caucho JSP 2000 h2000.jsp 105.0 31335 298.57 0.009528 28965 mod_caucho JSP 2000 h2000.jsp91.4 27210 297.76 0.010943 28965 mod_caucho JSP 2000 h2000.jsp88.8 52440 590.56 0.011262 28965 mod_caucho JSP 2000 h2000.jsp93.5 55129 589.64 0.010696 28965 The numbers seem to indicate that the resin/mod_caucho java environment gets slow the longer tests are run for, but also that the results stabilize the longer tests are run for. As a result, when I post benchmarks, it will be with 10 minute tests run in the future. The idea here is stable, repeatable benchmarks we can use. Here were the 60 second #s I had posted before just to show how far off the mod_caucho #s were. Test Name Test File Hits/sec Total Hits Total Time sec/Hits Bytes/Hit -- -- -- -- -- -- Apache::ASP v2.19 2000 h2000.asp 226.2 13578 hits 60.02 sec 0.004421 28998 byte Apache::Registry v2.01 2000 mod_per h2000.reg 339.5 20376 hits 60.02 sec 0.002945 28179 byte HTML::Embperl v1.3.0 2000 h2000.epl 111.3 6677 hits 60.00 sec 0.008987 28841 byte HTML::Mason v1.03 2000 h2000.mas83.5 5014 hits 60.02 sec 0.011969 28799 byte HTML::Template v2.3 2000h2000.htmp 98.2 5892 hits 60.00 sec 0.010183 29152 byte mod_caucho JSP 2000 h2000.jsp76.6 4595 hits 60.01 sec 0.013060 28965 byte --Josh
Re: [ANNOUNCE] Hello World Benchmarks, updated
One thing caught my eye; how come mod_perl handler (808.4 hits per second) performed better than HTML static (768.2 hits per second)? And sorry for my newbie-ish question, but what is the difference between mod_perl handler and Apache::Registry mod_perl?
Re: [ANNOUNCE] Hello World Benchmarks, updated
Good work as usual, Joshua. mod_caucho used to look a lot faster, but my testing methodology changed. I used to take the results of the second benchmark run, and publish those, but this time only ran the -test for minor caching after starting resin ( tomcat ). So, I'm guessing that mod_caucho compiles aggresively in the beginning, killing performance for a dry run ( even 60 seconds! ). To improve the numbers for mod_caucho using this methodology might require and longer test cycle than 60 seconds. Ouch! I would think it's worth doing one full run to prime each system. Or do you feel a need to include the initial compilation time? - Perrin
Re: [ANNOUNCE] Hello World Benchmarks, updated
Good work as usual, Joshua. mod_caucho used to look a lot faster, but my testing methodology changed. I used to take the results of the second benchmark run, and publish those, but this time only ran the -test for minor caching after starting resin ( tomcat ). So, I'm guessing that mod_caucho compiles aggresively in the beginning, killing performance for a dry run ( even 60 seconds! ). To improve the numbers for mod_caucho using this methodology might require and longer test cycle than 60 seconds. Ouch! I would think it's worth doing one full run to prime each system. Or do you feel a need to include the initial compilation time? - Perrin
Re: [ANNOUNCE] Hello World Benchmarks, updated
Perrin Harkins wrote: mod_caucho used to look a lot faster, but my testing methodology changed. I used to take the results of the second benchmark run, and publish those, but this time only ran the -test for minor caching after starting resin ( tomcat ). So, I'm guessing that mod_caucho compiles aggresively in the beginning, killing performance for a dry run ( even 60 seconds! ). To improve the numbers for mod_caucho using this methodology might require and longer test cycle than 60 seconds. Ouch! I would think it's worth doing one full run to prime each system. Or do you feel a need to include the initial compilation time? I do feel that compile time matters, but really with 60 seconds and high MaxRequestsPerChild, these systems are getting plenty of compiling caching., these are thousands of requests we are talking about, what if we had lower MaxRequests, or big sites with lots of templates ... if we wanted to do away with compile time entirely, we'd make sure that each system got to precompile all its templates like Apache::RegistryLoader, Apache::ASP-Loader() Embperl's Execute() in the parent httpd. Gerald suggested this before I think it could be good, but for two reasons: compile time is a very real problem for some types of apps like large web sites, and there is an increased burden on optimizing each benchmark, which implies an expertise in the development environment that many users may not have normally. I have been trying for more out of box benchmarks, and not highly optimized benchmarks, using mostly the shipping config for a system. I have thought about having lower MaxClients as an option to the bench to help test compile times, but this doesn't affect the java engines which effectively have their own backend web servers running, like mod_proxy/mod_perl dual httpds. People have suggested before ( you? ) to do two runs, and average the results, and I think that this is a fair approach, and might be better than just doubling the test time because the systems might interact, better yet, do them in one order, and then reverse them, to average out sequential interactions of the tests on a system ( ??? ) --Josh _ Joshua Chamas Chamas Enterprises Inc. NodeWorks Founder Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051
Re: [ANNOUNCE] Hello World Benchmarks, updated
I do feel that compile time matters, but really with 60 seconds and high MaxRequestsPerChild, these systems are getting plenty of compiling caching. The thing is, if mod_caucho takes 5 seconds the first time it hits each template, but is the fastest afterwards, these numbers don't give a very accurate picture of that. Most people expect a hit on the first access. if we wanted to do away with compile time entirely, we'd make sure that each system got to precompile all its templates like Apache::RegistryLoader, Apache::ASP-Loader() Embperl's Execute() in the parent httpd. No need for anything that fancy. I'd say just run the test once as a primer and throw away the results, like you were doing before. It could be for 10 seconds instead of 60. compile time is a very real problem for some types of apps like large web sites, and there is an increased burden on optimizing each benchmark, which implies an expertise in the development environment that many users may not have normally. I have been trying for more out of box benchmarks, and not highly optimized benchmarks, using mostly the shipping config for a system. I feel like allowing the templates to compile isn't tuning, just ignoring startup costs. People have suggested before ( you? ) to do two runs, and average the results, and I think that this is a fair approach, and might be better than just doubling the test time because the systems might interact, better yet, do them in one order, and then reverse them, to average out sequential interactions of the tests on a system ( ??? ) I wouldn't worry about them interacting so much, but I did suggest running multiple times and averaging. I think it helps smooth out random bad runs, which do happen now and then. Any numbers on the new Apache::ASP CGI mode? - Perrin
Re: [ANNOUNCE] Hello World Benchmarks, updated
Perrin Harkins wrote: I do feel that compile time matters, but really with 60 seconds and high MaxRequestsPerChild, these systems are getting plenty of compiling caching. The thing is, if mod_caucho takes 5 seconds the first time it hits each template, but is the fastest afterwards, these numbers don't give a very accurate picture of that. Most people expect a hit on the first access. I get it, but I don't like it... why should an app get to spend ( hypothetically ) 60 seconds compiling a template into highly optimized assembly at runtime, and not have to show this cost in the benchmark. I'd be willing to run a test for longer, or run it multiple times, but to entirely throw out the compilation phase results just seems wrong to me. I'd like to see some middle ground here. No need for anything that fancy. I'd say just run the test once as a primer and throw away the results, like you were doing before. It could be for 10 seconds instead of 60. Another reason not to throw out the results is that it represents a legitimate web request as part of the apps life cycle. There might be some user sitting at the end of that 5 second delay. But then if we talk about throwing out highs lows this starts to sound almost scientific. Like run 10 time slices, throw out the highest lowest times? I wouldn't worry about them interacting so much, but I did suggest running multiple times and averaging. I think it helps smooth out random bad runs, which do happen now and then. Any numbers on the new Apache::ASP CGI mode? Horrid. Won't set up a benchmark yet, but its something like 3 hits/sec on my system compared with mod_cgi CGI.pm which is 11 hits/sec. ASP won't be optimized for mod_cgi type execution any time soon, which would require not loading in all the code per request which it does now for best mod_perl use. --Josh
Re: [ANNOUNCE] Hello World Benchmarks, updated
On Wed, 11 Jul 2001, Philip Mak wrote: And sorry for my newbie-ish question, but what is the difference between mod_perl handler and Apache::Registry mod_perl? http://perl.apache.org/guide/performance.html#Apache_Registry_PerlHandler_vs_ including the benchmarks _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://eXtropia.com/ http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
