Joshua Chamas wrote:
>
> mod_caucho
> used to look a lot faster, but my testing methodology changed.
> I used to take the results of the second benchmark run, and
> publish those, but this time only ran the -test for minor
> caching after starting resin ( & tomcat ). So, I'm guessing
> that mod_caucho compiles aggresively in the beginning, killing
> performance for a dry run ( even 60 seconds! ). To improve
> the numbers for mod_caucho using this methodology might require
> and longer test cycle than 60 seconds.
>
Perrin convinced me that getting "steady-state" numbers
was important enough to ignore the compile penalties
for such apps as mod_caucho which does runtime java
compilation.
To this end, I have added a -prime switch to the bench.pl
which will run all the tests for a couple seconds, before
doing the real benchmark where the results are scored.
This allows any system caching to be done before the
numbers start counting.
However, despite the new -prime setting, I was still not
getting reproducable resin/mod_caucho results, varying
from 80 hits/sec one run to 280 hits/sec the next for
the Hello World 2000 JSP benchmark.
What I found is that in order for the mod_caucho results
to be reproducable to 5% variation from one run to the
next, I need to run the benchmark for 10 minutes! At 3-5
minutes, there was a 10-15% variation, which seems too high.
Benchmarks < 120 seconds, which were my benchmarks from before
seem next to meaningless now.
Here's some #s to show what I mean:
Test Name Test File Hits/sec # of Hits Time(sec)
secs/Hit Bytes/Hit
--------- --------- --------- --------- ---------
--------- ---------
mod_caucho JSP 2000 h2000.jsp 75.4 9069 120.33
0.013269 28965
mod_caucho JSP 2000 h2000.jsp 249.4 2505 10.05
0.004010 28965
mod_caucho JSP 2000 h2000.jsp 274.5 4118 15.00
0.003643 28965
mod_caucho JSP 2000 h2000.jsp 191.9 5760 30.01
0.005211 28965
mod_caucho JSP 2000 h2000.jsp 133.7 8022 60.01
0.007481 28965
mod_caucho JSP 2000 h2000.jsp 104.8 12591 120.10
0.009539 28965
mod_caucho JSP 2000 h2000.jsp 96.0 11581 120.66
0.010419 28965
mod_caucho JSP 2000 h2000.jsp 113.6 19937 175.51
0.008803 28965
mod_caucho JSP 2000 h2000.jsp 103.9 18283 175.98
0.009625 28965
mod_caucho JSP 2000 h2000.jsp 91.4 27146 297.16
0.010947 28965
mod_caucho JSP 2000 h2000.jsp 105.0 31335 298.57
0.009528 28965
mod_caucho JSP 2000 h2000.jsp 91.4 27210 297.76
0.010943 28965
mod_caucho JSP 2000 h2000.jsp 88.8 52440 590.56
0.011262 28965
mod_caucho JSP 2000 h2000.jsp 93.5 55129 589.64
0.010696 28965
The numbers seem to indicate that the resin/mod_caucho java
environment gets slow the longer tests are run for, but also
that the results stabilize the longer tests are run for.
As a result, when I post benchmarks, it will be with
10 minute tests run in the future. The idea here is
stable, repeatable benchmarks we can use.
Here were the 60 second #s I had posted before just to show
how far off the mod_caucho #s were.
> Test Name Test File Hits/sec Total Hits Total Time
>sec/Hits Bytes/Hit
> ------------ ---------- ---------- ---------- ----------
>---------- ----------
> Apache::ASP v2.19 2000 h2000.asp 226.2 13578 hits 60.02 sec
>0.004421 28998 byte
> Apache::Registry v2.01 2000 mod_per h2000.reg 339.5 20376 hits 60.02 sec
>0.002945 28179 byte
> HTML::Embperl v1.3.0 2000 h2000.epl 111.3 6677 hits 60.00 sec
>0.008987 28841 byte
> HTML::Mason v1.03 2000 h2000.mas 83.5 5014 hits 60.02 sec
>0.011969 28799 byte
> HTML::Template v2.3 2000 h2000.htmp 98.2 5892 hits 60.00 sec
>0.010183 29152 byte
> mod_caucho JSP 2000 h2000.jsp 76.6 4595 hits 60.01 sec
>0.013060 28965 byte
--Josh
