Perrin Harkins wrote:
>
> > mod_caucho
> > used to look a lot faster, but my testing methodology changed.
> > I used to take the results of the second benchmark run, and
> > publish those, but this time only ran the -test for minor
> > caching after starting resin ( & tomcat ). So, I'm guessing
> > that mod_caucho compiles aggresively in the beginning, killing
> > performance for a dry run ( even 60 seconds! ). To improve
> > the numbers for mod_caucho using this methodology might require
> > and longer test cycle than 60 seconds.
>
> Ouch! I would think it's worth doing one full run to prime each system. Or
> do you feel a need to include the initial compilation time?
>
I do feel that compile time matters, but really with 60 seconds
and high MaxRequestsPerChild, these systems are getting plenty
of compiling caching., these are thousands of requests we are
talking about, what if we had lower MaxRequests, or big sites
with lots of templates ... if we wanted to do away with compile
time entirely, we'd make sure that each system got to precompile
all its templates like Apache::RegistryLoader, Apache::ASP->Loader()
& Embperl's Execute() in the parent httpd.
Gerald suggested this before & I think it could be good, but for
two reasons: compile time is a very real problem for some
types of apps like large web sites, and there is an increased
burden on optimizing each benchmark, which implies an expertise
in the development environment that many users may not
have normally. I have been trying for more out of box benchmarks,
and not highly optimized benchmarks, using mostly the shipping
config for a system.
I have thought about having lower MaxClients as an option to the
bench to help test compile times, but this doesn't affect the
java engines which effectively have their own backend web servers
running, like mod_proxy/mod_perl dual httpds.
People have suggested before ( you? ) to do two runs, and
average the results, and I think that this is a fair approach,
and might be better than just doubling the test time because
the systems might interact, better yet, do them in one order,
and then reverse them, to average out sequential interactions
of the tests on a system ( ??? )
--Josh
_________________________________________________________________
Joshua Chamas Chamas Enterprises Inc.
NodeWorks Founder Huntington Beach, CA USA
http://www.nodeworks.com 1-714-625-4051
