> I think too that the OS/machine results at
> http://www.chamas.com/bench/hello_bycode.html could be more accurate
> in comparing results if the results are also grouped by tester,
> network connection type, and testing client so each grouping would
> well reflect the relative speed differences web applications on the
> same platform.
Agreed.
> I would argue that we should keep the code type grouping listed at
> http://www.chamas.com/bench/hello_bycode.html because it gives
> a good feel for how some operating systems & web servers are faster
> than others, i.e., Solaris slower than Linux, WinNT good for static
> HTML, Apache::ASP faster than IIS/ASP PerlScript, etc.
See, I don't think you can even make statements like that based on these
benchmarks. Where is the test on Solaris x86 and Linux done by the same
person under the same conditions? I don't see one. Where is the test of NT
and Linux on the same machine by the same person? Even the Apache::ASP vs
PerlScript comparisons you did seem to be using different clients, netowork
setups, and versions of NT.
I'm not criticizing you for not being able to get lab-quality results, but I
think we have to be careful what conclusions we draw from these.
> Finally, I would very much like to keep the fastest benchmark page
> as the first page, disclaiming it to death if necessary, the reason
> being that I would like to encourage future submissions, with
> new & faster hardware & OS configurations, and the best way to do
> that is to have something of a benchmark competition happening on the
> first page of the results.
I can understand that; I just don't want mod_perl users to get a reputation
as the Mindcraft of web application benchmarks.
> It seems that HTTP 1.1 submissions represent a small subset of
> skewed results, should these be dropped or presented separately?
I'd say they're as meaningful as any of the others if you consider them
independently of the other contributions.
> I also need to clarify some results, or back them up somehow.
> What should I do with results that seem skewed in general?
> Not post them until there is secondary confirmation ?
Your call. Again, to my mind each person's contribution can only be viewed
in its own private context, so one is no more skewed than any other.
- Perrin
