Daniel wrote (in part):
>My guess is that NT pro people studied carefully Linux to find
>a weak point on which NT is superior, and then invent a benchmark to
>claim general superiority. They forgot to check whether the
>effective superiority corresponds to a widespread need.
Certainly people do examine the "competition" to see if they can
develop benchmarks that prove the point they want to make. But I
generally don't think that much effort is put into it because doing
it badly and getting caught is a very big risk.
No I think the fact is benchmarks really are poorly done overall. For
example I worked in a lab for a major computer maker. We ran
client-server benchmarks in the lab regularly against new servers.
The principle measure they chose was number of clients served.
Every few months they upgraded the clients used in testing. What's
wrong with this picture? Well the clients got smarter every few
months and the base line of the test changed. So the servers had to
get substantially better to show any improvement. If they wanted to
show improvement they should have stayed with XT systems forever.
The people doing the job don't know how to present the data they have
and the customers obviously don't see the flaws in the presentation.
When it came to displaying the client-server performance data
everyone shows a continuous curve with magnitude up and down and
number of clients left and right. The number of clients is always a
whole number so you can't have a continuous curve.
What are my points?
1. With benchmarks I think the level of chicanery is low and so it
the value of the information.
2. There is no agreement on what is a reasonable workload.
3. The level of design experience applied to benchmarking is not as
high as other areas in the industry.
END
*************** *************** *************** **********
Douglas W. St.Clair
Tir na nOg
400 Burton Highway
Wilton, NH 03086
PH: 603-654-9321
FAX: 603-654-5440
CELL: 617-233-3387 or 617-beef-fur
*************** *************** *************** **********
-
Linux SMP list: FIRST see FAQ at http://www.irisa.fr/prive/mentre/smp-faq/
To Unsubscribe: send "unsubscribe linux-smp" to [EMAIL PROTECTED]
