Robert G. Brown wrote:
This, in turn, was supposed to stimulate a discussion on whether or not
benchmarks of this sort "need" to be licensed and controlled (I would
argue a resounding "no") and if so controlled by whom, to prevent vendor
I agree. Benchmarks, and all data around them are an example of a need
for transparency. Without such transparency, well, you never really
know what you have. Unfortunately benchmark design and reporting is
rather suspect as well. Anyone advocating taking a large
multi-dimensional space and reducing it to a zero-dimensional quantity
by means of a non-intuitive averaging scheme ought to think hard about
the vast quantities of information being destroyed by such a process.
IMO that is.
abuse or benchmark drift or the healthy discussion, criticism, and
modification that results in a true open source process. This might
have lead to discussing e.g. the top500 rankings and linpack and other
(more open) macro benchmarks, whether the "member organizations" of SPEC
Heh... some of us have gotten together to try to get something
reasonable (http://www.scalableinformatics.com/bbs), and completely open
source created for LS benchmarking. All tests, test cases, and code is
OSS, and specifically the stuff being used by researchers is used. We
have had lots of people pull it down (over 1000) in the 2 years it has
been out, but alas there seems to be little interest in the community to
contributing test cases and code snippets, and lots of interest in
simply consuming baseline results. So work on the subsequent versions
takes a long time as it is done in one persons "copious" spare time.
are above reproach and have no self-interest to promote (Oh wait! The
member list, who doubtless pony up the actual cash that runs the
non-profit, is made up of all the top COMPANIES that sell HARDWARE? No
hint of self-interest there...;-).
Hmmm.... we recently responded to a government RFP where they "require"
runs on the hardware from the spec suite.
[soap box]
This is counter productive IMO as the spec suite really doesn't do
much for you in terms of meaningful performance measurements. Does
about the same as Linpack/HPL. Yeah, some people argue otherwise, and
compiler vendors and hardware OEMs/resellers work really hard to put out
great benchmark data. I don't believe such data is terribly meaningful,
and customers who rely upon it for purchase decisions may wind up making
decisions in part based upon "data" which has little intrinsic value to
the tasks at hand.
The most important benchmark is the one that uses the same code you use
in the way you are going to use it. Anything else is an entropy
generator.
[/soap box]
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: [EMAIL PROTECTED]
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 734 786 8452
cell : +1 734 612 4615
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
